Improve discoverability of your work on HF

[NielsRogge]

Hi,

Niels here from the open-source team at Hugging Face. Congrats on your work! I found it based on the paper page: https://huggingface.co/papers/2406.03344 (feel free to claim the paper with your HF account so that it appears under your profile).

However, I've got some suggestions regarding how to improve the integration with HF.

1. Upload checkpoints to the hub

It seems that currently, all AudioMamba models are hosted on Google Drive. It would be great to push them to the hub, along with pipeline tags like audio-classification. The easiest way to fix that is by leveraging the PyTorchModelHubMixin class, as it adds push_to_hub and from_pretrained capabilities to any custom nn.Module. It creates a config.json along with safetensors for each model, enforcing downloads to work.

This way, people will find your models when searching https://huggingface.co/models?pipeline_tag=audio-classification&sort=trending.

2. Make the model Transformers compatible

In case you want your models to be usable through the Transformers library with trust_remote_code=True, I highly recommend following this guide: https://huggingface.co/docs/transformers/custom_models. It basically allows people to use your backbones using the AutoModel and AutoModelForAudioClassification APIs.

We recently did the same with the MambaVision author as can be seen here: https://huggingface.co/collections/nvidia/mambavision-66943871a6b36c9e78b327d3, which was a great success!

Let me know if you need any help regarding this!

Cheers,

Niels
ML Engineer @ HF 🤗

[iamrobzy]

Hello @NielsRogge!

I am planning to finetune the AudioMamba model and would like to set up the Inference API for this finetuned model via the Transformers library. Making the model Transformers compatible seems to be a pain in butt due to the demand of having everything in one directory. The original code contains many imports from many different sub-directories...

Do you have any recommendations on how to go about it? Thanks

[NielsRogge]

Hi,

Yes for a model to be Transformers compatible, modeling_audio_mamba.py and configuration_audio_mamba.py files would need to be defined. An example is the MambaVision model.

The authors defined a standalone package "mambavision" which needs to be installed besides Transformers for the model to be used: https://huggingface.co/nvidia/MambaVision-T-1K#model-usage