[docs] Redesign

[stevhliu]

The main goal of this PR is to redesign the Transformers docs to:

1. Be more developer-friendly.
2. Improve navigation by replacing the existing structure with a more organic one that scales naturally instead of forcing content into the 4 current predefined sections.
3. Create a more unified docs experience by integrating content rather than adding it on.

This PR proposes a potential structure for achieving 2 and 3. Once the structure is in place, each doc will be rewritten to achieve 1.

If you're interested in more details about the redesign's motivation, please read this blog post. If you want more details about 1, 2, and 3, please read this post and this one too.

All feedback, alternative structures, and comments are welcomed! Thanks 🙂

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[gante]

Like! 👍

[gante]

A type of model that's becoming increasingly common are VLMs: they are the same as LLMs, but also accept image inputs.

Would it make sense to call this section "LLMs and VLMs"?

[stevhliu]

For sure! Let's rename the section when we have some VLM-specific docs?

[ydshieh]

Indeed easier to read ❤️ . But there are a few places need to be moved if I understand correctly?

[stevhliu]

I've kicked off the redesign with the "Get Started" section. Feel free to review this section while I start on the next one (Base classes)!

The main changes are:

index.md

• A cleaner index page that better describes what Transformers is in terms of its features and design. I believe this is more impactful than listing all the tasks you can solve across modalities. The focus shouldn't be on the tasks that you can solve; it should be on the models themselves. By describing the type of models available, I think users will understand that they can use them for their tasks. Having a more holistic description of the library here is more important than focusing on the different tasks/modalities.
• More of a question here, but would it be better to maybe have badges on each model API doc that indicate whether it supports PyTorch/TensorFlow/Flax? Instead of having/maintaining such a long list that clutters up the main landing page, I think it'd be a lot cleaner to have this information on each model page. This way, users can see everything at once on the model page.

quicktour.md

• Removed the "vertical" PyTorch/TensorFlow blocks in favor of the "horizontal" ones which I think it cleaner and less overwhelming.
• Removed the big table of tasks available to Pipeline with just three code examples. I think this makes it simpler and more approachable. I also removed the Pipeline video because it felt very NLP-heavy, but we can add it back if we want to keep it.
• Updated the AutoClass section to also be simpler and faster for users to start. A lot of these details (eg, tensors are outputted before the final activation function, custom model builds) can be explained in more depth in later docs. Also took this opportunity to introduce the generate API.
• A better Next steps section directing users to topics of interest.

installation.md

• Removed the options for downloading files in favor of just one method to keep it simple, and link to the Download files from the Hub doc for more details.

[stevhliu]

Hi, I'm back with an update! I've wrapped up the technical guides in the Models section. I'll circle back to the more conceptual docs later and also create some visual diagrams in Figma. Next up, I'll start working on the Preprocessors section. 🙂

The main focus is on how to load, customize, share, and contribute a model, basically a one-stop section for all your general model docs. The Load and Contribute docs have more significant changes:

models.md

• Repurposed to show how to load a model. I start with a quick example of AutoModelFor.from_pretrained() so you can immediately get started, and then progressively peel back the layers. From how models and configurations interact, to the AutoClass API, and then model-specific classes. To make it easier to find how to load any model, I also added big models (device_map="auto") and custom models (trust_remote_code="True")to this page.

add_new_model.md

• Updated structure to make the steps more discoverable. Before, many of the steps were hidden in "5.-14. Port BrandNewBert to Transformers" but now what these actual steps are more clear.

[stevhliu]

Finished the first draft of the Tokenizers doc, and I'm pretty excited that it reduces "content creep" from 6 different docs to just 1! 😎

[stevhliu]

The first draft of the practical guides in the base classes section is finished now! Please feel free to check it out and leave any comments or feedback (not sure why the feature extractor and processor docs aren't showing in the preview at the moment) 😄

I'll start working on the inference section after I review the first draft.

[Ninjadeveloper007]

Ok guys let me the self taught developer have it roast my ass! just provide much needed mentorship afterwards?!🫣😬🥸, i

luv u guys?

🥹