Document common CV patterns

[GregoryComer]

📚 The doc issue

There are a number of CV-specific tips, tricks, and techniques that are helpful for deploying CV models on ExecuTorch. From internal adoption, we've seen that CV models make up a sizable portion of on-device use cases, so I think it's worth having some docs to cover a few things. It will also allow me to link to it in the getting started doc (which uses torchvision v2) to avoid needing to explain input transformations and such there.

I'm not entirely sure where to put this yet. I don't necessarily want to stick it under Usage/, since I'm trying to keep that higher-level. Maybe we need a new section for moderate to advanced usage docs for this type of thing. We can also maybe stick docs on export compliance tricks (#8731) and model optimization techniques there, as well.

Things to cover:

• Wrapping a model with common input transformations:
  • Image resize
  • Normalization
  • Dtype conversion for u8 input
• Input and output format for common classes of models
  • Channels first vs channels last format
  • Examples of loading images and converting to tensor format on Android + iOS
  • Briefly cover interpreting output class scores, masks, instance segmentation outputs, etc. Don't go into too much detail, but give a quick overview and maybe some Pytorch or torchvision links.

Suggest a potential alternative/fix

No response

cc @mergennachin @AlannaBurke @byjlw

[oalvarobraz]

Hi @GregoryComer ! I see this issue was recently tagged for the Docathon.[ Are you still working on it, or can I take it over? /assigntome

[GregoryComer]

@oalvarobraz Sure!

Here are a few additional thoughts, as the issue, as written, is somewhat vague. I'd maybe consider adding a new page under Advanced called "Working with CV Models" or something similar.

I was generally thinking of covering the following:

• Input transformations - resize and crop. Many models have specific input sizes they expect.
  • Showing how to do resizing and cropping in the exported model.
  • Showing how to do resizing and cropping in app code.
• Interpreting outputs
  • For image classifiers, show how to find the maximum class from the output tensor and mapping it back to a label.
  • For instance segmentation, how to interpret outputs for common models.

Looking at the existing DL3 example might be a good place to start. Also, feel free to reach out if you have any questions. Thanks!