Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create standard way for preprocessors to do graphic resizing #867

Open
jeffbl opened this issue Aug 5, 2024 · 5 comments
Open

Create standard way for preprocessors to do graphic resizing #867

jeffbl opened this issue Aug 5, 2024 · 5 comments
Assignees
Labels
enhancement New feature or request

Comments

@jeffbl
Copy link
Member

jeffbl commented Aug 5, 2024

Talking with @AndyBaiMQC this morning, it will likely be necessary to resize graphics going to local ML models, each of which may have its own constraints. We've been burned by resizing issues numerous times throughout the project (example example , and there are others). Goal of this work item is to come up with a clear and documented way for preprocessors to resize graphics. Key functionality includes:

  • Must support a wide variety of graphic formats (jpg, png, gif, etc.)
  • good performance
  • easy to integrate (easy, clean API that is hard to mess up)

Assigning to @JRegimbal first to weigh in on architecture, before going to @jaydeepsingh25 for implementation. Some options:

  • Choose a library and have each preprocessor use it directly
  • Implement a service that encapsulates resizing so that if we want to change it in the future, we only have to change it in one place (e.g., to support a new graphic format, or if the library we initially choose turns out to have flaws or is no longer supported, etc)
  • Have the orchestrator resize before sending to preprocessor based on a parameter in docker-compose
  • ???

BONUS: If we centralize this, we could also use it to transform photos into a common format, to support a wider variety of weird graphic types. E.g., if the library can transform .png into .jpg as well, then we don't have to rely on preprocessors supporting .png. Probably should break out into separate issue, or generalize this one into "transform graphics before they go to preprocessors)

@jeffbl jeffbl added the enhancement New feature or request label Aug 5, 2024
@jeffbl
Copy link
Member Author

jeffbl commented Aug 5, 2024

Putting in current sprint to at least decide what to do, even if implementation is later. @AndyBaiMQC please don't spend time on resizing now, so that we can decide on an overall plan.

@JRegimbal
Copy link
Collaborator

Hmm...if we really want to have image resizing as a shared feature I think the most convenient options would be to build it into the orchestrator or have all the preprocessors just use pillow. My immediate reaction to including it in the orchestrator is hesitancy, since we want to keep it small to avoid bugs, but we could probably use something like sharp in cases where 1) a graphic is included in the request, and 2) the preprocessor uses labels to request some maximum size/format.

Is resizing/reencoding the only tasks that would be necessary, @AndyBaiMQC?

@JRegimbal
Copy link
Collaborator

Reassigning to @AndyBaiMQC and @jaydeepsingh25 since we're taking the common library route.

@jeffbl
Copy link
Member Author

jeffbl commented Aug 23, 2024

Moving to backlog until there is space in a sprint for this.

@AndyBaiMQC
Copy link
Contributor

@jeffbl Circling back to this one: Need a comprehensive review on list of models/methods for existing preprocessors, and the changes following Ollama adoption (which hopefully makes it easier to do 'one size fits as many as possible')

@jeffbl jeffbl assigned shahdyousefak and unassigned AndyBaiMQC Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

5 participants