feat: detach resources from providers #53
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Uses an internal project json to record the label rows that have been completely downloaded, thus avoiding any further check on their images. Also, label rows' contents that are already up-to-date won't be downloaded again, even when `overwrite_annotations` is set to `True` (via last edited datetime comparison).
Reduce list_label_rows_v2 calls on already downloaded projects while keeping a small overhead on unseen datasets.
Add singleton pattern to providers in order to ensure that loading data from sources is performed after all built-in imports have been resolved. Otherwise, we will find partially initialised module errors.
eloy-encord
force-pushed
the
eloy/feat-detach-resources-from-providers
branch
from
9621488 to
1b3d592
Compare
Providers not longer need to call the prepare method as it's now embedded in the remaining methods.
frederik-encord
approved these changes
Apr 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Types and definitions of models and datasets are now dynamically fetched from a
sourcesfolder. With this change, previously built-in resources can be modified, silenced or eliminated if the user wants to. Also, enables seamless addition of new models and datasets by just adding the corresponding implementation (new type) or definition (instance of an existing type).On the side, refactored the naming convention of several components that were moved with the previous changes, so they share the same patterns. E.g. the model base class went from
CLIPModeltoModelas the class itself is related to a higher abstraction than the one of clip.Also, added the singleton pattern and some minor changes to the dataset and model providers in order to avoid partially initialised module errors that appeared when dynamic resources were (tried) loaded before all local resources were successfully imported. The singleton pattern is specifically used to avoid the creation of several provider instances as the original global provider couldn't be loaded on initialisation.