Add number of extracted, loaded, and failed resources in batch responses #693
Conversation
This field is a SerializerMethodField that fetches the progression for each resource from redis
Jasopaum
changed the title
Jasopaum
requested review from
tevariou and
simonvadee
and removed request for
tevariou
There was a problem hiding this comment.
IMO it's best to use a dedicated table for Progression. having a jsonfield with a custom structure in the Batch table doesn't make sense in a relational db, especially if we want to make queries involving a progression counter by resource_id.
@tevariou what's your opinion ?
I agree it may be more practical to have a dedicated table for filtering |
|
Also, @tevariou, @simonvadee, do you know if there is an easy way to know the number of resources that have been updated compared to inserted? |
good question: there isn't. In |
Jasopaum
changed the title
| @@ -11,6 +15,7 @@ class Meta: | |||
|
|
|||
| class BatchSerializer(serializers.ModelSerializer): | |||
| errors = ErrorSerializer(many=True, read_only=True) | |||
| progressions = serializers.SerializerMethodField() | |||
There was a problem hiding this comment.
Since we created a model for this field, maybe we need to do something like that https://www.django-rest-framework.org/api-guide/relations/#nested-relationships
But it adds complexity (and I know Valentin didn't like nested serializers).. maybe it would be better to create a new route
There was a problem hiding this comment.
Right, but I didn't know how to combine a SerializerMethodField and model Serializers... Do you know how we could do that?
There was a problem hiding this comment.
Changed to stop using SerializerMethodField
Jasopaum
changed the title
Jasopaum
force-pushed
the
jason/display-extract-load
branch
from
9d1791c to
b6981ee
Compare
| @@ -21,6 +42,40 @@ class Meta: | |||
| "completed_at": {"allow_null": True}, | |||
| } | |||
|
|
|||
| def get_progressions(self, obj) -> List[Tuple[str, Dict]]: | |||
There was a problem hiding this comment.
I think this is not used anymore: you use ProgressionSerializer and ResourceForProgressionSerializer instead
| progression_factory.create(batch=batch, resource=r1) | ||
| progression_factory.create(batch=batch, resource=r2) |
There was a problem hiding this comment.
can you define extracted/loaded/failed here instead of directly in the factory ? easier to understand
| progression_factory.create(batch=batch, resource=r1) | ||
| progression_factory.create(batch=batch, resource=r2) |
|
|
||
| response_get = api_client.get(url) | ||
| assert response_get.json()["canceled_at"] is not None | ||
| print(response_get.json()["progressions"]) |
| extracted = 100 | ||
| loaded = 50 | ||
| failed = None |
TODO: front
Fixes
Fixes #668 by @MiskoG
Description
progressionsfield toBatchTechnical details
If the batch is running, the progression is fetched from redis. Otherwise, it's persisted and fetched from the DB
Definition of Done