Class bases ETL for Glide, Emdat. #208
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Rup-Narayan-Rajbanshi
changed the base branch from
develop
to
feature/separate-historical-and-latest-extraction
Rup-Narayan-Rajbanshi
changed the base branch from
feature/separate-historical-and-latest-extraction
to
project/separate-historical-and-latest-extraction
Rup-Narayan-Rajbanshi
force-pushed
the
feature/apply-class-based-etl
branch
from
d115cb1 to
280a069
Compare
Rup-Narayan-Rajbanshi
force-pushed
the
project/separate-historical-and-latest-extraction
branch
from
37d6cc7 to
1dab3b0
Compare
Base automatically changed from
project/separate-historical-and-latest-extraction
to
develop
March 18, 2025 10:57
Rup-Narayan-Rajbanshi
force-pushed
the
feature/apply-class-based-etl
branch
from
e06c951 to
ecd874b
Compare
Rup-Narayan-Rajbanshi
changed the title
Rup-Narayan-Rajbanshi
force-pushed
the
feature/apply-class-based-etl
branch
from
ecd874b to
15f9e69
Compare
Rup-Narayan-Rajbanshi
force-pushed
the
feature/apply-class-based-etl
branch
2 times, most recently
from
2757a22 to
6a0945f
Compare
tnagorra
requested changes
Mar 20, 2025
| @@ -11,4 +11,4 @@ class Command(BaseCommand): | |||
| help = "Import data from EM-DAT" | |||
|
|
|||
| def handle(self, *args, **options): | |||
| ext_and_transform_emdat_historical_data.delay() | |||
| ext_and_transform_emdat_historical_data() | |||
There was a problem hiding this comment.
We removed delay here as it has chain(EMDATExtraction.task.s(QUERY, variables), EMDATTransformHandler.task.s()).apply_async() running internally which is asynchronous.
tnagorra
force-pushed
the
feature/apply-class-based-etl
branch
2 times, most recently
from
bb83195 to
bf5fb82
Compare
Rup-Narayan-Rajbanshi
force-pushed
the
feature/apply-class-based-etl
branch
3 times, most recently
from
f6b8170 to
9bb07c5
Compare
- Add start date from env variable for gdacs, glide, emdat, ifrc - Load pystac item with status equals to pending. - Set conrtab for idu historical data. - Fix glide url - Send data into pdc transformer in a temp file.
Rup-Narayan-Rajbanshi
force-pushed
the
feature/apply-class-based-etl
branch
3 times, most recently
from
9bc7f0f to
ae8ed22
Compare
- Re-format graphql query - Define query variables for emdat and glide - Simplify logic for glide task
tnagorra
approved these changes
Mar 20, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses
Depends on
Changes
This PR doesn't introduce any:
printThis PR contains valid: