Skip to content

Commit cd76bd2

Browse files
committed
Fix result filename on metadata and tests
1 parent c727de4 commit cd76bd2

8 files changed

+945
-749
lines changed

airflow/plugins/operators/gtfs_csv_to_jsonl_operator.py

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ def metadata(self) -> dict:
6464
return {
6565
"csv_dialect": self.dialect,
6666
"extract_config": self.unzip_results.get("extract").get("config"),
67-
"filename": self.filename,
67+
"filename": f"{self.filetype()}.jsonl.gz",
6868
"gtfs_filename": self.filetype(),
6969
"num_lines": len(self.lines),
7070
"ts": self.current_date.isoformat(),
@@ -192,20 +192,18 @@ def execute(self, context: Context) -> str:
192192
},
193193
)
194194

195-
report = results.report()
196-
197195
self.gcs_hook().upload(
198196
bucket_name=self.destination_name(),
199197
object_name=os.path.join(
200198
f"{results.filename}_parsing_results", self.results_path_fragment
201199
),
202-
data=json.dumps(report, separators=(",", ":")),
200+
data=json.dumps(results.report(), separators=(",", ":")),
203201
mime_type="application/jsonl",
204202
gzip=False,
205203
metadata={
206204
"PARTITIONED_ARTIFACT_METADATA": json.dumps(
207205
{
208-
"filename": os.path.basename(self.results_path_fragment),
206+
"filename": "results.jsonl",
209207
"ts": dag_run.logical_date.isoformat(),
210208
}
211209
)

airflow/tests/operators/cassettes/test_gtfs_csv_to_jsonl_operator/TestGTFSCSVToJSONLOperator.test_execute.yaml

Lines changed: 267 additions & 144 deletions
Large diffs are not rendered by default.

airflow/tests/operators/cassettes/test_gtfs_csv_to_jsonl_operator/TestGTFSCSVToJSONLOperatorFileNotFound.test_execute.yaml

Lines changed: 0 additions & 150 deletions
This file was deleted.

0 commit comments

Comments
 (0)