Ingest & Transform of stock data on Azure Blob Storage using Pyspark

Analytical ETL

Step 1

• Moving Avg

Step 2

• Previous day closing prices

Step 3

• Union Quote and Trade records

Step 4

• Last trade moving average

Step 5

• Filter for Quote records

Step 6

• Broadcast join

Track Job Status

Extract

• Succesful extract.py job run

End of day load

• Succesful EOD_load.py job run

Analytics

• Succesful analytical_ETL.py job run

Job status in PostgreSQL table

• Succesfully updated job status table for each job run