The Mara ELT Framework is a Python framework to build data pipelines.
It has been designed and build by Dr. Martin Loetzsch at Project A as a lightweight data transformation framework for their ventures. Today the Mara Packages are maintained and developed further by the Mara Open Source community.
Runnable apps that demonstrates how to build a data warehouse app with mara:
- Mara Example Project 1 - example ELT integrates publicly available e-commerce and marketing data into a more general modeling and structure for highlighting the capabilities of the Mara framework
- Mara Example Project 2 - example ELT integrates PyPi download stats and GitHub repo activitiy metrics into more general Python project activity stats
The core modules of the framework. You might not need all of them.
- mara-acl - adding basic permission management to the Mara Flask UI
- mara-app - a Mara Flask app
- mara-db - access and interact with databases
- mara-page - minimal API for defining Flask pages
- ⭐ mara-pipelines - define DAG pipelines and execute them
- mara-storage - access and manage Blob storage data
- ⭐ mara-schema - managing star schema entity models and use them
- mara-dbt - using the popular dbt engine inside a mara pipeline
- mara-markdown-docs - display markdown documentation in mara UI
- mara-etl-tools - Utilities for creating ETL pipelines in PostgreSQL with mara
- mara-data-explorer - a minimal Flask based UI for data analytics
- mara-metabase - a client integration for Metabase from Mara Schema
- mara-mondrian - a server/client integration for Mondrian Server with from Mara Schema
- mara-superset - a client integration for Apache Superset from Mara Schema
We suggest loading data via tools like Airbyte, Fivetran or Stitch, but there are some data loading packages available which are e.g. used in example projects:
- mara-olist-ecommerce-data - (demo) downloading sample data from Olist E-Commerce Data in PostgreSQL
- mara-google-sheet-downloader - Download Google Sheets to a DB table in a mara ETL
- project-a/mara-campaign-tree-editor - Flask based Mara UI for correcting wrong UTM parameters or changing campaign structure
- mara-cron - (experimental) managing crontab files from mara source code
- project-a/mara-prophet - a lightweight framework for producing, integrating and historizing Facebook-prophet forecasts for time series data in Mara
- leo-schick/mara-spark - a basic implementation of pyspark into a mara pipeline
- December 19, 2017 - Reducing pain in Data Engineering | Martin Loetzsch at YouTube | Slides
- April 08, 2018 - code.talks commerce 2018 - Data Warehousing with Python at YouTube
- October 11, 2018 - Open Source: Project A Releases Its Data Warehouse Infrastructure at Medium
- August 29, 2018 - Dr. Martin Loetzsch - ETL Patterns with Postgres at Youtube | Slides
- May 24, 2019 - Lampenwelt: How they used Mara to build a Data Warehouse at Medium
