0.2.0a23

What's Changed

• fixes multiple unnecessary alter tables to add columns by @rudolfix in #163

Full Changelog: 0.2.0a22...0.2.0a23

0.2.0a22

dlt library changes

• DATE type is supported on all destinations
• table and column names will be shortened to fit into particular destination
• duckdb database will be created in the current working directory by default #148
• fixes connection problem on duckdb 0.7.1
• allows to configure naming conventions or adopt naming conventions preferred by a destination
• streamlit app generated by dlt pipeline ... show does not display deprecation warnings

PRs

• iterating on blog posts by @TyDunn in #144
• improves naming conventions by @rudolfix in #157
• Add support for date type by @steinitzu in #160
• fixes duckdb database default location by @rudolfix in #159

0.2.0.a21

What's Changed

• improves naming convention protocol by @rudolfix in #143

This is mostly a 🐛 bugfix release: names of flattened columns were incorrectly normalized
It also introduces improved interface to naming convention used by the schemas in pipelines (try Schema.naming)

Full Changelog: 0.2.0a20...0.2.0a21

0.2.0a20

What's Changed

• groups all config and secret values used by all sources from a single python module in a single section (https://github.com/dlt-hub/dlt/blob/devel/docs/technical/secrets_and_config.md#the-sources-section)
• allows to provide alternatives of credentials for sources with multiple auth methods (https://github.com/dlt-hub/dlt/blob/devel/docs/technical/secrets_and_config.md#working-with-alternatives-of-credentials-union-types)
• gives write access to schemas in the source decorated function to allow to generate/modify settings and tables (https://github.com/dlt-hub/dlt/blob/devel/docs/technical/working_with_schemas.md#schema-is-modified-in-the-source-function-body)
• all generators are closed if there's an exception in the pipe during extraction

Full Changelog: 0.2.0a19...0.2.0a20

0.2.0a19

What's Changed

• update 2023-02-16-dlthub-mission.md by @matthauskrzykowski in #131
• adding blog + mission post stub by @TyDunn in #130
• json/normalizer performance improvements and bugfixes by @rudolfix in #133
• adds technical documentation for contributors by @rudolfix in #139

New Contributors

• @matthauskrzykowski made their first contribution in #131

dlt library changes

• orjson is used instead of simplejson by default (with a fallback for pypy)
• spawn method now supported to run normalizer stage in parallel (so works on MacOS and Windows)
• indexes are not created by default on duckdb (inserting slow), but enabled in postgres
• bumped duckdb to 0.7.0 and newest dbt-duckdb
• bugfixes and additional tests

Full Changelog: 0.2.0a18...0.2.0a19

0.2.0a18

What's Changed

• small improvements to duckdb docs by @TyDunn in #128
• Fix current columns and new columns referencing the same object by @steinitzu in #127
• tests file rotation on schema changes + bumps duckdb to 0.7 by @rudolfix in #129

New Contributors

• @steinitzu made their first contribution in #127

0.2.0a17

What's Changed

• adds duckdb destination and dbt support by @rudolfix in #124
  The 🦆 db destination is added and may be used like any other destination. The multithreaded loading is quite fast thanks to duckdb dropping the GIL when called. See more in our docs
  We also support the dbt-duckdb adapter, the jaffle shop example was converted to duckdb to showcase this ability

• transaction support was added to sql_client and Big Query got multi-statement transactions via Sessions

Full Changelog: 0.2.0a16...0.2.0a17

0.2.0a16

What's Changed

• uses structured data types to store json by @rudolfix in #121
• brings back the functionality to run dbt packages by @rudolfix in #122

This update changes how the complex data type is being stored. Most often complex data type is generated to hold json data when nesting_level is limited in the source. The complex types are stored as JSONB (postgres), SUPER (redshift), JSON (BigQuery)

This update also allows to easily run dbt packages together with dlt pipelines. Most of the work went into making it user friendly and avoiding any dependency conflicts. This feature is also quite well tested. Take a look at those two examples:
https://github.com/dlt-hub/dlt/blob/devel/docs/examples/dbt_run_jaffle.py
https://github.com/dlt-hub/dlt/blob/devel/docs/examples/chess/chess_dbt.py
https://github.com/dlt-hub/dlt/tree/devel/docs/examples/chess/dbt_transform

Runing the dbt package is just two lines and you have more control over it that with the CLI

0.1.0a15

What's Changed

• improving docs UX by @TyDunn in #110
• docstings and cli improvements by @rudolfix in #114
• allows hash mismatch when loading schema that needs migration
• makes full_refresh flag configurable
• fixes config/secrets injection when union with base configuration is used, fixes secret hint detection
• properly restores pipeline state when extract step fails

0.2.0a10

What's Changed

• only specific default config values are generated in dlt init ie BigQuery Location
• correct postgres port generated
• sends traces to sentry if RUNTIME__SENTRY_DSN is present
• sends slack notification if RUNTIME__SLACK_INCOMING_HOOK is present