Code

Microbatch parallelism#10958

Code

Merged

MichelleArk merged 44 commits intomaindbt-labs/dbt-core:mainfrom microbatch-parallelismdbt-labs/dbt-core:microbatch-parallelismCopy head branch name to clipboardNov 21, 2024

Commits

Commits on Oct 30, 2024

• first pass: microbatch parallelism: results + build command not working

  MichelleArk

  committed
  2314f5e

  Copy full SHA for 2314f5e

Commits on Oct 31, 2024

• merge batch-level results + log final result

  MichelleArk

  committed
  34fdcf3

  Copy full SHA for 34fdcf3

Commits on Nov 1, 2024

• Merge branch 'main' into microbatch-parallelism

  MichelleArk

  committed
  330ca88

  Copy full SHA for 330ca88

Commits on Nov 8, 2024

• unit test for MicrobatchModelRunner: remove result builder test as it will need to be re-written with new structure

  MichelleArk

  committed
  8cbd2c6

  Copy full SHA for 8cbd2c6

• move relation_exists to MicrobatchModelRunner

  MichelleArk

  committed
  b3bc0b4

  Copy full SHA for b3bc0b4

• move batches to MicrobatchModelRunner

  MichelleArk

  committed
  b3317c5

  Copy full SHA for b3317c5

• avoid adding microbatch results to run_results twice

  MichelleArk

  committed
  c658be8

  Copy full SHA for c658be8

• propagate previously successful batches to final result on retry

  MichelleArk

  committed
  9e56650

  Copy full SHA for 9e56650

• simplify MicrobatchModelRunner state

  MichelleArk

  committed
  92b9aa2

  Copy full SHA for 92b9aa2

• add tracking

  MichelleArk

  committed
  3d09292

  Copy full SHA for 3d09292

• order batch_results prior to returning

  MichelleArk

  committed
  801aede

  Copy full SHA for 801aede

Commits on Nov 11, 2024

• Merge branch 'main' into microbatch-parallelism

  MichelleArk

  committed
  f6c3860

  Copy full SHA for f6c3860

• refactor: remove batch_callback in favor of batch_results.append

  MichelleArk

  committed
  8b9e669

  Copy full SHA for 8b9e669

• refactor checkpoint: unify sequential and concurrent while loop in microbatch handling, TestMicrobatchWithInputWithoutEventTime failing

  MichelleArk

  committed
  71bd3d7

  Copy full SHA for 71bd3d7

Commits on Nov 14, 2024

• Merge branch 'main' into microbatch-parallelism

  MichelleArk

  committed
  69c088d

  Copy full SHA for 69c088d

Commits on Nov 15, 2024

• refactor into RunTask.handle_microbatch_model

  MichelleArk

  committed
  109fbd4

  Copy full SHA for 109fbd4

• Use RunTask.handle_microbatch_model in BuildTask

  MichelleArk

  committed
  62f93ab

  Copy full SHA for 62f93ab

• Run batches in parallel dependent on if this is in jinja

  QMalcolm

  committed
  69df6a2

  Copy full SHA for 69df6a2

• Ensure KeyboardInterrupts/SystemExits get handled in call_runner

  QMalcolm

  committed
  2c37abc

  Copy full SHA for 2c37abc

Commits on Nov 18, 2024

• Switch to integration test for checking that KeyboardInterrupts stop batch execution

  Show description for f1cea30
  QMalcolm

  committed
  f1cea30

  Copy full SHA for f1cea30

• Add optional parallel_batches to node config

  QMalcolm

  committed
  edaeb4c

  Copy full SHA for edaeb4c

• Regenerate v12 manifest.json to include parallel_batches config

  QMalcolm

  committed
  d21fdda

  Copy full SHA for d21fdda

• Validate parallel_batches config value in the context of a microbatch model

  QMalcolm

  committed
  05cfdb5

  Copy full SHA for 05cfdb5

• Use combination of relation existence, parallel_batches, usage of this to determine batch parallelism

  Show description for b2f2c88
  QMalcolm

  committed
  b2f2c88

  Copy full SHA for b2f2c88

• Move _should_run_in_parallel onto MicrobatchModelRunner

  QMalcolm

  committed
  4047b1c

  Copy full SHA for 4047b1c

• Add unit test for all _should_run_in_parallel cases

  QMalcolm

  committed
  281a8d8

  Copy full SHA for 281a8d8

• Add integration test to ensure microbatch batches can be run in parallel or sequentially

  QMalcolm

  committed
  c716a2c

  Copy full SHA for c716a2c

• Add changelog

  QMalcolm

  committed
  c3d2917

  Copy full SHA for c3d2917

• Rename parallel_batches to concurrent_batches

  QMalcolm

  committed
  422cd8b

  Copy full SHA for 422cd8b

Commits on Nov 19, 2024

• fix unit tests: ensure _has_this is not available in flat_graph

  MichelleArk

  committed
  7cd4599

  Copy full SHA for 7cd4599

• fix function artifacts and list tests

  MichelleArk

  committed
  5166379

  Copy full SHA for 5166379

• only run in parallel if adapter supports MicrobatchConcurrency capability

  MichelleArk

  committed
  59b9889

  Copy full SHA for 59b9889

Commits on Nov 20, 2024

• Merge branch 'main' into microbatch-parallelism

  MichelleArk

  committed
  1afad96

  Copy full SHA for 1afad96

• dbt-adapters dev-requirements @microbatch-concurrency-capability

  MichelleArk

  committed
  8bf1504

  Copy full SHA for 8bf1504

• Update test of _should_run_in_parallel to consider adapter microbatch_concurrency

  QMalcolm

  committed
  743715f

  Copy full SHA for 743715f

• Revert "dbt-adapters dev-requirements @microbatch-concurrency-capability"

  Show description for 9779a41
  QMalcolm

  committed
  9779a41

  Copy full SHA for 9779a41

• Remove threading information from batch info logs

  QMalcolm

  committed
  1ef88b3

  Copy full SHA for 1ef88b3

• Add MicrobatchExecutionDebug event type

  QMalcolm

  committed
  f127b7d

  Copy full SHA for f127b7d

• Log whether a batch will be run in parallel/concurrently or not

  QMalcolm

  committed
  7b3e54a

  Copy full SHA for 7b3e54a

Commits on Nov 21, 2024

• Move _build_run_microbatch_model_result to MicrobatchModelRunner

  QMalcolm

  committed
  3981e7b

  Copy full SHA for 3981e7b

• empty commit to retrigger full suite of github actions

  QMalcolm

  committed
  12db4ec

  Copy full SHA for 12db4ec

• fix core_types_pb2.py to version 28.3

  MichelleArk

  committed
  daa713e

  Copy full SHA for daa713e

• Bump dbt-adapters minimum to 1.10.1

  QMalcolm

  committed
  bb5e267

  Copy full SHA for bb5e267

• safer .name access

  MichelleArk

  committed
  b78f251

  Copy full SHA for b78f251