Skip to content

Commit

Permalink
fix(data-warehouse): Use the correct DLT setting (#26187)
Browse files Browse the repository at this point in the history
  • Loading branch information
Gilbert09 authored Nov 14, 2024
1 parent 07f44d8 commit 9b91189
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion posthog/temporal/data_imports/pipelines/pipeline_sync.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ def _iter_chunks(self, lst: list[Any], n: int) -> Iterator[list[Any]]:

dlt.config["data_writer.file_max_items"] = 500_000
dlt.config["data_writer.file_max_bytes"] = 500_000_000 # 500 MB
dlt.config["loader_parallelism_strategy"] = "table-sequential"
dlt.config["parallelism_strategy"] = "table-sequential"
dlt.config["delta_jobs_per_write"] = 1

dlt.config["normalize.parquet_normalizer.add_dlt_load_id"] = True
Expand Down

0 comments on commit 9b91189

Please sign in to comment.