perf: prewarm listing statistics#2261
Draft
lonless9 wants to merge 11 commits into
Draft
Conversation
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #2261 +/- ##
==========================================
- Coverage 79.40% 78.12% -1.29%
==========================================
Files 916 916
Lines 174656 175047 +391
==========================================
- Hits 138687 136752 -1935
- Misses 35969 38295 +2326
*This pull request uses carry forward flags. Click here to find out more.
... and 65 files with indirect coverage changes 🚀 New features to boost your workflow:
|
Spark 3.5.7 Test ReportCommit Information
Test Summary
Test DetailsError CountsPassed Tests Diff(empty) Failed Tests |
Spark 4.1.1 Test ReportCommit Information
Test Summary
Test DetailsError CountsPassed Tests Diff(empty) Failed Tests(truncated) |
Ibis Test ReportCommit Information
Test Summary
Test DetailsError CountsPassed Tests Diff--- before.txt 2026-07-18 11:43:16.914287310 +0000
+++ after.txt 2026-07-18 11:43:17.139289033 +0000
@@ -531,0 +532 @@
+ibis/backends/tests/test_export.py::test_table_to_csv[pyspark]Failed Tests |
lonless9
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pre-warm reusable listing file statistics after a catalog table has been successfully created.
This moves Parquet statistics collection from the first query to
CREATE TABLE, while keepingTableSourcecreation free of pre-warming side effects. Statistics are stored in the session-scoped DataFusionRuntimeEnvcache, so they remain reusable when Sail recreates aListingTableSourceduring later Analyze or Execute requests.Background
While investigating ClickBench performance, the DataFusion community found that
collect_statistics=truecreated a statistics cache but did not populate it during listing table creation. Consequently, the first query—especiallyCOUNT(*)—paid the cost of reading Parquet metadata and deriving file statistics.This was reported in apache/datafusion#18952, tracked as part of the ClickBench performance work, and fixed in apache/datafusion#18971 by pre-warming statistics in
ListingTableFactory::create.Sail did not inherit that fix because lakehq/sail#1877 replaced DataFusion's
ListingTableprovider with Sail's ownListingTableSourceand physical planner.Integration point
DataFusion's default logical
TableSourcewraps a registered, long-livedTableProvider. ItsListingTableFactory::createtherefore represents a stable table-creation lifecycle hook.Sail deliberately separates the two planning layers:
ListingTableSourcedescribes the logical data source.ListingPhysicalPlannerperforms file listing, statistics collection, grouping, and physical scan construction.A source may be recreated for schema analysis, execution, or separate DataFrame actions. Pre-warming from
create_source()would therefore:spark.read.parquet(...)operations;This PR instead binds pre-warming to successful catalog registration and stores the reusable result in
RuntimeEnv, whose lifetime spans those source instances.Implementation
CatalogCommand::CreateTable.CREATE TABLEcommands.TableStatusreturned by the catalog.TableFormat::prewarm_statisticshook with a default no-op implementation.ListingTableFormat::create_source()free of statistics-prewarming side effects.SessionConfigand sharedRuntimeEnv, enumerate visible files, and populate per-file statistics and ordering.ObjectMeta.CREATE TABLE.Unlike DataFusion's implementation, the warmup does not run the entire scan preparation path. It collects reusable per-file metadata without performing file grouping or aggregate scan-statistics work that would immediately be discarded.
Trigger semantics
Pre-warming occurs for:
CREATE TABLE;It does not occur for:
CREATE TABLE IF NOT EXISTSwhen the table already exists;spark.read.parquet(...);CTAS intentionally does not pre-warm during its catalog precondition because the table is registered before output files are written. A future post-write hook would be the correct place to optimize that path.
Tests
Added unit coverage proving that:
create_source()does not pre-warm file statistics;ListingTableSourceare reused by a separately created source.The end-to-end Spark SQL path was also verified with: