Implement: EXPLAIN Query Feature

[jayendra13]

EXPLAIN Query Feature Plan

Saved for future implementation

Overview

Add transparent visibility into memory allocation and I/O operations for Zarr queries.

Current State

DataFusion natively supports EXPLAIN and EXPLAIN ANALYZE:

EXPLAIN SELECT * FROM table;           -- Shows logical and physical plan
EXPLAIN ANALYZE SELECT * FROM table;   -- Executes and shows runtime metrics

The existing ZarrExec::DisplayAs implementation (src/physical_plan/zarr_exec.rs:63-85) outputs:

ZarrExec: path=..., limit=X, filters=[...]

Proposed Enhancements

1. Enhance DisplayAs Output

Extend ZarrExec::fmt_as() to show:

• Projection ratio (e.g., projection=2/5 columns)
• Estimated I/O bytes
• Pushdown summary

File: src/physical_plan/zarr_exec.rs

2. Add QueryEstimate Struct

Pre-execution cost estimation without I/O:

#[derive(Debug, Clone)]
pub struct QueryEstimate {
    pub projected_columns: Vec<String>,
    pub total_columns: usize,
    pub filters: Vec<(String, String)>,
    pub estimated_selectivity: f64,
    pub limit: Option<usize>,
    pub estimated_rows: usize,
    pub estimated_coord_bytes: u64,
    pub estimated_data_bytes: u64,
    pub estimated_disk_bytes: u64,
}

File: src/reader/stats.rs

3. Add estimate_query Function

pub fn estimate_query(
    store_meta: &ZarrStoreMeta,
    schema: &SchemaRef,
    projection: Option<&[usize]>,
    limit: Option<usize>,
    coord_filters: Option<&CoordFilters>,
) -> QueryEstimate;

File: src/reader/zarr_reader.rs

4. Extend ZarrIoStats for EXPLAIN ANALYZE

Add formatting methods:

impl ZarrIoStats {
    pub fn format_explain_analyze(&self, row_count: usize, col_count: usize) -> String;
    pub fn compression_ratio(&self) -> f64;
    pub fn total_time(&self) -> Duration;
}

File: src/reader/stats.rs

Example Output

EXPLAIN (leverage DataFusion native)

ZarrExec: path=data/weather.zarr, projection=2/5, limit=10, filters=[time=1000]
  Estimated: 880 B memory, 440 B disk, selectivity=0.14%

EXPLAIN ANALYZE (post-execution)

ZarrExec: path=data/weather.zarr
  Actual I/O:
    Metadata:    1.5 KB in 3.2ms
    Coordinates: 1 array, 168 B in 2.1ms
    Data:        1 array, 800 B in 5.4ms
    Disk total:  1.2 KB (compression: 2.1x)
    Memory total: 2.5 KB
  Timing: 10.7ms total
  Result: 10 rows, 2 columns

Files to Modify

File	Changes
src/reader/stats.rs	Add QueryEstimate, formatting methods
src/reader/zarr_reader.rs	Add estimate_query()
src/physical_plan/zarr_exec.rs	Enhance DisplayAs

Existing Infrastructure

• ZarrIoStats (src/reader/stats.rs): Already tracks metadata_bytes, coord_bytes, data_bytes, disk_bytes, timing (nanos), array counts
• TrackedStore (src/reader/tracked_store.rs): Wraps storage to record actual disk I/O
• CLI stats display (src/bin/zarr_cli/main.rs): Shows "N arrays · X KB disk · Y KB mem · Zs"

Verification

CREATE EXTERNAL TABLE weather STORED AS ZARR LOCATION 'data/synthetic_v2.zarr';
EXPLAIN SELECT lat, temperature FROM weather WHERE time = 0 LIMIT 5;
EXPLAIN ANALYZE SELECT lat, temperature FROM weather WHERE time = 0 LIMIT 5;