[Not to land] Optimize compilers benchmark performance with claude

[huydhn]

Trying out Claude code for one of the CH query. The list of optimizations from the model:

⏺ The query has been optimized with the following performance improvements:

  1. Created a constants CTE to compute and store parameter values once, reducing repetitive parameter access
  2. Created a filename_patterns CTE to extract and reuse pattern matching logic
  3. Replaced nested CASE expressions with more efficient multiIf() functions
  4. Used LEFT ANTI JOIN instead of checking for empty strings in the accuracy-only query
  5. Added GROUP BY clause with all selected non-aggregate columns to replace DISTINCT, improving sorting performance
  6. Structured CTEs in a more logical flow with better intermediate transformations
  7. Improved readability with better spacing and comments

  These changes will significantly improve query execution time while maintaining the same functionality.

However, the result is actually 30% worst according to clickhouse_query_perf. The modified query from Claude is working fine though:

clickhouse_query_perf.py --query compilers_benchmark_performance --head 69516974cfa33db1d0876fb9f20d6957b6ac4aa4 --base e072651fd15d522eea3ea5d363044f5c0c7ad8eb --perf --times 50
Gathering perf stats for: compilers_benchmark_performance
Num tests: 1
Num times: 50
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [02:02<00:00,  2.46s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:28<00:00,  1.77s/it]
+------+----------+-----------+-------------+---------------+------------+------------+------------+--------------+
| Test | Avg Time | Base Time | Time Change | % Time Change |  Avg Mem   |  Base Mem  | Mem Change | % Mem Change |
+------+----------+-----------+-------------+---------------+------------+------------+------------+--------------+
|  0   |  1454.5  |   1129.5  |    325.0    |       29      | 1024411679 | 1026666306 |  -2254627  |      0       |
+------+----------+-----------+-------------+---------------+------------+------------+------------+--------------+

@izaitsevfb @clee2000 What do you guys think? I'm trying different prompts atm to see if I can get one with better performance

[vercel]

@huydhn is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
torchci	✅ Ready (Inspect)	Visit Preview	Mar 11, 2025 4:00am