fix: avoid full listings for cached pruned partitions

[kumarUjjawal]

Which issue does this PR close?

• Closes datasource listing performs full listing despite user pruning partitions #23341.

Rationale for this change

The list files cache currently performs a full table listing on a cold cache miss, even when partition pruning has already narrowed the scan to a specific path prefix.

This makes selective partition queries pay the cost of listing the full table unless users disable the cache.

What changes are included in this PR?

• List only the requested prefix on cold list-cache misses.
• Keep using full-table cache entries to serve later prefix queries.
• Ensure prefix-scoped cache entries do not satisfy full-table lookups.
• Invalidate matching prefix-scoped list-cache entries on table writes.
• Add regression tests that check the actual object store list prefix.

Are these changes tested?

Yes

Are there any user-facing changes?

Partition-pruned listing table queries with the list files cache enabled avoid full table listings on cold cache misses.
No API Change

[codecov-commenter]

Codecov Report

❌ Patch coverage is 86.60714% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.67%. Comparing base (1e58928) to head (5c8fcd3).

Files with missing lines	Patch %	Lines
datafusion/datasource/src/url.rs	88.99%	0 Missing and 12 partials ⚠️
datafusion/catalog-listing/src/table.rs	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23402      +/-   ##
==========================================
- Coverage   80.67%   80.67%   -0.01%     
==========================================
  Files        1086     1086              
  Lines      366800   366897      +97     
  Branches   366800   366897      +97     
==========================================
+ Hits       295912   295985      +73     
- Misses      53260    53267       +7     
- Partials    17628    17645      +17     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.