[kernel-spark] Enable a few E2E dsv2 streaming tests and fix offset management edge cases

[zikangh]

🥞 Stacked PR

Use this link to review incremental changes.

• stack/integration2 [Files changed]
  • stack/reader [Files changed]
    • stack/lazy [Files changed]
      • stack/lazy2 [Files changed]
        • stack/snapshot [Files changed]

---

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

Add integration tests for kernel-spark DSv2 streaming and fix edge cases in offset handling.

Changes:

1. Fixed offset capping in getBatch(): Cap endVersion to the latest available snapshot version to handle offsets pointing to non-existent future versions (which can occur when
   buildOffsetFromIndexedFile advances to the next version).

2. Fixed starting version caching: Made getStartingVersion() synchronized and added caching to ensure idempotent behavior - startingVersion: "latest" must resolve to the same version across
   multiple calls.

3. Created DeltaSourceDSv2Suite: Runs a subset of existing DSv1 streaming tests using DSv2 to verify API compatibility for startingVersion options.

How was this patch tested?

End-to-end streaming tests.

Does this PR introduce any user-facing changes?

No.

[gengliangwang]

We can make it more efficient

private volatile Optional<Long> cachedStartingVersion = null;

Optional<Long> getStartingVersion() {
    Optional<Long> cached = cachedStartingVersion;  // Single volatile read
    if (cached != null) {
        return cached;
    }
    synchronized (this) {
        // Double-check after acquiring lock
        if (cachedStartingVersion != null) {
            return cachedStartingVersion;
        }
        // ... compute result ...
        cachedStartingVersion = result;
        return result;
    }
}

[zikangh]

Thank you. Perhaps this is overkill? We only call getStartingVersion() once per stream.

[gengliangwang]

yeah this is a minor comment. I am fine with the current implementation