CI: add stateful streaming tests for C++ and Java#48
Open
zhanglistar wants to merge 31 commits into
Open
Conversation
Add StatefulStreamingTest.cc (C++) and StatefulStreamingTest.java (Java) covering: - Basic streaming data flow through StatefulTask::next() - Multiple batch consumption from Values node - Checkpoint snapshot/notifyCheckpointComplete/notifyCheckpointAborted - Mid-stream snapshot before task finishes - Watermark notification C++ tests use StatefulTask directly with StatefulPlanNode wrapping. Java tests use SerialTask.statefulGet() through the JNI bridge. Note: C++ tests pass individually but may crash when run together due to velox MemoryManager pool lifecycle management in the test fixture. Java tests all pass.
zhanglistar
force-pushed
the
codex/stateful-streaming-tests
branch
from
c795958 to
ae1aaf6
Compare
Fixes the CMake 'tools' target conflict that occurs when both RocksDB and gRPC are built BUNDLED (each creates an add_custom_target named 'tools'). By pre-installing these as system libraries in /usr/local, Velox's AUTO dependency resolution will find them via find_package and skip the BUNDLED FetchContent build entirely. Also cleans up all build artifacts and downloaded sources to minimize image size.
The original Velox grpc.cmake uses 'git apply' for PATCH_COMMAND, which silently fails because FetchContent with URL downloads a tarball (not a git repo). This leaves the 'tools' target unrenamed, causing a conflict with RocksDB's 'tools' target. The override uses 'patch -p1 -i' which works on tarball sources.
Velox's setup-ubuntu.sh installs fmt, folly, boost, protobuf, thrift, arrow, geos, stemmer, duckdb, librdkafka, cppkafka, fizz, wangle, mvfst, fbthrift into /usr/local. But it does NOT install: c-ares, double-conversion, xsimd, simdjson, absl, re2, gRPC, RocksDB, GTest. These were being built via FetchContent on every CI run, causing: 1. Slow CI (recompiling deps each time) 2. RocksDB/gRPC 'tools' target conflict Now all deps are pre-installed in the Docker image so Velox's AUTO resolution finds them via find_package() and skips BUNDLED entirely. Dep upgrades only require rebuilding the Docker image.
zhanglistar
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add StatefulStreamingTest.cc (C++) and StatefulStreamingTest.java (Java) covering:
C++ tests use StatefulTask directly with StatefulPlanNode wrapping. Java tests use SerialTask.statefulGet() through the JNI bridge.
Note: C++ tests pass individually but may crash when run together due to velox MemoryManager pool lifecycle management in the test fixture. Java tests all pass.