Clp integration s3 cleanup #2

wraymo · 2025-03-17T18:14:40Z

Summary by CodeRabbit

New Features
- Introduces a new optional CLP connector for advanced data sourcing and querying capabilities.
- Enhances search functionality with a new library for efficient query execution.
- Provides configuration settings allowing users to enable or disable the connector as needed.
Chores
- Improved build configuration and dependency management to support the new features.
- Added new CMake files for managing dependencies and integrating libraries.

coderabbitai · 2025-03-17T18:14:47Z

Walkthrough

This pull request introduces several new components and updates across the repository. An exception is added in the .gitignore, and a new submodule is defined in .gitmodules. Multiple CMake files are added to integrate external dependencies such as ANTLR4, msgpack-cxx, and spdlog. The main CMake configuration now includes an option to enable the CLP connector with its associated dependency management. The Makefile is updated to no longer treat warnings as errors. A comprehensive set of new source and header files is provided for the CLP connector and its search library, which includes new classes for data sourcing, query execution, and schema handling.

Changes

Files	Change Summary
`.gitignore`, `.gitmodules`	Added an exception in `.gitignore` to un-ignore `third-party/antlr/antlr-4.13.1-complete.jar`; added new submodule entry for `velox/connectors/clp/search_lib/clp_src` in `.gitmodules`.
`CMake/ExternalAntlr4Cpp.cmake`, `CMake/FindANTLR.cmake`, `CMake/resolve_dependency_modules/msgpack-cxx.cmake`, `CMake/resolve_dependency_modules/spdlog.cmake`	New CMake modules introduced for integration of external dependencies (ANTLR4 C++ runtime, msgpack-cxx, spdlog) and to define custom macros and dependency resolution logic.
`CMakeLists.txt`, `velox/connectors/CMakeLists.txt`	Updated to add the option `VELOX_ENABLE_CLP_CONNECTOR` with conditional logic to enable/disable the connector and resolve related dependencies (including spdlog, msgpack-cxx, absl); updated `BOOST_INCLUDE_LIBRARIES` to include `url`.
`Makefile`	Modified the `TREAT_WARNINGS_AS_ERRORS` variable from a conditional assignment (`?= 1`) to a fixed assignment (`= 0`), changing the treatment of warnings during build.
`velox/connectors/clp/...`	New files added for the CLP connector, including headers and source files for `ClpColumnHandle`, `ClpConfig`, `ClpConnector` (and its factory), `ClpConnectorSplit`, `ClpDataSource`, and `ClpTableHandle`, establishing the connector’s core functionality.
`velox/connectors/clp/search_lib/...`	New search library integrated via files such as `CMakeLists.txt`, `Cursor.cpp/.h`, `OrderedProjection.cpp/.h`, and `QueryRunner.cpp/.h` to manage query execution and data retrieval for the CLP connector.
`velox/connectors/clp/search_lib/clp_src`	New subproject commit reference added as a submodule, linking a specific state of the clp source code.

Sequence Diagram(s)

sequenceDiagram
    participant U as User/System
    participant CF as ClpConnectorFactory
    participant C as ClpConnector
    participant DS as ClpDataSource

    U->>CF: newConnector(id, config, executor)
    CF->>C: Instantiate ClpConnector with ClpConfig
    C->>U: Return new ClpConnector instance
    U->>C: Invoke createDataSource(...)
    C->>DS: Create and return ClpDataSource instance

sequenceDiagram
    participant DS as ClpDataSource
    participant Cur as Cursor
    participant QR as QueryRunner

    DS->>Cur: addSplit(split)
    Cur->>Cur: load_archive()
    Cur->>Cur: execute_query(query, output_columns)
    Cur->>QR: Pass query execution results
    QR->>Cur: Return data/results
    Cur->>DS: Provide fetched rows
    DS->>DS: next() returns RowVector with data

Tip

⚡🧪 Multi-step agentic review comment chat (experimental)

We're introducing multi-step agentic chat in review comments. This experimental feature enhances review discussions with the CodeRabbit agentic chat by enabling advanced interactions, including the ability to create pull requests directly from comments.
- To enable this feature, set early_access to true under in the settings.

✨ Finishing Touches

📝 Generate Docstrings

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 9

🧹 Nitpick comments (42)

CMake/FindANTLR.cmake (3)

18-18: Address the TODO regarding ANTLR cleanup.

You have a TODO comment here suggesting cleanup of the ANTLR CMake files. Would you like assistance to refactor or remove outdated parts? Taking action on this can help maintain clarity and reduce technical debt.

37-54: Consider adding a fallback for ANTLR version detection failures.

When the execute_process(...) invocation fails to detect the ANTLR version, you currently terminate with an error. It can be beneficial to provide a fallback or more descriptive error handling to guide users in correcting their environment setup.

55-161: Ensure the generated ANTLR outputs don’t clutter source directories.

The macro’s logic to generate source files in antlr4cpp_generated_src under CMAKE_CURRENT_BINARY_DIR is typically good practice. Still, be sure your .gitignore excludes or manages generated files. This helps avoid accidental staging of these files and keeps the source tree clean.

velox/connectors/clp/ClpTableHandle.cpp (2)

4-6: Consider whether a pass-through implementation is necessary.

The toString() method is currently just forwarding to the parent class implementation. If no specialized behavior is needed, consider whether this method needs to be explicitly implemented at all.

8-10: Consider whether a pass-through implementation is necessary.

Similarly, the serialize() method is just calling the parent class implementation. If no specialized behavior is required, this method may be unnecessary.

velox/connectors/clp/ClpColumnHandle.h (2)

27-29: Consider whether const member variables are appropriate for all fields.

Using const member variables ensures immutability, which is good for thread safety. However, it also means these values cannot be modified after construction. Ensure this restriction aligns with the intended usage of the class.

6-30: Consider adding serialization/deserialization support.

For connector components, serialization and deserialization capabilities are often needed. Consider adding methods like serialize() and a static deserialize() method if this class needs to be transferred over the network or persisted.
velox/connectors/clp/CMakeLists.txt (1)
3-4: Ensure SOURCE_PATH_SIZE definition is properly scoped

The SOURCE_PATH_SIZE definition is added globally using add_definitions(). Consider using target_compile_definitions() to scope this definition to only the targets that need it, rather than affecting the entire build.
-string(LENGTH "${CMAKE_SOURCE_DIR}/" SOURCE_PATH_SIZE)
-add_definitions("-DSOURCE_PATH_SIZE=${SOURCE_PATH_SIZE}")
+string(LENGTH "${CMAKE_SOURCE_DIR}/" SOURCE_PATH_SIZE)
+target_compile_definitions(velox_clp_connector PRIVATE 
+  "SOURCE_PATH_SIZE=${SOURCE_PATH_SIZE}")
velox/connectors/clp/search_lib/CMakeLists.txt (1)

14-152: Consider organizing source files into logical groups with comments.

The source file list is quite extensive and would benefit from organization into logical groups with comments explaining each group's purpose (e.g., readers, writers, parsers, etc.). This would enhance maintainability and make it easier for developers to understand the codebase structure.
CMakeLists.txt (3)
126-126: CLP connector enabled by default.

The CLP connector is enabled by default (VELOX_ENABLE_CLP_CONNECTOR is set to ON), which might increase build times and dependencies for users who don't need this connector. Consider setting it to OFF by default to follow the pattern of other optional connectors like S3, GCS, and ABFS.
-option(VELOX_ENABLE_CLP_CONNECTOR "Build CLP connector." ON)
+option(VELOX_ENABLE_CLP_CONNECTOR "Build CLP connector." OFF)
489-501: Inconsistent code formatting in conditional blocks.

The conditional blocks for CLP connector dependencies have inconsistent spacing. There's a space after "if" but before the opening parenthesis, which differs from the coding style used in the rest of the file.
-if (${VELOX_ENABLE_CLP_CONNECTOR})
+if(${VELOX_ENABLE_CLP_CONNECTOR})
Similar fix should be applied to the second conditional block as well.

489-496: Consider adding explicit version requirements for all external dependencies.

While spdlog has a version requirement (1.12.0), other dependencies like msgpack-cxx don't have explicit version requirements. This could lead to compatibility issues if different versions are used.
  velox_set_source(msgpack-cxx)
-  velox_resolve_dependency(msgpack-cxx)
+  velox_resolve_dependency(msgpack-cxx 4.1.1)
velox/connectors/clp/ClpTableHandle.h (2)
8-14: Inconsistent parameter handling in constructor.

The constructor takes tableName as a const reference but stores a copy, while query is taken as a shared_ptr and moved. Consider consistent parameter handling, especially if you're taking ownership of both parameters.
  ClpTableHandle(
      std::string connectorId,
-      const std::string& tableName,
+      std::string tableName,
      std::shared_ptr<std::string> query)
      : ConnectorTableHandle(connectorId),
-        tableName_(tableName),
+        tableName_(std::move(tableName)),
        query_(std::move(query)) {}
16-22: Unnecessary nodiscard attribute on accessor methods.

Using the [[nodiscard]] attribute on simple getters is unnecessary as they don't produce results that should be used. This attribute is more appropriate for operations that return a value that should not be ignored.
-  [[nodiscard]] const std::string& tableName() const {
+  const std::string& tableName() const {
    return tableName_;
  }

-  [[nodiscard]] const std::shared_ptr<std::string>& query() const {
+  const std::shared_ptr<std::string>& query() const {
    return query_;
  }
CMake/ExternalAntlr4Cpp.cmake (1)
135-138: Consider enabling C++ standard configuration.

The file has commented-out options for setting the C++ standard for ANTLR4. Consider uncommenting one of these lines to ensure compatibility with the project's C++ standard (C++17).
            -DCMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE}
            -DWITH_STATIC_CRT:BOOL=${ANTLR4_WITH_STATIC_CRT}
            -DDISABLE_WARNINGS:BOOL=ON
            # -DCMAKE_CXX_STANDARD:STRING=17 # if desired, compile the runtime with a different C++ standard
-            # -DCMAKE_CXX_STANDARD:STRING=${CMAKE_CXX_STANDARD} # alternatively, compile the runtime with the same C++ standard as the outer project
+            -DCMAKE_CXX_STANDARD:STRING=${CMAKE_CXX_STANDARD} # compile the runtime with the same C++ standard as the outer project
            INSTALL_COMMAND ""
            EXCLUDE_FROM_ALL 1)
velox/connectors/clp/ClpConnector.cpp (3)

5-11: Ensure executor pointer validity.
The constructor sets executor_ but does not verify its validity. Consider adding a check or defensive code in case executor is null.

13-26: Consider error handling for invalid column handles.
While this method currently relies on ClpDataSource for validation, a pre-check here could produce more targeted error messages for the caller.

28-34: Implement or track unimplemented data sink functionality.
createDataSink is currently unimplemented. Ensure you have a plan or placeholder test to track this missing functionality.

Do you want to create a stub that returns a descriptive error to alert calling code?

velox/connectors/clp/ClpDataSource.cpp (2)

15-92: Use consistent logging instead of std::cout.
Line 128 prints with std::cout, which may not align with Velox logging conventions. Consider replacing with VELOX_INFO, spdlog, or another consistent logging mechanism.

94-112: Validate split details further.
The logic branches on inputSource_ but does not safeguard archiveDir_ or archiveId. If they are empty or invalid, the cursor might not initialize properly.

velox/connectors/clp/search_lib/OrderedProjection.cpp (2)

32-61: Confirm column type coverage.
The switch statement handles the known ColumnType cases. If new types are introduced later, ensure they are also supported in this method to prevent runtime exceptions.

70-128: Improve diagnostic details for partial matches.
When a column partially matches a path and then fails later tokens, there is no detailed log or user-facing error. Consider logging or throwing a specialized error to aid debugging in complex schemas.

velox/connectors/clp/search_lib/Cursor.cpp (4)

16-18: Discourage usage of using namespace in the implementation file
While convenient, this can introduce name collisions or confusion about symbol origins in larger codebases. Consider using fully qualified names to improve maintainability and clarity.

75-75: Address the TODO comment
Line 75 references a TODO: fix it. Leaving it unresolved might create confusion for future maintainers. Would you like assistance in opening an issue to track this item or proposing an immediate fix?

153-174: Consider adding dedicated unit tests for query preprocessing
The parsing and logical transformations here are non-trivial. Adding targeted tests will enable early detection of syntax or logical errors and ensure robustness.

265-334: Refactor duplicated fetch_next methods
Both overloads share similar logic for cycling through archives and schemas. Consider introducing a common helper function or reorganizing the flow to reduce code duplication and simplify maintenance.

Also applies to: 336-405

velox/connectors/clp/ClpConfig.h (1)

1-1: Apply required formatting
The pipeline indicates formatting issues at line 1. Kindly run the project’s standard formatting tool (e.g., clang-format) to resolve these warnings.

🧰 Tools

🪛 GitHub Actions: Run Checks

[warning] 1-1: Found format issues. Please apply fix using the provided patch.

velox/connectors/clp/search_lib/OrderedProjection.h (2)

1-1: Run code formatting on file
The pipeline has flagged format issues. Running the consistent formatting tool will keep code style uniform and resolve any spacing or alignment discrepancies.

🧰 Tools

🪛 GitHub Actions: Run Checks

[warning] 1-1: Found format issues. Please apply fix using the provided patch.

11-19: Improve error messages in OperationFailed
Providing clearer error messages (e.g., including a descriptive string) facilitates easier troubleshooting. Consider augmenting the exception with contextual details about the operation that failed.

velox/connectors/clp/ClpDataSource.h (3)

31-40: Clarify concurrency behaviour for split addition.

The addSplit method is declared, but the concurrency model is unclear. If multiple threads may invoke addSplit, consider thread-safety measures (e.g. locking a mutex) to avoid race conditions.

42-53: Consider atomic or synchronized counters.

getCompletedBytes() and getCompletedRows() return totals from private members that could be updated concurrently if this class is used in a multi-threaded environment. Use atomic operations or appropriate locking for reliable concurrency support.

55-71: Use of Boost.Filesystem instead of std::filesystem.

C++17 and later provide <filesystem> in the standard library. Depending on your minimum compiler requirements, consider switching to std::filesystem to reduce external dependencies.

velox/connectors/clp/search_lib/Cursor.h (3)

1-16: Harmonize ingestion of standard and library headers.

Ensure that your inclusion of headers like <optional>, <vector>, and <string> is consistent across the codebase. This helps maintain clarity and consistency for future maintainers.

19-28: Extend error codes for thorough troubleshooting.

The ErrorCode enum might benefit from additional entries to capture different granular errors (e.g. network failures) if your system can encounter them. This would facilitate more refined error handling in the future.

30-35: Document the Field structure.

Consider including brief documentation for what each field member represents and how to interpret ColumnType. This helps clarify usage for new contributors or future references.

velox/connectors/clp/ClpConnector.h (3)

1-5: Apply code formatting to align with project standards.

The pipeline has flagged format issues in this file. Please run the recommended code style tool or apply the provided patch to resolve these warnings.

🧰 Tools

🪛 GitHub Actions: Run Checks

[warning] 1-1: Found format issues. Please apply fix using the provided patch.

14-18: Provide rationale for excluding dynamic filters.

canAddDynamicFilter() always returns false. If dynamic filters are valuable in the future, consider adding references or comments either explaining the limitation or pointing to JIRA tasks for potential implementation.

35-40: Ensure the data sink is implemented or intentionally left out.

createDataSink is declared but may not be fully implemented if CLP is read-only. If so, consider clarifying this with either a comment or a VELOX_NYI.

Would you like me to add a placeholder for createDataSink with a VELOX_NYI or a comment indicating that data insertion is not supported?

velox/connectors/clp/search_lib/QueryRunner.h (1)

78-80: Empty init method implementation.
It appears that this init method is intentionally left empty. If it is a placeholder, consider adding a TODO comment or removing it if it is no longer needed.
velox/connectors/clp/search_lib/QueryRunner.cpp (3)
108-127: Suspicious debugging code at line 118.
Inside the fetch_next method with VectorPtrs, the statement int a = 1; does nothing. This might be leftover debug code. Consider removing it to avoid confusion.
-        int a = 1;
278-352: Use false == ret instead of !ret for consistency with coding guidelines.
On line 344, the code uses !ret. The guidelines recommend preferring false == ret rather than !ret. Updating this would maintain style consistency.
-    ret = cur->is_inverted() ? !ret : ret;
+    ret = cur->is_inverted() ? (false == ret) : ret;
354-412: Use const reference to avoid copying map entries.
Within the loop on line 377 (for (auto entry : m_datestring_readers)), you are copying the map entries rather than referencing them. Using a const reference might improve efficiency and clarity.
-for (auto entry : m_datestring_readers) {
+for (auto const& entry : m_datestring_readers) {

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d2229ed and 4b74119.

⛔ Files ignored due to path filters (1)

third-party/antlr/antlr-4.13.1-complete.jar is excluded by !**/*.jar

📒 Files selected for processing (27)

.gitignore (1 hunks)
.gitmodules (1 hunks)
CMake/ExternalAntlr4Cpp.cmake (1 hunks)
CMake/FindANTLR.cmake (1 hunks)
CMake/resolve_dependency_modules/msgpack-cxx.cmake (1 hunks)
CMake/resolve_dependency_modules/spdlog.cmake (1 hunks)
CMakeLists.txt (4 hunks)
Makefile (1 hunks)
velox/connectors/CMakeLists.txt (1 hunks)
velox/connectors/clp/CMakeLists.txt (1 hunks)
velox/connectors/clp/ClpColumnHandle.h (1 hunks)
velox/connectors/clp/ClpConfig.h (1 hunks)
velox/connectors/clp/ClpConnector.cpp (1 hunks)
velox/connectors/clp/ClpConnector.h (1 hunks)
velox/connectors/clp/ClpConnectorSplit.h (1 hunks)
velox/connectors/clp/ClpDataSource.cpp (1 hunks)
velox/connectors/clp/ClpDataSource.h (1 hunks)
velox/connectors/clp/ClpTableHandle.cpp (1 hunks)
velox/connectors/clp/ClpTableHandle.h (1 hunks)
velox/connectors/clp/search_lib/CMakeLists.txt (1 hunks)
velox/connectors/clp/search_lib/Cursor.cpp (1 hunks)
velox/connectors/clp/search_lib/Cursor.h (1 hunks)
velox/connectors/clp/search_lib/OrderedProjection.cpp (1 hunks)
velox/connectors/clp/search_lib/OrderedProjection.h (1 hunks)
velox/connectors/clp/search_lib/QueryRunner.cpp (1 hunks)
velox/connectors/clp/search_lib/QueryRunner.h (1 hunks)
velox/connectors/clp/search_lib/clp_src (1 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

`**/*.{cpp,hpp,java,js,jsx,tpp,ts,tsx}`: - Prefer `false == ...

**/*.{cpp,hpp,java,js,jsx,tpp,ts,tsx}: - Prefer false == <expression> rather than !<expression>.

velox/connectors/clp/ClpTableHandle.cpp
velox/connectors/clp/ClpDataSource.cpp
velox/connectors/clp/search_lib/OrderedProjection.cpp
velox/connectors/clp/search_lib/Cursor.cpp
velox/connectors/clp/ClpConnector.cpp
velox/connectors/clp/search_lib/QueryRunner.cpp

🪛 GitHub Actions: Run Checks

velox/connectors/CMakeLists.txt