MySQL: Track session-specific system variables changes

[wazir-ahmed]

• Configure session_track_system_variables and session_track_state_change in all backend connections.
• Utilize notifications from backend servers to capture system variable changes that cannot be handled by MySQL_Set_Stmt_Parser
• Update both client and server variable maps based on backend responses.

[gemini-code-assist]

Summary of Changes

Hello @wazir-ahmed, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces comprehensive support for tracking session-specific system variable changes within MySQL backend connections. It establishes a robust mechanism to configure the necessary session tracking settings, process notifications from the backend when variables change, and synchronize these changes across the internal client and server variable states. This enhancement improves the accuracy of ProxySQL's understanding of the backend session state, particularly for variables that are not easily parsed from SQL statements.

Highlights

• Session Variable Tracking Configuration: Implemented logic to configure session_track_system_variables and session_track_state_change on all backend connections to enable server-side tracking of session variable changes.
• Backend Notification Processing: Added mechanisms to utilize notifications from backend servers to capture system variable changes that cannot be reliably handled by statement parsing alone.
• Variable Map Synchronization: Ensured that both client and server internal variable maps are accurately updated based on the session variable changes communicated from the backend responses, maintaining state consistency.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces tracking for session-specific system variables in MySQL by leveraging session_track_system_variables and session_track_state_change. The changes include new handlers and logic to set and process these variables, ensuring that changes are captured and synchronized between the client and server. My review focuses on improving code safety, clarity, and performance in the new implementations. I've suggested using safer string comparison functions, reducing variable scopes for better readability, and making minor performance optimizations in string handling. I also pointed out a commented-out line that should be removed.

[renecannao]

PR 5166: Comprehensive Review and Concerns - Detailed Analysis

First, excellent work on implementing session variable tracking! This is a valuable enhancement that addresses real limitations in ProxySQL's session state management. The architectural approach leveraging MySQL's native session tracking capabilities is sound and well-designed.

However, I have identified several critical issues and areas for improvement that should be addressed before merge. This analysis provides detailed technical background for each concern.

🔴 Critical Issues (Blockers)

1. Missing Capability Detection - Protocol Level Compatibility Gap

Problem: The implementation lacks proper capability detection for CLIENT_SESSION_TRACK, unlike the existing GTID tracking.

Technical Background:
In MySQL's client/server protocol, both client and server exchange capability flags during connection handshake. CLIENT_SESSION_TRACK is a capability flag that indicates the server supports MySQL 5.7+ session tracking features. When this flag is not set:

• Server does not track session variable changes
• Server cannot send session tracking information in OK packets
• Attempting to SET session_track_system_variables will likely fail
• MySQL client library functions like mysql_session_track_get_first() will not work

Current Implementation Gap:

// ✅ GTID tracking correctly checks capability:
bool MySQL_Session::handler_again___verify_backend_session_track_gtids() {
    if ((mybe->server_myds->myconn->mysql->server_capabilities & CLIENT_SESSION_TRACK) == 0) {
        // the backend doesn't support CLIENT_SESSION_TRACK
        return ret; // Gracefully handles unsupported backends
    }
    // ... proceed with configuration
}

// ❌ Variable tracking attempts SET commands without capability check:
bool MySQL_Session::handler_again___verify_backend_session_track_variables() {
    if (mysql_thread___session_track_variables == session_track_variables::DISABLED) {
        return false;
    }
    // ❌ No CLIENT_SESSION_TRACK check - will try to SET on unsupported servers!
    if (mybe->server_myds->myconn->options.session_track_variables_sent == false) {
        // This will fail on MySQL < 5.7, MariaDB, SQLite3, etc.
        NEXT_IMMEDIATE_NEW(SETTING_SESSION_TRACK_VARIABLES);
    }
}

Impact: This will cause connection failures on:

• MySQL < 5.7: Session tracking was introduced in MySQL 5.7.4
• MariaDB < 10.2: MariaDB implemented this later than MySQL
• ProxySQL's own admin interface: Uses SQLite3 backend which doesn't support MySQL protocol extensions
• sqlite3-server module: SQLite3 backend in ProxySQL
• Any non-MySQL backend that uses the MySQL Protocol

Fix Required: Add the same capability check that GTID tracking uses before attempting any SET commands.

2. CLIENT_DEPRECATE_EOF Dependency Not Handled

Technical Background:
MySQL session tracking requires two capability flags to work properly:

1. CLIENT_SESSION_TRACK : Enables session tracking protocol support
2. CLIENT_DEPRECATE_EOF : Replaces EOF packets with OK packets, required for session tracking data

When CLIENT_DEPRECATE_EOF is not set, MySQL server sends EOF packets instead of OK packets with session tracking data. Session tracking information is only transmitted in OK packets, not EOF packets.

Current State: The implementation assumes backends support session tracking without verifying both required capabilities.

Impact: Even if a server supports CLIENT_SESSION_TRACK, without CLIENT_DEPRECATE_EOF the session tracking data cannot be transmitted to the client. I actually expect that CLIENT_DEPRECATE_EOF is a requirement for CLIENT_SESSION_TRACK.

🟡 Major Concerns

3. TAP Test Modifications Need Explanation

Observation: Two TAP tests were modified to explicitly disable session tracking:

// test_binlog_reader-t.cpp:
MYSQL_QUERY_T(proxysql_admin, "SET mysql-session_track_variables=0");
MYSQL_QUERY_T(proxysql_admin, "LOAD MYSQL VARIABLES TO RUNTIME");

// reg_test_4264-commit_rollback-t.cpp:
MYSQL_QUERY_T(admin, "SET mysql-session_track_variables=0");

Possible Root Causes:

• Test Failures: Tests likely fail when session tracking is enabled due to the capability issues above
• State Pollution: The SET commands might affect test state in unexpected ways
• Variable Behavior Changes: Session tracking changes how variables are managed, potentially breaking test assumptions

Action Required: We need to understand the exact failure modes before merge. These test modifications suggest issues were encountered but worked around them instead of fixing the root cause.

4. Fast Forward Mode Compatibility

Technical Background:
Fast forward mode in ProxySQL provides high-performance query forwarding with minimal protocol overhead. It's designed for scenarios where ProxySQL acts as a transparent proxy with minimal processing.

Session Tracking Impact:

• Packet Format Changes: Session tracking adds data to OK packets, changing the protocol stream
• Client Capability Mismatch: Fast forward may assign connections to clients that don't support session tracking

Current Gap: No checking of client capabilities when assigning connections in fast forward mode.

Impact: Clients without session tracking support may receive packets they cannot parse or unexpected protocol overhead.

🔵 Enhancement Opportunities

5. Enhanced Configuration Modes (Production-Ready Design)

Current Limitation: Binary enable/disable (0/1) is too simplistic for production environments with mixed backends.

Proposed Three-Tier Configuration:

enum mode {
    DISABLED = 0,  // Current behavior: feature completely disabled
    OPTIONAL = 1,  // Graceful fallback with logging for mixed environments
    ENFORCED = 2   // Strict mode: fail if requirements cannot be met
};

Detailed Mode Behavior:

OPTIONAL Mode (1):

• Attempts to enable session tracking on supported backends
• Logs INFO message when backend doesn't support session tracking
• Logs WARNING if mysql-enable_server_deprecate_eof is false (required capability)
• Continues gracefully without session tracking on unsupported backends
• Use Case: Gradual rollout in mixed-version environments

ENFORCED Mode (2):

• Automatically enables CLIENT_DEPRECATE_EOF regardless of mysql-enable_server_deprecate_eof setting
• Fails connection if session_track_system_variables or session_track_state_change cannot be set
• Logs ERROR and drops backend connection if requirements not met
• Use Case: Production environments requiring consistent behavior

6. Frontend Variable Propagation (Complete Implementation)

Current Gap: The implementation tracks variables internally for ProxySQL's state management but doesn't propagate changes to clients that support session tracking.

Technical Requirements:

1. Client Capability Detection: Check if client connection has CLIENT_SESSION_TRACK flag
2. OK Packet Enhancement: Add session tracking data to OK packets sent to supporting clients

Implementation Complexity:

• OK packet format changes require careful protocol handling
• Need to buffer session tracking data from backend and forward to client
• Must handle cases where client and backend support different levels of tracking
• session_track_system_variables or session_track_state_change themselves should become tracked variables

📋 Required Testing

7. Comprehensive CI Testing with Feature Enabled

Current State: Tests run with feature disabled (see TAP test modifications above)

Required Testing Matrix:

A. Version Compatibility with mysql-session_track_variables=1 and mysql-session_track_variables=2:

• MySQL 5.6 (no session tracking support)
• MySQL 8.0+ (full session tracking support)

B. Protocol-Level Testing:

• OK packet format validation
• Client compatibility verification
• Error handling for unsupported backends

🎯 Recommended Implementation Order

Phase 1 - Critical Fixes (Merge Blockers)

1. Add CLIENT_SESSION_TRACKING capability detection before attempting SET commands
2. Add CLIENT_DEPRECATE_EOF dependency handling
3. Fix TAP test failures - understand root causes instead of disabling feature
4. Add comprehensive error handling for unsupported backends

Phase 2 - Compatibility Hardening

1. Fast forward mode compatibility checks
2. Comprehensive CI testing with feature enabled
3. Performance baseline measurement

Phase 3 - Production Enhancements

1. Implement 0/1/2 configuration modes
2. Frontend variable propagation
3. Performance optimization for high-frequency workloads
4. Add metrics and monitoring for session tracking usage

📝 Additional Technical Implementation Details

Error Handling Strategy Example:

bool MySQL_Session::handler_again___verify_backend_session_track_variables() {
    if (mysql_thread___session_track_variables == session_track_variables::DISABLED) {
        return false;
    }

    // Check if backend supports session tracking
    if ((mybe->server_myds->myconn->mysql->server_capabilities & CLIENT_SESSION_TRACK) == 0) {
        if (mysql_thread___session_track_variables == ENFORCED) {
            proxy_error("Backend %s:%d does not support required session tracking capabilities (CLIENT_SESSION_TRACK)",
                       mybe->host, mybe->port);
            return true; // Trigger connection failure
        } else if (mysql_thread___session_track_variables == OPTIONAL) {
            proxy_info("Backend %s:%d does not support session tracking, continuing without it",
                      mybe->host, mybe->port);
            return false; // Skip session tracking setup
        }
        return false;
    }

    // Check CLIENT_DEPRECATE_EOF requirement
    if ((mybe->server_myds->myconn->mysql->server_capabilities & CLIENT_DEPRECATE_EOF) == 0) {
        if (mysql_thread___session_track_variables == ENFORCED) {
            proxy_error("Backend %s:%d does not support CLIENT_DEPRECATE_EOF required for session tracking",
                       mybe->host, mybe->port);
            return true; // Trigger connection failure
        } else if (mysql_thread___session_track_variables == OPTIONAL) {
            proxy_info("Backend %s:%d does not support CLIENT_DEPRECATE_EOF, session tracking will be limited",
                      mybe->host, mybe->port);
            // Could proceed with limited functionality or skip entirely
        }
    }

    // Proceed with session tracking setup...
}

Summary

This PR implements valuable functionality that addresses real production needs. However, the current implementation has critical gaps that could cause connection failures in production environments.

Critical Issues Summary:

1. Protocol Capability Gap: Missing CLIENT_SESSION_TRACK detection will cause failures on older MySQL/MariaDB versions
2. Dependency Gap: Missing CLIENT_DEPRECATE_EOF handling

Recommendation:

• Address critical issues first before merge consideration
• Implement enhanced configuration modes for production robustness
• Add comprehensive testing across different MySQL versions and configurations
• Consider Phase 2 enhancements (frontend propagation) as follow-up PRs

The architectural foundation is solid and the approach is sound - we just need to harden the implementation for production use across diverse backend environments.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[renecannao]

The variable may have been changed after connections were established, thus this check is unnecessary.
Also, we would implement a logic that if session_track_variables is enabled, enable_server_deprecate_eof is ignored and CLIENT_DEPRECATE_EOF is always enabled for backend connections

[renecannao]

Yeah, that is what I mean in my previous comment:
"f session_track_variables is enabled, enable_server_deprecate_eof is ignored and CLIENT_DEPRECATE_EOF is always enabled for backend connections"

[renecannao]

We should also add CLIENT_SESSION_TRACK

[wazir-ahmed]

CLIENT_SESSION_TRACK is set by mariadb-client, by default. It is part of CLIENT_CAPABILITIES macro.