Bug Report: UnicodeDecodeError when limit_bytes truncates UTF-8 mid-character #2756

Summary

When using read_namespaced_pod_log() with _preload_content=True and limit_bytes parameter, the client raises UnicodeDecodeError if the byte limit cuts a multi-byte UTF-8 character in the middle.

Environment

• kubernetes-client version: 36.0.3 (also occurs in versions < 36.0.0)
• Python version: 3.12
• Operating System: Linux

Problem Description

The limit_bytes parameter works at the byte level and can cut a multi-byte UTF-8 character mid-sequence. When _preload_content=True (the default), the client attempts to decode the truncated byte stream as UTF-8 in rest.py:232, which raises UnicodeDecodeError.

Root Cause

In kubernetes/client/rest.py, line 232:

r.data = r.data.decode('utf8')

This assumes:

1. The response is valid UTF-8
2. The byte stream is complete

However, the Kubernetes API /api/v1/namespaces/{namespace}/pods/{name}/log endpoint:

• Returns logs as a byte stream with no encoding specified (content-type: text/plain)
• The limitBytes parameter cuts at byte boundaries, not character boundaries
• Can therefore return incomplete UTF-8 sequences when the limit is reached

Evidence

Stack Trace

File "/opt/app-root/lib/python3.12/site-packages/kubernetes/client/rest.py", line 232, in request
    r.data = r.data.decode('utf8')
             ^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 10485759: unexpected end of data

Analysis

• Position 10485759: This is exactly 10MB - 1 byte (10485760 = 10 * 1024 * 1024)
• Byte 0xc2: Start of a 2-byte UTF-8 sequence (continuation byte missing)
• The error occurs precisely at the limit_bytes boundary

Full Call Stack

File "/opt/app-root/lib/python3.12/site-packages/kubernetes/client/api/core_v1_api.py", line 24264, in read_namespaced_pod_log
    return self.read_namespaced_pod_log_with_http_info(name, namespace, **kwargs)
File "/opt/app-root/lib/python3.12/site-packages/kubernetes/client/api/core_v1_api.py", line 24387, in read_namespaced_pod_log_with_http_info
    return self.api_client.call_api(...)
File "/opt/app-root/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method, ...)
File "/opt/app-root/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(...)
File "/opt/app-root/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url, ...)
File "/opt/app-root/lib/python3.12/site-packages/kubernetes/client/rest.py", line 244, in GET
    return self.request("GET", url, ...)
File "/opt/app-root/lib/python3.12/site-packages/kubernetes/client/rest.py", line 232, in request
    r.data = r.data.decode('utf8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 10485759: unexpected end of data

Reproduction

from kubernetes import client, config

config.load_kube_config()
v1 = client.CoreV1Api()

# This will fail if limit_bytes cuts a UTF-8 character mid-sequence
try:
    log = v1.read_namespaced_pod_log(
        name="pod-with-large-logs",
        namespace="default",
        limit_bytes=10 * 1024 * 1024,  # 10 MB
        _preload_content=True,  # Default
    )
except UnicodeDecodeError as e:
    print(f"Error at position {e.start}: {e.reason}")
    # Output: Error at position 10485759: 'utf-8' codec can't decode byte 0xc2

Proposed Solutions

Option 1: Handle Incomplete UTF-8 (Recommended)

Modify rest.py:232 to handle incomplete UTF-8 sequences:

# Current code:
r.data = r.data.decode('utf8')

# Proposed fix:
r.data = r.data.decode('utf8', errors='ignore')  # Or 'replace'

Or better, strip incomplete sequences at the end:

def safe_decode_utf8(data):
    """Decode UTF-8, handling incomplete sequences at the end."""
    try:
        return data.decode('utf8')
    except UnicodeDecodeError as e:
        # If error is at the end, try truncating incomplete sequence
        if e.start >= len(data) - 4:  # UTF-8 max 4 bytes
            return data[:e.start].decode('utf8', errors='strict')
        raise

Impact

This affects any application using:

• read_namespaced_pod_log() with limit_bytes
• Default _preload_content=True
• Pod logs containing non-ASCII characters (common in error messages, stack traces)

References

• Related issue: v36.0.0 regression fetching container logs #2596 (different manifestation of string handling in v36.x)
• Discussion: https://gitlab.cee.redhat.com/ccx/ccx-ocp-core/-/merge_requests/1146

Expected Behavior

The client should handle incomplete UTF-8 sequences gracefully, OR

Actual Behavior

The client raises an unhandled UnicodeDecodeError that propagates to the application.