You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ciphers/base32.py and ciphers/base85.py encode Unicode code points instead of bytes: output diverges from the spec for non-ASCII, round-trips corrupt chars >= U+0400, arbitrary bytes crash #15636
None needed — both modules only use the standard library.
Expected behavior
Base32 and Ascii85 are byte encodings: base32_encode(data) should return the RFC 4648 encoding of the input bytes (matching base64.b32encode), and ascii85_encode(data) should match base64.a85encode. They should accept arbitrary bytes, as the type hints (data: bytes) suggest. The sibling modules base16.py and base64_cipher.py do exactly this and match the standard library for every input.
Actual behavior
ciphers/base32.py and ciphers/base85.py start with data.decode("utf-8") and then encode Unicode code points (bin(ord(d))) instead of bytes. All three consequences show up as soon as the input is not pure ASCII:
Output diverges from the standard for non-ASCII text:
>>> from ciphers.base32 import base32_encode
>>> import base64
>>> base32_encode("héllo".encode())
b'NDUWY3DPA======='
>>> base64.b32encode("héllo".encode())
b'NDB2S3DMN4======' # correct RFC 4648 result
Encode→decode silently corrupts data for characters ≥ U+0400 (their code point is wider than 8 bits, shifting every following bit):
>>> from ciphers.base32 import base32_encode, base32_decode
>>> base32_decode(base32_encode("€100".encode()))
b'\xc2\x82\xc2\xb0\xc3\x84\xc3\x80\xc3\x80' # should round-trip to b'\xe2\x82\xac100'
Same for ascii85_decode(ascii85_encode("€100".encode())) → b'\xc2\x82\xc2\xb0\xc3\x84\xc3\x80\xc3\x80'.
Arbitrary non-UTF-8 bytes — perfectly valid input for a byte codec — raise UnicodeDecodeError:
>>> base32_encode(bytes([0xff, 0xfe, 0x01]))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0
The doctests only use ASCII strings, so the test suite never exercises any of this.
ciphers/base32.py:19 and ciphers/base85.py:25 both build the bit string from "".join(bin(ord(d))[2:].zfill(8) for d in data.decode("utf-8")). Operating on the raw bytes (bin(byte)[2:].zfill(8) for byte in data, like base16.py does) and assembling the decode side from byte values instead of chr(int(chunk, 2)) would keep both codecs on-spec and lossless for every input.
Repository commit
6e03a48
Python version (python --version)
Python 3.13.5
Dependencies version (pip freeze)
None needed — both modules only use the standard library.
Expected behavior
Base32 and Ascii85 are byte encodings:
base32_encode(data)should return the RFC 4648 encoding of the input bytes (matchingbase64.b32encode), andascii85_encode(data)should matchbase64.a85encode. They should accept arbitrary bytes, as the type hints (data: bytes) suggest. The sibling modulesbase16.pyandbase64_cipher.pydo exactly this and match the standard library for every input.Actual behavior
ciphers/base32.pyandciphers/base85.pystart withdata.decode("utf-8")and then encode Unicode code points (bin(ord(d))) instead of bytes. All three consequences show up as soon as the input is not pure ASCII:Same for
ascii85_decode(ascii85_encode("€100".encode()))→b'\xc2\x82\xc2\xb0\xc3\x84\xc3\x80\xc3\x80'.UnicodeDecodeError:The doctests only use ASCII strings, so the test suite never exercises any of this.
Steps to reproduce
Suspected cause
ciphers/base32.py:19andciphers/base85.py:25both build the bit string from"".join(bin(ord(d))[2:].zfill(8) for d in data.decode("utf-8")). Operating on the raw bytes (bin(byte)[2:].zfill(8) for byte in data, likebase16.pydoes) and assembling the decode side from byte values instead ofchr(int(chunk, 2))would keep both codecs on-spec and lossless for every input.