Fix to_json(lines=False) producing invalid JSON for multi-batch datasets#8314
Open
Functionhx wants to merge 1 commit into
Open
Fix to_json(lines=False) producing invalid JSON for multi-batch datasets#8314Functionhx wants to merge 1 commit into
Functionhx wants to merge 1 commit into
Conversation
When to_json(lines=False) is called on a dataset larger than the batch
size, each batch produced a complete JSON array (e.g. [{...}, {...}]),
and these were concatenated directly in _write, producing
[{...}][{...}] which is invalid JSON.
For orient="records" (the default), fix by stripping the outer [ and ]
from each batch output and wrapping all batch contents in a single
JSON array.
For other orient values where batching does not naturally produce
concatenable output, load the full dataset into a single DataFrame and
write once.
Remove the NotImplementedError guard that previously blocked this
combination because the underlying issue is now fixed.
Root cause: _batch_json calls pandas.DataFrame.to_json which produces
a self-contained JSON string, but _write blindly concatenated these
strings without accounting for the array structure.
Test Plan:
- Ran existing test suite: 18/18 tests pass (tests/io/test_json.py)
- Manually verified with dataset of 2500 rows exceeding batch_size
across orient="records", orient="split", and lines=True
regression. All produce valid, parseable JSON.
Signed-off-by: Yuchen Fan <functionhx@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7037.
to_json(lines=False) concatenated batch JSON arrays directly (like [...][...][...]), which is invalid JSON. Fixed by stripping outer brackets from each batch with orient=records and wrapping in a single array.