Invalid Arrow data from JSONL

[lhoestq]

This code fails:

from datasets import Dataset

ds = Dataset.from_json(path_to_file)
ds.data.validate()

raises

ArrowInvalid: Column 2: In chunk 1: Invalid: Struct child array #3 invalid: Invalid: Length spanned by list offsets (4064) larger than values array (length 4063)

This causes many issues for @TevenLeScao:

• map fails because it fails to rewrite invalid arrow arrays

  ~/Desktop/hf/datasets/src/datasets/arrow_writer.py in write_examples_on_file(self)
      438             if all(isinstance(row[0][col], (pa.Array, pa.ChunkedArray)) for row in self.current_examples):
      439                 arrays = [row[0][col] for row in self.current_examples]
  --> 440                 batch_examples[col] = array_concat(arrays)
      441             else:
      442                 batch_examples[col] = [
  
  ~/Desktop/hf/datasets/src/datasets/table.py in array_concat(arrays)
  1885 
  1886     if not _is_extension_type(array_type):
  -> 1887         return pa.concat_arrays(arrays)
  1888 
  1889     def _offsets_concat(offsets):
  
  ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.concat_arrays()
  
  ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()
  
  ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
  
  ArrowIndexError: array slice would exceed array length

• to_dict() segfaults ⚠️

  /Users/runner/work/crossbow/crossbow/arrow/cpp/src/arrow/array/data.cc:99:  Check failed: (off) <= (length) Slice offset greater 
  than array length

To reproduce: unzip the archive and run the above code using sanity_oscar_en.jsonl
sanity_oscar_en.jsonl.zip

PS: reading using pandas and converting to Arrow works though (note that the dataset lives in RAM in this case):

ds = Dataset.from_pandas(pd.read_json(path_to_file, lines=True))
ds.data.validate()

[AngledLuffa]

Can I bump this? I am still running into the same crash, or a similar crash, a few years later:

>>> from datasets import load_dataset
>>> ds = load_dataset('oscar-corpus/community-oscar', data_files='data/2024-38/sd_meta/*.jsonl.zst', split='train', streaming=False)
data/2024-38/sd_meta/sd_meta.jsonl.zst: 100%|██████████████████████████████████████████████████████████████████████████████| 39.4M/39.4M [00:02<00:00, 14.2MB/s]
Generating train split: 0 examples [00:00, ? examples/s]
Traceback (most recent call last):
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/datasets/builder.py", line 1871, in _prepare_split_single
    writer.write_table(table)
    ~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/datasets/arrow_writer.py", line 642, in write_table
    pa_table = pa_table.combine_chunks()
  File "pyarrow/table.pxi", line 4557, in pyarrow.lib.Table.combine_chunks
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowIndexError: array slice would exceed array length

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    ds = load_dataset('oscar-corpus/community-oscar', data_files='data/2024-38/sd_meta/*.jsonl.zst', split='train', streaming=False)
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/datasets/load.py", line 2084, in load_dataset
    builder_instance.download_and_prepare(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        download_config=download_config,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        storage_options=storage_options,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/datasets/builder.py", line 925, in download_and_prepare
    self._download_and_prepare(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        dl_manager=dl_manager,
        ^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        **download_and_prepare_kwargs,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/datasets/builder.py", line 1001, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/datasets/builder.py", line 1742, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        gen_kwargs=gen_kwargs, job_id=job_id, **_prepare_split_args
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/datasets/builder.py", line 1898, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
>>>

[lhoestq]

This is likely an issue to report on pyarrow. Do you know what's causing this in the data ? Is there a malformed JSON somewhere or something like that ?

[AngledLuffa]

I don't know anything about the underlying causes, unfortunately. I just found this issue while trying to figure out why I couldn't use load_dataset on Oscar community crawl.