Conversation
* feat: complete CPU kernel migration from `parents` to `offsets` (#3988) * next step in kernel migration from parents to offsets * linter fix * add jax bincount * add cpp kernel to convert parents to offsets * fix typo * fix typo in an auto-generated test * almost there * nearly there * cleanup * fix remaining kernels * final cleanup * format json data * format * initialize maxnextparents = -1 (a sentinel meaning "no bin was touched") * update the kernels to work on offsets! * format * migrate cupy rawkernels from parents to offsets * migrate jax reducers * fix for platfroms where the int64 counts can't be safely cast * add bincount for cupy backend * compact loc * fix windows build and add optional OpenMP support * update cuda kernels * feat: add `missing_repeat` kernel implementation using cuda.compute (#3922) * feat: add missing_repeat implementation using cuda.compute * keep the same name as in cpu kernel * add tests for repetitions>1 and regularsize>1 * style: pre-commit fixes * add keywords * style fixes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat: add `index_rpad_and_clip` kernels implementations using cuda.compute (#3923) * feat: add `index_rpad_and_clip` kernels implementation using cuda.compute * style fix * add a test for `index_rpad_and_clip_axis0` that would have target>length * add keyword names * style * Apply suggestions from Ianna Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch> * style: pre-commit fixes --------- Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: array attrs not being validated at creation and being of inconsistent type (#3996) * convert dict attrs to Attrs type * add test * fix: ensure jax backend uses arrays on cpu only (#3990) * ensure jax uses cpu by default * remove now redundant jax_platform_name setting * change error messages * better errors * remove DeviceArray mentions as that does not exist while we're at it too * almost there * cleanup * cleanup * remove old data file * update cupy kernels * remove exlicit test for depricated kernel * add complex kernels and port them to use offsets * fix complex reducers * add remaining kernels * add kernels for complex and bool sum * move to segmented_reduce * fix typo * promote type * fix complex bool reducer * try another algo * missed one * use type inference * make numba happy * remove reducer overloads * use sum op func for complex types * remove test of depricated code * avoid using == to compare floating-point products * fix boundary tests * cleanup * cleanup * remove dead code * revert lexsort to argmin/max * handrolled lexsort * remove dead code * remove bincount * import numba cuda for jit * fix tests --------- Co-authored-by: maxymnaumchyk <maxymnaumchyk@gmail.com> Co-authored-by: maxymnaumchyk <70752300+maxymnaumchyk@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Iason Krommydas <iason.krom@gmail.com> * chore: profile: add benchmark script (#4073) add benchmark script * feat: add some of the kernels using cuda.compute (#3981) * feat: add `awkward_IndexedArray_overlay_mask` kernel using cuda.compute * feat: add `awkward_IndexedArray_reduce_next_64` kernel using cuda.compute * feat: add `IndexedArray_reduce_next_nonlocal_nextshifts_64` kernel using cuda.compute * feat: add `ByteMaskedArray_getitem_nextcarry` kernel using cuda.compute * feat: add `awkward_ByteMaskedArray_numnull` kernel using cuda.compute * feat: add `awkward_RegularArray_getitem_jagged_expand` kernel using cuda.compute * add an upper bound * style: pre-commit fixes * feat: add `awkward_RegularArray_getitem_jagged_expand` kernel using cuda.compute * feat: add `awkward_UnionArray_simplify_one` kernel using cuda.compute * feat: add `awkward_ListArray_broadcast_tooffsets` kernel using cuda.compute * feat: add `awkward_ListArray_localindex` kernel using cuda.compute * fix the impl * feat: add `awkward_ListArray_compact_offsets` kernel using cuda.compute * feat: add `awkward_ListArray_combinations_length` kernel using cuda.compute * feat: add `awkward_ListArray_combinations` kernel using cuda.compute * feat: add `awkward_UnionArray_nestedfill_tags_index` kernel using cuda.compute * fix the tests for kernels that are deliberately raising errors * compare `starts` and `stops` separately * ignore `memptr` argument for pylint * style: pre-commit fixes * Add functions for indexing and repeating arrays * style: pre-commit fixes * return unary_transform call for segment_ids * style: pre-commit fixes * update the `awkward_IndexedArray_reduce_next_64` to work with offsets * style: pre-commit fixes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch> * feat: migrate some more kernels to cuda.compute (#4019) * feat: add reimplementation of the `awkward_RegularArray` kernels using cuda.compute * reimplement more kernels using cuda.compute * reimplement more kernels using cuda.compute * style: pre-commit fixes * style fix * use the named functions instead of lambdas * add new kernels * add new kernels * fix a type mismatch * style: pre-commit fixes * add new kernels * fix the overflow error * new batch of kernels (draft) * add comments and cleanup * fix the tests messages * add new `BitMaskedArray` kernels * modify the `test_1381_check_errors.py` test * resolve the conflicts --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fable fixes based on the report * sort kernel issues addressed * add tests * add benchmarks * add benchmarks * fix memory leak * add benchmarks/cuda_compute_cache_leak_reproducer.py * fix ruff checks * synchronization * fix it * fix memory leak * more fixes * more tests * reduce temporaries * more tests * relax conditions * fix bench * bigfix * fix bug * relax condition * add kernel fusion with parrot * the same for more reducers * cleanup * apply __restrict__ uniformly * rename macro-generated ABI symbols to match the kernel name * add tests * add O2 build * fix * add max_segment_size * add benchmarks and cccl profiler * pin cccl and clean up * cleanup * add test * fix overflow int64 --------- Co-authored-by: maxymnaumchyk <maxymnaumchyk@gmail.com> Co-authored-by: maxymnaumchyk <70752300+maxymnaumchyk@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Iason Krommydas <iason.krom@gmail.com>
* fix: wrong results from string sort/unique kernels The sort_asstrings kernel iterated the per-string character copy with a uint8_t counter initialised from an int64_t byte offset, so once the cumulative offset exceeded 255 the counter wrapped and strings were built from the wrong bytes. The loop now uses an int64_t index bounded directly by the stop offset. The unique_strings kernel compacts strings toward the front of the buffer in place, but compared each candidate against the previously kept string at its original input offset, a region the compaction may have already overwritten. It now compares against the kept string's location in the compacted output, so adjacent duplicates are correctly removed. Assisted-by: ClaudeCode:claude-opus-4.8 * test: regression tests for string sort/unique kernels Cover sorting and uniqueing of string arrays whose cumulative byte length exceeds 255 (exercising the former uint8 wraparound) and adjacent duplicate strings of differing lengths (exercising the in-place compaction comparison). Assisted-by: ClaudeCode:claude-opus-4.8 * chore: slim test suite to one focused test per bug fixed Removed three redundant test cases: - test_unique_strings_over_255_cumulative_chars (unique bug is about in-place compaction, not uint8 overflow; covered by adjacent-duplicates test) - test_unique_strings_short_then_long_duplicates (near-duplicate of adjacent-duplicates test) - test_sort_and_unique_mixed_lengths (covered by the two remaining tests) Assisted-by: ClaudeCode:claude-sonnet-4-6 --------- Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
…ching (#4090) * fix: int64 precision loss in sort comparators and per-comparison branching - Use `if constexpr (std::is_integral_v<T>)` in sort_order_ascending/ descending and argsort_order_ascending/descending so integer types compare directly without double-casting (fixes precision loss for int64/uint64 values > 2^53, and eliminates two casts + isnan per comparison for integral types). - Move explicit `template<>` bool specializations above the implicit instantiations that use them, fixing ill-formed translation units per [temp.expl.spec]. - Hoist ascending/stable dispatch outside the comparator lambda in awkward_argsort, matching the structure of awkward_sort so the branch is evaluated once per segment rather than once per comparison. Assisted-by: ClaudeCode:claude-sonnet-4-6 * test: regression tests for int64 sort precision fix (#4090) Tests that ak.sort and ak.argsort on int64/uint64 values > 2^53 match numpy exactly, verifying the double-cast precision bug is fixed. Also covers NaN float handling (unchanged), bool sort, descending order, stable sort, and multiple sublists. Assisted-by: ClaudeCode:claude-sonnet-4-6 * style: pre-commit fixes * chore: slim down PR #4090 comments and tests - Remove descriptive comments in C++ that restate what code does; keep the non-obvious constraint comment about specialization ordering. - Trim 13 regression tests to 5: one per distinct concern (int64 sort, int64 argsort, uint64 sort, float NaN unchanged, bool specialization). - Rename test file to use underscores to pass validate-test-names hook. Assisted-by: ClaudeCode:claude-sonnet-4-6 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
* fix: LayoutBuilder Union form JSON and Record map-type support
- Fix Union::form() producing invalid JSON (double-comma, missing comma)
when parameters are set; params string is now built leading-comma-free
like Record/Tuple do ("\"parameters\": { ... }, ") in
header-only/layout-builder/awkward/LayoutBuilder.h
- Fix Record<MAP,...>::ContentsFormFunctor hardcoding std::map so that
form() now compiles for any MAP type (e.g. std::unordered_map) by
using the class's UserDefinedMap alias instead
- Add <sstream> include to LayoutBuilder.h and utils.h to eliminate
reliance on transitive includes
- Fix CMakeLists.txt option(BUILD_TESTS ...) missing help-string argument
and add enable_testing() so ctest can discover tests
- Add regression tests in header-only/tests/test_4088-layoutbuilder-form-fixes.cpp
Assisted-by: ClaudeCode:claude-sonnet-4-6
* chore: slim down PR comments and tests per review
Remove comment suffix that restates the code ("uses class MAP type"),
and drop the redundant test_union_form_without_parameters test case
(no-params path was never broken) plus the ordered_map assertions in
test_record_unordered_map (std::map was always working). Inline the
two helper predicates into the remaining test.
Assisted-by: ClaudeCode:claude-sonnet-4-6
---------
Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
* fix: misc integration bugs (hist, avro, numexpr, RDataFrame) - hist.py: return NotImplemented singleton (not NotImplementedError class) on TypeError in broadcast_and_flatten - avro.py: decode_zigzag the correct variable (header_size, not pairs) in negative-pair-count header branch - numexpr.py: use import importlib.metadata instead of bare importlib (submodule not auto-imported) - rdataframe/from_rdataframe.py: always build intermediates with highlevel=True to avoid AttributeError on .layout when highlevel=False with keep_order=True or awkward-type columns; apply highlevel once at return Assisted-by: ClaudeCode:claude-sonnet-4-6 * chore: rename test file with PR number 4096 Assisted-by: ClaudeCode:claude-sonnet-4-6 * style: pre-commit fixes * chore: slim down test and fix file name convention - Rename test file to use underscores (required by validate-test-names.py) - Remove redundant happy-path test that doesn't verify the bugfix - Remove module docstring, function docstring, and inline comment that merely restate what the code does Assisted-by: ClaudeCode:claude-sonnet-4-6 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
* migrate `UnionArray` kernels * update `UnionArray` kernels * exclude `UnionArray_flatten_length` and `UnionArray_flatten_combine` kernels * fix the tests
* fix: modify cache key for macOS deployment in workflow Updated cache key for awkward-cpp wheel to include macOS version. * Update MACOSX_DEPLOYMENT_TARGET environment variable Set MACOSX_DEPLOYMENT_TARGET based on the runner type. * Apply suggestion from @ariostas Co-authored-by: Andres Rios Tascon <ariostas@gmail.com> --------- Co-authored-by: Andres Rios Tascon <ariostas@gmail.com>
* feat: add local_index kernel implementation using cuda.compute * feat: add `ListArray_getitem_jagged` and `IndexedArray_numnull` kernels using cuda.compute * feat: add `NumpyArray_reduce_adjust_starts_shifts` and `NumpyArray_reduce_mask_ByteMaskedArray` kernels implementations using cupy * feat: add `IndexedArray_flatten_nextcarry` kernel implementation using cuda.compute * small fix * style: pre-commit fixes * style: pre-commit fixes * Apply suggestion from @ianna --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
…rations (#4148) * docs: describe the returned object in Returns: for already-merged operations. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reduce Returns: to a noun clause; move narrative to the description. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: reject non-integer float indices instead of truncating Change is_integer_like to use __index__ protocol instead of __int__ in the fallback branch, and use operator.index() in normalize_integer_like and regularize_axis. Python float and Decimal no longer pass is_integer_like, so ak.Array([...])[1.5] now raises TypeError (matching NumPy behavior) instead of silently truncating to 1. axis=1.5 now also raises instead of being silently treated as axis=1. Assisted-by: ClaudeCode:claude-sonnet-4-6 * test: add regression tests for float index rejection (PR #4097) Cover: float scalar index raises, int/np.integer/__index__ objects work, axis=1.5 raises, np.float64 index raises, Decimal index raises. Assisted-by: ClaudeCode:claude-sonnet-4-6 * style: pre-commit fixes * test: slim down float-index regression tests Remove redundant parametrize variants and near-duplicate cases; keep one focused test per distinct behavior (float index raises, Decimal raises via __int__-not-__index__ boundary, __index__ protocol still accepted, int still works, float axis raises, int axis still works). Drop module docstring to match surrounding test density. Assisted-by: ClaudeCode:claude-sonnet-4-6 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: typetracer correctness issues affecting dask-awkward Fixes several typetracer correctness issues, mostly affecting dask-awkward, found via an automated multi-agent review: - typetracer min/max/sum(axis=None) recursed with axis=None instead of axis=0, causing a RecursionError; sum now reports NumPy's promoted dtype (e.g. bool/int32 -> platform int). - typetracer.stack built the result shape incorrectly (treated the prototype tuple as data, and normalized negative axis over the input ndim rather than ndim+1); now builds the shape symbolically and uses numpy.result_type for the dtype. - searchsorted returned the input dtype instead of intp insertion indices. - asarray now honors the requested dtype for non-TypeTracerArray inputs and only rejects copy=False when an incompatible dtype is requested. - reshape now permits zero-size dimensions, matching NumPy and the concrete backends. - TypeTracerArray.__repr__ no longer reads the touching .shape property (printing mutated the report); same dead None-shape branch removed in PlaceholderArray and VirtualNDArray. - cupy reducers (all/any/min/sum/max) now forward keepdims, preserving the cupy#3819 .item() workaround only for axis=None without keepdims. - NumpyArray._pad_none and Content._combinations_axis0 now guard against unknown_length, fixing ak.pad_none and ak.combinations(axis=0) on typetracer layouts. - Removed verified-dead branches in __getitem__, derive_slice_for_length, and broadcast_arrays. Assisted-by: ClaudeCode:claude-opus-4.8 * test: add regression tests for typetracer correctness fixes Assisted-by: ClaudeCode:claude-opus-4.8 * test: trim regression tests and restated comments Keep one focused test per fix and drop comments that merely restate the adjacent code. Assisted-by: ClaudeCode:claude-opus-4-8 --------- Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
* fix: BitMaskedArray mask inversion and placeholder detection - Use bitwise `~` instead of `logical_not` in `to_BitMaskedArray` when inverting `valid_when` with the same `lsb_order`, matching the different-lsb_order branch and preventing silent data corruption. - Remove erroneous `out._content` unwrap in `BitMaskedArray._unique` so it matches `ByteMaskedArray._unique` semantics (both now delegate to `IndexedOptionArray64._unique` and return the result unchanged). - Fix `_is_getitem_at_placeholder` in `BitMaskedArray`, `ByteMaskedArray`, and `UnionArray` to check `.data` on the Index wrapper, consistent with `ListOffsetArray` and `IndexedArray` siblings. Assisted-by: ClaudeCode:claude-sonnet-4-6 * test: add regression tests for BitMaskedArray fixes (#4092) Covers to_BitMaskedArray mask inversion, _unique consistency between BitMasked and ByteMasked option types, and _is_getitem_at_placeholder buffer check propagation. Assisted-by: ClaudeCode:claude-sonnet-4-6 * style: pre-commit fixes * refactor(tests): slim down and rename test_4092 regression tests Remove redundant round-trip, big-endian, and ByteMaskedArray tests that don't cover changed code paths. Remove placeholder tests that pass even without the fix. Rename file to use underscores per project convention. Two focused tests remain: one per distinct bug fixed. Assisted-by: ClaudeCode:claude-sonnet-4-6 * Apply suggestion from @ianna * revert Fix 2, Fixes 1 (bitmask inversion) and 3 (placeholder .data checks) are untouched and intact. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
…-file Parquet attrs (#4099) * fix: Arrow conversion without pandas, empty chunked arrays, and multi-file Parquet attrs - Map Arrow primitive types to NumPy dtypes directly instead of via DataType.to_pandas_dtype(), which hard-imports pandas in pyarrow >= 22 and broke ak.from_arrow / ak.from_parquet in pandas-free environments. - Handle empty (zero or all-empty-chunk) ChunkedArrays in from_arrow by building a length-zero array instead of calling concatenate([]). - Propagate parquet-stored .attrs across multi-file datasets, mirroring the single-file path. - Drop the dead multi-batch branch in the Table handler (combine_chunks yields at most one batch). Assisted-by: ClaudeCode:claude-fable-5 * test: add regression tests for pyarrow pandas-free, empty chunked, and multi-file parquet attrs Assisted-by: ClaudeCode:claude-fable-5 * chore: slim test suite — remove redundant cases, keep one per fix Assisted-by: ClaudeCode:claude-sonnet-4-6 --------- Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
update python version for mypy
* feat: port cuda kernels to cuda.compute * improve comments * cleanup * style: pre-commit fixes --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
…4098) * fix: BitMasked builder use-after-free and GrowableBuffer copy bugs Header-only LayoutBuilder/GrowableBuffer fixes found in a code review: - BitMasked stored a `uint8_t& current_byte_ref_` aliasing element 0 of the mask buffer's first panel. `append_begin()`/`clear()` assigned *through* the reference instead of rebinding it, so it forever aliased the freed panel after `clear()` (heap-use-after-free, confirmed with ASan), and mask bytes beyond the first landed in the wrong place. Removed the reference member; `current_byte_`/`current_index_` are now plain state and each completed byte is appended to `mask_` (with the `valid_when` inversion applied) when it fills, or written as the trailing partial byte at finalization (LayoutBuilder.h, BitMasked: constructors, clear, length, buffer_nbytes, to_buffers/to_buffer/to_char_buffers, append_begin/append_end, write_mask). - For BitMasked<valid_when=false> the inversion now happens where each completed byte lands in the buffer, fixing wrong mask bytes past 8 elements. - GrowableBuffer::Panel::concatenate_to_from recursed with `offset + length_` instead of `offset + (length_ - from)`, leaving a `from`-sized hole of uninitialised bytes when the source spanned multiple panels (hit by BitMasked masks > ~8190 elements) (GrowableBuffer.h:135). - GrowableBuffer memcpy sizes computed `length * sizeof(PRIMITIVE) - from`, mixing byte and element counts; corrected to `(length - from) * sizeof(...)` in Panel::append and concatenate_to_from (GrowableBuffer.h:119, :133). - Indexed::extend_index / IndexedOption::extend_valid computed `stop - 1` for size==0 on empty content, underflowing to SIZE_MAX and poisoning max_index_/last_valid_ so is_valid() falsely failed; guard size==0. Also removed the dead `else if (i < 0)` branch on a size_t (always false). Adds header-only/tests/test_4087-bitmasked-growablebuffer.cpp covering builder reuse after clear(), BitMasked valid_when=false, >8 elements, a >1-panel mask (20000 elements), zero-size extend_index/extend_valid, and multi-panel GrowableBuffer concatenate with non-zero `from` and sizeof(PRIMITIVE)>1; the UAF reproduces under ASan against the pre-fix headers. Also adds `enable_testing()` so ctest discovers the suite. The header-only library is standalone; the generated copies under awkward-cpp/ and src/awkward/_connect/header-only (produced by dev/copy-cpp-headers.py at prepare time) are not committed. Assisted-by: ClaudeCode:claude-fable-5 * chore: trim redundant comments and dead alias from BitMasked/GrowableBuffer PR Remove unused NumpyBuilder alias from test, drop restate-the-code comments (length formula, ceil expression, element-pattern description), and condense LayoutBuilder.h docstrings to remove padding while keeping the non-obvious rationale for the use-after-free fix and valid_when encoding constraint. Assisted-by: ClaudeCode:claude-sonnet-4-6 * fix: complete target_link_libraries for test_4087 header-only test (#4171) The test_4087-bitmasked-growablebuffer target was missing its `PRIVATE awkward::layout-builder)` line, so the call swallowed the next command and the header-only test suite failed to configure. Add the missing line so CMake can configure and the tests build. Claude-Session: https://claude.ai/code/session_013yRH8RuTZNTdyREnNLQTK9 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch> Co-authored-by: Tai Sakuma <tai.sakuma@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps [hypothesis-awkward](https://github.com/scikit-hep/hypothesis-awkward) from 0.18.0 to 0.18.1. - [Release notes](https://github.com/scikit-hep/hypothesis-awkward/releases) - [Changelog](https://github.com/scikit-hep/hypothesis-awkward/blob/main/CHANGELOG.md) - [Commits](scikit-hep/hypothesis-awkward@u0.18.0...u0.18.1) --- updated-dependencies: - dependency-name: hypothesis-awkward dependency-version: 0.18.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* docs: update README.md [skip ci] * docs: update .all-contributorsrc [skip ci] --------- Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
fix: include kernel-specification.yml in source distribution (#4178) tests/test_4056_cuda_compute_signature_consistency.py reads kernel-specification.yml from the repo root, but the file was not part of the sdist include list, so the test errored with FileNotFoundError when the suite ran against an unpacked source distribution (e.g. conda-forge). The tests/ directory already ships, so this adds the one data file it depends on. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Bumps the actions group with 2 updates: [actions/setup-python](https://github.com/actions/setup-python) and [actions/cache](https://github.com/actions/cache). Updates `actions/setup-python` from 6.2.0 to 6.3.0 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@a309ff8...ece7cb0) Updates `actions/cache` from 5.0.5 to 6.1.0 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@27d5ce7...55cc834) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: 6.3.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: actions/cache dependency-version: 6.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* write attrs when writing parquet row groups iteratively * document the attrs that are written to parquet files * typo Signed-off-by: Iason Krommydas <iason.krom@gmail.com> --------- Signed-off-by: Iason Krommydas <iason.krom@gmail.com>
…4215) Generate arrays with hypothesis-awkward's `arrays()` strategy, restricted to layouts that `ak.to_numpy` accepts: no unions, no variable-length lists, no records (#3690), no strings (NumPy `U`/`S` dtypes strip trailing NUL characters), and no `EmptyArray`. A second test recovers option-type coverage through `np.ma.MaskedArray` with `same_content_types=False`, excluding timedelta64, whose missing-value conversion raises in `create_missing_data`. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs: add summary lines to 5 math/numeric operation docstrings Add a Google-style one-line summary and `Returns:`/`Examples:` section headers to the 5 math/numeric operations in `src/awkward/operations/`: round, real, imag, angle, isclose. Follows the pilot in #3946 on `ak.flatten`. Original body text is preserved; only structural edits are applied. Refs #3980. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * docs: describe the returned object in Returns: lines. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reduce Returns: to a noun clause; move narrative to the description. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…#4137) * docs: add summary lines to 11 from_* file/format converter docstrings Add a Google-style one-line summary and `Returns:`/`Examples:` section headers to the 11 from_* file/format converters in `src/awkward/operations/`: from_arrow, from_arrow_schema, from_parquet, from_feather, from_json, from_avro_file, from_safetensors, from_iter, from_buffers, from_regular, from_categorical. Follows the pilot in #3946 on `ak.flatten`. Original body text is preserved; only structural edits are applied. Refs #3980. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * docs: describe the returned object in Returns: lines. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reduce Returns: to a noun clause; move narrative to the description. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…t` (#4218) * test: add property-based roundtrip test for `to_parquet`/`from_parquet` Generate arrays restricted to what the parquet path can roundtrip: no unions, no unknown types, and dtypes limited to those pyarrow can write and read back unchanged (complex excluded; datetime64 only "ns"; timedelta64 only "s"/"ms"/"us"/"ns"). A `parquet_writable` predicate excludes layouts that `to_arrow`/`from_parquet` currently mishandle, including a silent offsets-shift corruption for nullable var-length lists. Tests write to fsspec's in-memory filesystem, so no disk I/O. A second test covers option-type layouts (returned as bit-masked, so content classes are not compared), and a third checks that `attrs` survive the roundtrip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: cite issues for the parquet roundtrip exclusions The four defects excluded by `parquet_dtype` and `parquet_writable` are now reported: #4219 (datetime64[D] value corruption), #4220 (row-group statistics OverflowError), #4221 (zero-length-content IndexError), and #4222 (offsets-shift corruption for nullable lists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: fix parquet roundtrip test on Linux and Windows CI Skip when pyarrow is not installed (Windows CI), matching the other parquet tests, and exclude extended-precision floats: hypothesis-awkward builds its dtype list from the platform's NumPy, so Linux generates float128, which pyarrow cannot write. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* chore: remove Gitter chat and Stackoverflow links from README Removed Gitter chat room link from README. * Update issue template by removing contact links Removed StackOverflow and Gitter contact links from issue template. * Update feature request template links Removed StackOverflow and Gitter links from feature request template. * Remove Gitter link from documentation Removed Gitter link from external links section. * Remove Chat URL from project URLs Removed the 'Chat' URL from project URLs. * Remove Gitter chat link Removed Gitter chat link from pyproject.toml.
…ache (#4095) * fix: Record field assignment mutates shared layouts and stale cppyy cache Record.__setitem__ rebound a fresh ak.record.Record instead of mutating the shared low-level record in place, so assigning a field no longer leaks into sibling Records constructed from the same Record. Record.__init__ from another Record now mirrors Array.__init__, preserving a user-passed attrs= and inheriting behavior via behavior_of/attrs_of. Array._update_class now resets the cached _cpp_type, _generator, and _lookup (which hold raw pointers into the old buffers) so mutated arrays don't hand cppyy stale lookups, and __setstate__ no longer leaves _cpp_type uninitialised. ArrayBuilder.Record stores the record name as _record_name so it no longer shadows the class-level _name display label used by _Nested.__repr__. Assisted-by: ClaudeCode:claude-fable-5 * test: regression tests for highlevel Record/ArrayBuilder fixes Assisted-by: ClaudeCode:claude-fable-5 * chore: trim redundant tests in test_4095_highlevel_record Remove five tests that either duplicate another test's coverage or test implementation details rather than observable bug behaviour: - test_record_setitem_rebinds_layout (internal detail; sibling test covers the bug) - test_record_from_record_preserves_attrs_and_behavior (explicit override path; not the regressed path) - test_arraybuilder_record_repr_with_name (near-duplicate of repr_no_name) - test_arraybuilder_list_repr (unrelated to this PR's fixes) - test_cpp_type_caches_reset_on_setitem (redundant with reset_on_layout_set) Assisted-by: ClaudeCode:claude-sonnet-4-6 --------- Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
* fix: AwkwardForth VM and libawkward correctness issues Several correctness and safety fixes in awkward-cpp/src/libawkward: - io/json.cpp (nulls_for_optiontype): stop the switch falling through from FillIndexedOptionArray into KeyTableHeader, which double-pushed the instruction stack and left it unbalanced for nullable records, producing a spurious "JSON schema mismatch" when a nullable record had a missing option-type key. Nullable records now null-fill missing keys like non-nullable records. - forth/ForthMachine.cpp (Nbit read): build the bit mask with a 64-bit literal and special-case width 64, fixing UB/wrong masks for N >= 31 (is_nbit allows up to 64). - forth/ForthMachine.cpp (constructor): hold the seven scratch buffers in std::unique_ptr<T[]> so a std::invalid_argument thrown by tokenize/compile on bad AwkwardForth source no longer leaks them; destructor defaulted. - forth/ForthInputBuffer.cpp + ForthMachine.cpp read macros: reject negative and overflowed read sizes in read()/peek_byte() (sizes derive from stack-supplied item counts). - forth/ForthInputBuffer.cpp (read_textfloat): cap integral-mantissa accumulation at 18 digits and continue in double, avoiding signed int64 overflow UB on long literals. - forth/ForthMachine.cpp (is_integer): use std::stoll, catch out_of_range, and require full-string consumption so trailing junk and overlong literals are rejected. - forth/ForthMachine.cpp (enum/enumonly): bounds-check the keyword lookup on the error path so it no longer reads one past the token vector. - forth/ForthMachine.cpp (stack_at): fix off-by-one so stack_at(0) returns the top element instead of one past it. - forth/ForthOutputBuffer.cpp (maybe_resize): guarantee strict growth so an initial size of 0 or a resize factor <= 1.0 no longer loops forever. - builder/RecordBuilder.cpp (clear): keep the record structure (contents_, keys_, pointers_, keys_size_) consistent and reset length_ to 0, mirroring TupleBuilder::clear(); previously clear() left keys_ out of sync with contents_ (out-of-bounds reads) and a length of -1. - util.cpp (dtype_to_format): drop the stray "defined" in the uint32/uint64 preprocessor checks so they map to the correct buffer-format chars on 32-bit non-MSVC platforms. Assisted-by: ClaudeCode:claude-fable-5 * test: regression tests for AwkwardForth VM and libawkward fixes Covers the behaviorally-verifiable fixes: from_json with a nullable-record schema and missing option-type keys, ArrayBuilder clear-then-reuse, ForthMachine64 with output_initial_size=0 / resize factor 1.0, and Nbit reads with N >= 32. Assisted-by: ClaudeCode:claude-fable-5 * test: trim regression tests and comments for forth/libawkward fixes Drop the non-nullable from_json guard (already passed pre-fix), the duplicate output-resize test (same maybe_resize bug as the initial-size-zero case), and two interior nbit-mask parametrize cases. Condense the verbose test docstring-comments to match the repo's test style, and tighten the RecordBuilder::clear() comment. Assisted-by: ClaudeCode:claude-opus-4-8 --------- Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
* docs: add summary lines to 8 misc operation docstrings Add a Google-style one-line summary and `Returns:`/`Examples:` section headers to the 8 remaining operations in `src/awkward/operations/`: copy, backend, materialize, transform, where, type, merge_option_of_records, merge_union_of_records. Follows the pilot in #3946 on `ak.flatten`. Original body text is preserved; only structural edits are applied. Refs #3980. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * docs: describe the returned object in Returns: lines. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reduce Returns: to a noun clause; move narrative to the description. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: de-duplicate relocated Returns prose; single-clause materialize. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
fix: undo a rejected assignment The layout and behavior setters assigned before _update_class() ran __awkward_validation__, so a rejected `array["rho"] = ...` raised and applied the field anyway. They now restore the previous value if validation raises. Record.__setitem__ mutated layout._array in place, which left the setter with nothing to restore; it builds a new Record now. That is the same fix as item 1 of #4095.
* test: add stability roundtrip test for `to_numpy`/`from_numpy` The strict roundtrip tests exclude layouts that `to_numpy` converts lossily (unknown types, strings with trailing NULs, option-flavor normalization). The new test covers that domain with a weaker property: the converted form is a fixed point, i.e. `to_numpy` of the reconstructed array reproduces the same NumPy array, including dtype (except "U"/"S" itemsize, which trailing-NUL stripping narrows) and mask (compared explicitly, since `np.testing.assert_array_equal` treats masked positions as equal to anything). `allow_regular` stays off: option-over-regular layouts produce multidimensional masked arrays that crash `from_numpy` in two ways, to be reported separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: cite issues for the stability test's regular-array exclusion The two `from_numpy` crashes excluded by `allow_regular=False` are now reported: #4226 (AttributeError for multidimensional masked string arrays with no masked elements) and #4227 (ValueError for masked arrays with ndim >= 3 and a zero-length dimension). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs: add summary lines to 19 reducer operation docstrings Add a Google-style one-line summary and `Returns:`/`Examples:` section headers to the 19 reducer operations (28 functions, including the nan* variants) in `src/awkward/operations/`: all, any, sum, prod, count, count_nonzero, min, max, argmin, argmax, mean, std, var, moment, corr, covar, ptp, linear_fit, softmax. Follows the pilot in #3946 on `ak.flatten`. Original body text is preserved; only structural edits are applied. Refs #3980. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * docs: describe the returned object in Returns: lines. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reduce Returns: to a noun clause; move narrative to the description. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: move description text out of Examples: sections Per the Examples:-section checklist on #3980: narrative that describes behavior, arguments, or history belongs above Args: in the extended description; Examples: keeps only doctest walkthroughs and their connecting prose; "See also" lines go directly above Args:. Refs #3980. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add summary lines to 3 *_like operation docstrings Add a Google-style one-line summary and `Returns:`/`Examples:` section headers to full_like, ones_like, and zeros_like in `src/awkward/operations/`. Follows the pilot in #3946 on `ak.flatten`. Original body text is preserved; only structural edits are applied. Refs #3980. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * docs: describe the returned object in Returns: lines. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reduce Returns: to a noun clause; move narrative to the description. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: move description text out of Examples: sections Per the Examples:-section checklist on #3980: narrative that describes behavior, arguments, or history belongs above Args: in the extended description; Examples: keeps only doctest walkthroughs and their connecting prose; "See also" lines go directly above Args:. Refs #3980. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…4132) * docs: add summary lines to 5 sorting/indexing operation docstrings Add a Google-style one-line summary and `Returns:`/`Examples:` section headers to the 5 sorting/indexing operations in `src/awkward/operations/`: sort, argsort, run_lengths, local_index, num. Follows the pilot in #3946 on `ak.flatten`. Original body text is preserved; only structural edits are applied. Refs #3980. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * docs: describe the returned object in Returns: lines. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: move description text out of Examples: sections Per the Examples:-section checklist on #3980: narrative that describes behavior, arguments, or history belongs above Args: in the extended description; Examples: keeps only doctest walkthroughs and their connecting prose; "See also" lines go directly above Args:. Refs #3980. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs: add summary lines to 10 type/validity operation docstrings Add a Google-style one-line summary and `Returns:`/`Examples:` section headers to the 10 type/validity/equality operations in `src/awkward/operations/`: almost_equal, array_equal, is_valid, validity_error, is_categorical, is_tuple, categories, enforce_type, values_astype, strings_astype. Follows the pilot in #3946 on `ak.flatten`. Original body text is preserved; only structural edits are applied. Refs #3980. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * docs: describe the returned object in Returns: lines. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reduce Returns: to a noun clause; move narrative to the description. Refs #3980. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: move description text out of Examples: sections Per the Examples:-section checklist on #3980: narrative that describes behavior, arguments, or history belongs above Args: in the extended description; Examples: keeps only doctest walkthroughs and their connecting prose; "See also" lines go directly above Args:. Refs #3980. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…4231) * test: add property-based roundtrip tests for `to_arrow`/`from_arrow` Four tests: a strict roundtrip that includes unions (unique to the Arrow pair; contents must be pairwise non-mergeable because `from_arrow` rebuilds unions with `UnionArray.simplified`), a masked roundtrip through Arrow validity bitmaps, a table roundtrip that also checks `from_arrow_schema` against the returned form, and a stability test asserting that one full roundtrip brings the conversion to a fixed point, compared on the Arrow side with `Array.equals`. Generation is restricted by a dtype filter (complex and float128 unrepresentable; datetime64/timedelta64 limited to "s"/"ms"/"us"/"ns", with datetime64[D] corrupted by #4219) and a layout predicate excluding known defects: #4221, #4222 (including a new trigger via conversion-time offset shifts), #4228, #4229, and #4230. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: restructure arrow roundtrip predicates by exclusion kind Split the generation predicates of test_to_from_arrow.py into the structure introduced by the feather sibling (#4239): `arrow_compatible` and `table_compatible` hold the permanent limits of the conversion (mergeable union contents merge on read, an optioned unknown gains an option, plus the table-wrapping record clauses), while `has_issues` composes one `_has_issue_NNNN` function per known issue (#4221, #4222, #4228, #4229) so each released fix deletes exactly one function. The filters name both parts inline, and the stability test's filter is now literally `not has_issues(a)`. No behavioral change: a 40,000-example equivalence check against the original merged predicates over all four generation domains found no disagreement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…er` (#4239) Add three Hypothesis tests in tests/properties/operations/: - test_roundtrip: option-free arrays reconstruct exactly - test_roundtrip_masked: option-type arrays roundtrip through nullable feather columns, without comparing content classes - test_roundtrip_stable: one write/read cycle brings the conversion to a fixed point, compared as Arrow tables `to_feather` requires a real filesystem path (its destination goes through `os.fsdecode`), so unlike the parquet sibling the tests write to a module-scoped `tmp_path_factory` directory, with a per-thread filename for pytest-run-parallel. Generation excludes arrays affected by known issues, one predicate function per issue so each released fix deletes exactly one function: #4221, #4222, and #4229 in `has_issues`, #4230 and #4238 via strategy flags, and #4219 in the dtype filter. The format's own limits live in `feather_compatible` and `feather_stable`, which are all that remains when the cited issues are fixed. Validated with 10,000 examples per test (nightly profile), fixed seeds 1-8, and under pytest-xdist. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…4245) Add four Hypothesis tests in tests/properties/operations/: - test_roundtrip: option-free arrays on the inference-stable domain (bool, int64, float64, strings) reconstruct exactly - test_roundtrip_masked: option-type arrays roundtrip through JSON nulls, without comparing content classes (`from_json` returns every option as an IndexedOptionArray) - test_to_json_matches_json_loads: an oracle check of the writer against Python's own JSON parser on the widest serializable domain - test_roundtrip_stable: one roundtrip brings the conversion to a fixed point, compared as JSON text Schemaless `from_json` rebuilds types purely from the values present in the JSON text, so `json_compatible` requires a witness for every part of the type: no zero-length branches, no all-valid or all-null options, no tuples (written as objects with stringified keys). Generation excludes arrays affected by known issues, one predicate function per issue: #4241 (floats parse one ULP off) and #4242 (crash on null alongside heterogeneous values) in `has_issues`; #4243 (uint64 wrap) is noted in the stability test's docstring, which keeps uint64 in its domain. Unlike the parquet and feather siblings, no `deadline=None` is needed: first calls cost under 0.2 ms, and ten default-profile runs under the 200 ms deadline never tripped it. Validated with 10,000 examples per test (nightly profile), fixed seeds, and under pytest-xdist. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
#4240) * rgsort reducer * add tests * add argsort * sort nan aware bugfix * enable all argsort tests * add test for better coverage * address order-preserving unsigned key transform * fuse kernels and increase coverage
* overflow-safe stats without needless copies * add test * fix lint * address reviewer comments
…#4234) Per the Examples:-section checklist on #3980: narrative that describes behavior, arguments, or history belongs above Args: in the extended description; Examples: keeps only doctest walkthroughs and their connecting prose; "See also" lines go directly above Args:. Refs #3980. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ngs (#4249) * fix string lowering * add test * update * avoids handling the LLVM shape tuple entirely * Update src/awkward/_connect/numba/builder.py
Stop requiring future annotations
…/corr via fused two-pass sum-of-squares/powers reducers (#4232) * Add a dtype= accumulator to ak.sum (overflow-safe, copy-free integer sums) * add tests * add more tests to increase coverage * add a dedicated sum-of-squares reducer * style: pre-commit fixes * add tests * remove obsolete tests * add the kernel * jax backend too * GPU sum-of-squares path has no host transfer -- fix the test * the promote copy is now eliminated everywhere * two-pass/one-pass fallback * the internal centering is an implementation detail and shouldn't be gated by named-axis metadata * more tests to increase coverage * recalculate xmean and ymean with the user's original keepdims parameter * all stays on device * more test coverage * platform integer differs * integer-power fast path for moment, and a NumPy axis=None path for sum-of-squares * fix bottleneck * apply the same axis=None scalar-centering to covar/corr * address review comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…nt data corruption (#4250) * stop making numba strings through pyobjects * add tests * set the one-dimensional shape and stride explicitly * add invalid-UTF-8 tests * change ValueError branches with UnicodeDecodeError for consistency with Python's decoder * revert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.