Adding `copy_store` convenience method for `Group` #3612

melonora · 2025-12-03T15:51:49Z

[Description of PR]
This PR adds a convenience method for copying the store of a Group to a destination store.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/user-guide/*.md
Changes documented as a new file in changes/
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

codecov · 2025-12-03T16:01:45Z

Codecov Report

❌ Patch coverage is 89.28571% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.96%. Comparing base (ee0e69a) to head (3398325).

Files with missing lines	Patch %	Lines
src/zarr/core/group.py	88.88%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3612      +/-   ##
==========================================
+ Coverage   59.14%   60.96%   +1.82%     
==========================================
  Files          86       86              
  Lines       10172    10197      +25     
==========================================
+ Hits         6016     6217     +201     
+ Misses       4156     3980     -176

Files with missing lines	Coverage Δ
src/zarr/testing/strategies.py	`97.82% <100.00%> (ø)`
src/zarr/core/group.py	`70.85% <88.88%> (+0.57%)`	⬆️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

melonora · 2025-12-03T16:37:04Z

@d-v-b I know right now this won't work when zarr v2 is used. I can add that functionality as well or we say the convenience function is not supported for zarr v2.

melonora · 2025-12-03T17:53:12Z

added zarr v2 support. If in general the code is approved, I will add documentation.

src/zarr/core/group.py

d-v-b · 2025-12-03T18:04:36Z

src/zarr/core/group.py

+        store: StoreLike,
+        *,
+        overwrite: bool = False,
+        consolidate_metadata: bool | None = None,


why do we need this here? IMO it's simpler if copying is just copying. If people want to add consolidated metadata, they should do that after copying.

adjusted it. It now consolidates if the source store was consolidated, but this does not require an argument. One more thing, I only take into account when consolidation happened at the root level. It could be (although the user should not do this) that consolidation happened at the subgroup level. This could be accounted for by checking if member. metadata.consolidated_metadata when looping through members. WDYT should I add this?

I would copy the original group and its children exactly, including their consolidated metadata.

What would be the purpose of copying over stale consolidated metadata?

it keeps the scope of this function simple -- it just copies the original group exactly. If this tool produces a different group, then it's not really copying.

regarding the from_store method, I would open a separate PR for this.

Consolidated metadata is already in the group metadata document for zarr v3 if "kind" is set to "inline".

👍 but doing something other (more or less) than just copying would require interpreting that which is too far IMO.

FYI @joshmoore the method under discussion here is a method on the Group class, not on stores. maybe this is a point of confusion.

IMO the only consolidated metadata-related parameter we need is whether iterating over the child groups uses their consolidated metadata (if present). See the relevant kwarg in the signature of members.

I added both, only thing remaining is probably adding consolidated_metadata in the async group call.

src/zarr/core/group.py

melonora · 2025-12-05T22:55:36Z

@d-v-b @joshmoore I think I have now addressed all comments and I also added documentation, so I think this would be ready for a final round of review. @d-v-b thanks for your patience.

melonora · 2025-12-10T11:02:42Z

@d-v-b @joshmoore , just bumping in case you have some time to see whether you approve the current state:)

src/zarr/core/group.py

d-v-b · 2025-12-10T11:10:02Z

src/zarr/core/group.py

+        self,
+        store: StoreLike,
+        *,
+        overwrite: bool = False,
+        use_consolidated_for_children: bool = True,


unlike create_group, this function doesn't take a path parameter, which means it's not possible to easily specify a location within a store for the group. until we make it easy to include this information in the store itself (#2272 ) we should probably take a path parameter.

I am not certain yet, first the group is opened by calling self.open(self.store, zarr_format=target_zarr_format). If we allow for path to be passed I interpret it as subgroup within existing zarr store to which to copy whatever store you pass. Am I following you correctly?
In other words are you saying we should allow to write a store to a subgroup destination?

self.open(self.store, zarr_format=target_zarr_format)

I am still confused why this line is necessary. It will open an array or group at the root of the store, which is not what we want:

>>> import zarr >>> zg = zarr.create_group(store={}, path='group') >>> zg <Group memory://133419005213760/group> >>> zg.open(zg.store) <Group memory://133419005213760>

In other words are you saying we should allow to write a store to a subgroup destination?

this method does not write a store, it writes a group into a store. In normal group creation, you declare the store, and also the path within the store. See the signature of create_group. The question is whether copy_to should follow the same pattern, and take a path argument.

will implement it also for the sake of symmetric API

On second look I think it is still a bit confusing, the signature for create_group in the Group class itself does not have path. However, create_group in the async and sync API does include this. I take it the sole reason for create_group in the classes is to create a subgroup so I still think implementing path would be good in that case to have it model more the module level functions rather than the instance methods.

However, create_group in the async and sync API does include this.

IMO we should treat the async / sync api create_group as the reference here. We are long overdue a cleanup of these methods / functions

@d-v-b implemented, let me know what you think when you have time!

d-v-b · 2025-12-10T11:13:44Z

src/zarr/core/group.py

+        self,
+        store: StoreLike,
+        *,
+        overwrite: bool = False,
+        use_consolidated_for_children: bool = True,


we should add a test that checks the signature of this function against the signature of the async version. otherwise, they have a tendency to drift. Here's an example of such a test:

zarr-python/tests/test_api/test_synchronous.py

Lines 26 to 41 in ee0e69a

def test_docstrings_match(callable_name: str) -> None:

"""

Tests that the docstrings for the sync and async define identical parameters.

"""

callable_a = getattr(synchronous, callable_name)

callable_b = getattr(asynchronous, callable_name)

if callable_a.__doc__ is None:

assert callable_b.__doc__ is None

else:

params_a = NumpyDocString(callable_a.__doc__)["Parameters"]

params_b = NumpyDocString(callable_b.__doc__)["Parameters"]

mismatch = []

for idx, (a, b) in enumerate(zip(params_a, params_b, strict=False)):

if a != b:

mismatch.append((idx, (a, b)))

assert mismatch == []

so I did not do the actual docstrings itself, but I did write a test for all methods for classes and their async counterparts. I think this is generalizable to other classes as well, though currently only Group and AsyncGroup are tested. With this, already some mismatches are detected which for now are skipped but could be addressed in a follow up PR. Doing so would require deprecating some parameters. If you agree I can create an issue for it.

Please see test_class_method_parameters_match

melonora added 2 commits December 3, 2025 16:31

add copy_store convenience method

e3ee33b

add synchronous call

83550a3

github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Dec 3, 2025

change comment, test not async

d473d6e

melonora added 2 commits December 3, 2025 17:37

fix test

bb1405e

support zarr v2

c62543a

d-v-b reviewed Dec 3, 2025

View reviewed changes

src/zarr/core/group.py Outdated Show resolved Hide resolved

d-v-b reviewed Dec 3, 2025

View reviewed changes

src/zarr/core/group.py Outdated Show resolved Hide resolved

d-v-b reviewed Dec 3, 2025

View reviewed changes

melonora added 4 commits December 3, 2025 22:44

remove use of _iter_chunk_regions

cdbc2f7

change method name

e6e10df

remove consolidate_metadata argument

9c42567

consolidate if consolidated

63c652e

d-v-b reviewed Dec 4, 2025

View reviewed changes

src/zarr/core/group.py Outdated Show resolved Hide resolved

melonora added 8 commits December 5, 2025 01:43

add consolidated_metadata argument

d4924f5

add docstring and argument

e83dda5

add support subgroup consolidated metadata

59b18ea

add argument to docstring

b65d257

add example to docs groups

1056b9e

adjust docs

eadb647

partial fix pre-commit

8c3471c

add to changes

2cbb9b9

github-actions bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Dec 5, 2025

d-v-b reviewed Dec 10, 2025

View reviewed changes

src/zarr/core/group.py Outdated Show resolved Hide resolved

d-v-b reviewed Dec 10, 2025

View reviewed changes

melonora added 4 commits December 10, 2025 12:26

change to call using self

128b924

obtain consolidated metadata from self

fa95e9c

add test for matching parameters

848811f

implement path and add test

3398325

melonora mentioned this pull request Dec 11, 2025

Writing dask arrays to a ZipStore causes a corrupt zip file #3516

Open

	def test_docstrings_match(callable_name: str) -> None:
	"""
	Tests that the docstrings for the sync and async define identical parameters.
	"""
	callable_a = getattr(synchronous, callable_name)
	callable_b = getattr(asynchronous, callable_name)
	if callable_a.__doc__ is None:
	assert callable_b.__doc__ is None
	else:
	params_a = NumpyDocString(callable_a.__doc__)["Parameters"]
	params_b = NumpyDocString(callable_b.__doc__)["Parameters"]
	mismatch = []
	for idx, (a, b) in enumerate(zip(params_a, params_b, strict=False)):
	if a != b:
	mismatch.append((idx, (a, b)))
	assert mismatch == []

Uh oh!

Adding copy_store convenience method for Group #3612

Are you sure you want to change the base?

Adding copy_store convenience method for Group #3612

Conversation

melonora commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

melonora commented Dec 3, 2025

Uh oh!

melonora commented Dec 3, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

melonora commented Dec 5, 2025

Uh oh!

melonora commented Dec 10, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

melonora Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Adding `copy_store` convenience method for `Group` #3612

Adding `copy_store` convenience method for `Group` #3612

melonora commented Dec 3, 2025 •

edited

Loading

codecov bot commented Dec 3, 2025 •

edited

Loading

melonora Dec 11, 2025 •

edited

Loading