Skip to content

GH-34449 [Python] array(pd.Categorical) raising for arrow-backed cat#34456

Closed
phofl wants to merge 1 commit into
apache:mainfrom
phofl:34449
Closed

GH-34449 [Python] array(pd.Categorical) raising for arrow-backed cat#34456
phofl wants to merge 1 commit into
apache:mainfrom
phofl:34449

Conversation

@phofl

@phofl phofl commented Mar 4, 2023

Copy link
Copy Markdown
Contributor

Rationale for this change

This occurs when saving pandas arrow-backed categoricals to parquet. There might be a better way to do this than calling combine_chunks, but couldn't find a way to convert a ChunkedArray to a DictionaryArray

What changes are included in this PR?

Bugfix and tests only

Are these changes tested?

Yes

Are there any user-facing changes?

Yes, bug fix

@phofl phofl requested a review from AlenkaF as a code owner March 4, 2023 20:55
@github-actions github-actions Bot added the awaiting review Awaiting review label Mar 4, 2023
@github-actions

github-actions Bot commented Mar 4, 2023

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Mar 4, 2023

Copy link
Copy Markdown

⚠️ GitHub issue #34449 has been automatically assigned in GitHub to PR creator.

@AlenkaF

AlenkaF commented Mar 6, 2023

Copy link
Copy Markdown
Member

Thank you @phofl for contributing!

As I mentioned in the issue comments, there is already a PR for a duplicated GitHub issue: #34289

As you can see in the comments of the duplicated PR issue #33727 (comment), the failure happens because pa.DictionaryArray.from_array doesn't accept chunked arrays. In this case, the ChunkedArray will always have only one chunk and so a better way of solving the issue is to return an array in case of single chunked data when calling _handle_arrow_array_protocol (__arrow_array__).

@phofl

phofl commented Mar 6, 2023

Copy link
Copy Markdown
Contributor Author

Sorry I should have checked the arrow issue. I came here through debugging the pandas equivalent. Closing then, sorry for the overhead

@phofl phofl closed this Mar 6, 2023
@AlenkaF

AlenkaF commented Mar 6, 2023

Copy link
Copy Markdown
Member

No problem at all! Thank you for making an effort 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting review Awaiting review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Python] to_parquet fails with a category field backed by pyarrow string

2 participants