Skip to content

VPQ Dataset: write codes one word at a time during building#1487

Draft
achirkin wants to merge 3 commits into
NVIDIA:mainfrom
achirkin:enh-vpq-dataset-faster-codewriting
Draft

VPQ Dataset: write codes one word at a time during building#1487
achirkin wants to merge 3 commits into
NVIDIA:mainfrom
achirkin:enh-vpq-dataset-faster-codewriting

Conversation

@achirkin

@achirkin achirkin commented Nov 3, 2025

Copy link
Copy Markdown
Contributor

Improve the efficiency of process_and_fill_codes_kernel by writing the codes in larger chunks.

@achirkin achirkin requested a review from a team as a code owner November 3, 2025 15:24
@achirkin achirkin added the improvement Improves an existing functionality label Nov 3, 2025
@achirkin achirkin added the non-breaking Introduces a non-breaking change label Nov 3, 2025
@achirkin achirkin moved this from Todo to In Progress in Unstructured Data Processing Nov 3, 2025
@achirkin achirkin marked this pull request as draft November 4, 2025 14:55
@copy-pr-bot

copy-pr-bot Bot commented Nov 4, 2025

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@achirkin

achirkin commented Nov 4, 2025

Copy link
Copy Markdown
Contributor Author

Moving this to draft: although the write efficiency improves by 2-5x according to the nsight profiler (less store instructions), the overall kernel runtime barely changes at all, because the bottleneck is data reading and ALU (encoding). So the value of the PR is in question.

@aamijar

aamijar commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

Hi @achirkin, should we close this PR for now if we don't need it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Improves an existing functionality non-breaking Introduces a non-breaking change stale-active

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

4 participants