Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

preserve existing run_data and append to it when calibration is run #806

Closed
wants to merge 5 commits into from
Closed

preserve existing run_data and append to it when calibration is run #806

wants to merge 5 commits into from
+400 −206

Conversation

rogthefrog
Copy link
Contributor

@rogthefrog rogthefrog requested a review from a team as a code owner January 18, 2025 03:27
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 18, 2025 03:27 — with GitHub Actions Inactive
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 18, 2025 03:27 — with GitHub Actions Inactive
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 18, 2025 03:27 — with GitHub Actions Inactive
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 18, 2025 •
edited
Loading

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 18, 2025 04:39 — with GitHub Actions Inactive
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 18, 2025 04:39 — with GitHub Actions Inactive
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 18, 2025 04:39 — with GitHub Actions Inactive
@wpietri
Copy link
Contributor

wpietri commented Jan 18, 2025

I'm glad you're tackling this.

I was thinking that as part of cleaning this up, we would split the calibration into lumps matching a single reasonable run. E.g., that the reference_standards section would be broken up by 1.0-en_us-official, 1.0-fr_fr-practice, and so on. In which case you could attach a run_info block to each lump.

That could also be changed later, but as long as you're in this I thought I'd mention it.

@rogthefrog rogthefrog force-pushed the feat/append-metadata branch from 5a4b66c to ee1621d Compare January 21, 2025 18:56
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 21, 2025 18:56 — with GitHub Actions Inactive
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 21, 2025 18:56 — with GitHub Actions Inactive
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 21, 2025 18:57 — with GitHub Actions Inactive
@rogthefrog rogthefrog force-pushed the feat/append-metadata branch from aa68cb5 to c6fd0e4 Compare January 23, 2025 02:29
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 23, 2025 03:02 — with GitHub Actions Inactive
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 23, 2025 03:02 — with GitHub Actions Inactive
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 23, 2025 03:02 — with GitHub Actions Inactive
@rogthefrog rogthefrog force-pushed the feat/append-metadata branch from c2cb66b to 5526455 Compare January 23, 2025 21:30
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 23, 2025 21:30 — with GitHub Actions Inactive
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 23, 2025 21:30 — with GitHub Actions Inactive
@rogthefrog rogthefrog temporarily deployed to Scheduled Testing January 23, 2025 21:30 — with GitHub Actions Inactive
@rogthefrog rogthefrog marked this pull request as draft January 23, 2025 21:31
@rogthefrog
refactor: split calibration from displaying the current standard valu…
cdda449 
…es; add support for calibration using specific prompt sets and locales
@rogthefrog
Copy link
Contributor Author

Closing until we pick up the work again.

@rogthefrog rogthefrog closed this Mar 24, 2025
@github-actions github-actions bot locked and limited conversation to collaborators Mar 24, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@rogthefrog @wpietri