feat(models): add opt-in custom Whisper model discovery by CJHwong · Pull Request #622 · cjpais/Handy

CJHwong · 2026-01-19T09:49:46Z

Before Submitting This PR

Please confirm you have done the following:

I have searched existing issues and pull requests (including closed ones) to ensure this isn't a duplicate
I have read CONTRIBUTING.md

If this is a feature or change that was previously closed/rejected:

I have explained in the description below why this should be reconsidered
I have gathered community feedback (link to discussion below)

Human Written Description

I work with mixed-language speech (Chinese and English) and found that the built-in Whisper models struggle with this use case.

The problem was that using a custom model required either modifying the source code or a workaround of renaming the file to match a built-in model name. This PR adds the ability to auto-discover custom Whisper GGML models from the models directory.

To address the concern about users expecting support when custom models don't work, the feature is opt-in only: it lives behind a toggle in Debug settings (off by default).

Related Issues/Discussions

Fixes #
Discussion: #609

Community Feedback

#609 (comment)

Testing

Tested with:

MediaTek-Research/Breeze-ASR-25 from Hugging Face (GGML version)
Invalid file - fails gracefully with "Failed to load..." error, no crash
Toggle off (default) - no custom models discovered
Toggle persistence - setting saves correctly across restarts

Screenshots/Videos (if applicable)

Toggle off

Successfully loaded

Load failed

AI Assistance

No AI was used in this PR
AI was used (please describe below)

If AI was used:

Tools used: Claude Code
How extensively: Wrote most of the code and handled conflict resolution, 100% human-reviewed

Enable automatic discovery of custom GGML-format Whisper models (.bin files) placed in the models directory, so users don't need to modify source code to use their own fine-tuned models. Backend changes: - Add discover_custom_whisper_models() to scan for .bin files - Generate display names from filenames (e.g., "my-model" → "My Model") - Skip predefined model filenames to avoid duplicates - Add 3 unit tests for discovery logic Frontend changes: - Split model dropdown into 3 sections: Custom, Downloaded, Downloadable - Add collapsible section for downloadable models to reduce clutter - Add max-height with scroll for long model lists - Add "Custom" badge for user-provided models

[why] Address maintainer concern about support burden from community models. Custom models should be a power-user feature, not enabled by default. [how] Add custom_models_enabled setting (default: false) in Debug settings. Discovery only runs when enabled. Models show "Not officially supported" messaging. Documentation updated with enable steps.

cjpais · 2026-01-20T02:30:18Z

I see you made some tweaks to the UI as well. You might wanna wait for #478! It will probably come in first and then I'll probably be pulling this in. That PR and probably subsequent PRs are going to be changing this bit of UI quite a bit.

cjpais · 2026-02-08T04:19:36Z

@CJHwong mind refactoring this code, theres a new model ui

VirenMohindra

nice feature - the rust backend for model discovery is well-designed. discover_custom_whisper_models() function handles edge cases well (hidden files, non-.bin files, directories, predefined filenames). tests are a solid addition too!

that said, main has changed significantly since this PR was created as CJ mentioned (#478 merged a new models settings page). here's what needs updating for the rebase~

won't compile on current main
ModelInfo now has 3 additional required fields. discover_custom_whisper_models function needs to set these~

supports_translation: false,
is_recommended: false, 
supported_languages: vec![],  // or populate with whisper's supported languages

frontend likely needs a full rewrite
ModelDropdown.tsx was completely overhauled by by PR (sorry 😬 ) - it's now a simple dumb component that only shows downloaded models. all download management moved to a new ModelsSettings page

for the rebase i'd suggest~

ModelDropdown: just add custom models alongside regular downloaded models with a "custom" badge. keep it minimal and simple
ModelsSettings: add a "custom models" section here (this is where users manage models now)

bindings.ts should be regenerated

the rust backend logic is sound - main work is adapting the frontend to the new UI architecture. happy to help if you have questions about the new component structure

VirenMohindra · 2026-02-08T06:46:45Z

src/i18n/locales/en/translation.json

    "recommended": "Recommended",
    "download": "Download",
    "downloading": "Downloading...",
+    "customModelDescription": "Not officially supported",


missing i18n keys for all new 15 non-English locales (i18next falls back to english, so not a user-crash, but bun run check:translations will fail CI)

Added all 5 new keys to all 16 locales with proper translations. Tried my best validating them through Google Translate and LLM.

VirenMohindra · 2026-02-08T06:47:02Z

src-tauri/src/managers/model.rs

+                    accuracy_score: 0.75,
+                    speed_score: 0.75,


the hardcoded is misleading for unknown models - think we should hide the scores or using a sentinel

Good call. Changed to 0.0 sentinel for both scores. ModelCard now hides score bars entirely when both are 0, and also hides language tags when supported_languages is empty.

VirenMohindra · 2026-02-08T06:47:17Z

src/components/model-selector/ModelDropdown.tsx

+  const downloadedModels = models.filter(
+    (m) => m.is_downloaded && m.url !== null,
+  );
+  const customModels = models.filter((m) => m.url === null);


this check to identify custom models is fragile - a dedicated is_custom field on ModelInfo would be more robust

Yep, way cleaner.

VirenMohindra · 2026-02-08T06:47:33Z

src-tauri/src/managers/model.rs

+    /// Discover custom Whisper models (.bin files) in the models directory.
+    /// Skips files that match predefined model filenames.
+    fn discover_custom_whisper_models(
+        models_dir: &PathBuf,


change &PathBuf to &Path in function signature per rust convention

cjpais · 2026-02-08T08:10:49Z

Hey @CJHwong fwiw, I added breeze asr support directly, it'll be out in the next version

Brings in PR cjpais#478 (models settings page), Breeze ASR model, paste delay setting, download cancellation, and other changes from main. Resolves conflicts keeping custom model discovery functionality while adopting main's new model UI.

…w UI [why] PR review requested an explicit is_custom field instead of inferring custom status from url === null. Custom models also need proper integration with the new models settings page from PR cjpais#478. [how] - Add is_custom: bool to ModelInfo struct, set true on discovered models - Change discover_custom_whisper_models signature from &PathBuf to &Path - Use 0.0 sentinel scores so UI hides score bars for custom models - Add "Custom Models" section to ModelsSettings with 3-way model split - Show "Custom" badge in ModelCard and ModelDropdown - Hide language/translation tags when supported_languages is empty - Remove custom models from available_models on delete instead of just marking as not downloaded (they have no re-download URL) - Update tests with new fields and assertions

[why] Custom model feature introduces 5 new translation keys that need to be present in all 15 non-English locales for CI to pass. [how] Add English placeholder values for: customModelDescription, modelSelector.custom, settings.models.customModels, settings.debug.customModels.label, settings.debug.customModels.description

[why] The language filter was inside a conditionally rendered section that disappeared when no downloaded models matched the selected language, leaving the user stuck with no way to change the filter. [how] Always render the "Your Models" header row with the language filter, only conditionally render the model cards below it.

Model selector → Models settings page.

CJHwong · 2026-02-08T10:49:42Z

@VirenMohindra Thanks for the thorough review! All three points addressed:

Merged main and added the 3 missing fields (supports_translation, is_recommended, supported_languages) to custom model construction.
Fully integrated with the new models page. I added a dedicated "Custom Models" section in ModelsSettings, and a "Custom" badge in ModelDropdown. Your new models settings page is genuinely great work btw, made it really easy to slot custom models in.
Bindings updated with is_custom on ModelInfo, custom_models_enabled on AppSettings, and changeCustomModelsEnabledSetting command.

Also btw, fixed a pre-existing bug where the language filter disappears when no models match the selected language due to the filter being nested inside the downloaded models section, so it vanished along with it.

CJHwong · 2026-02-08T11:35:32Z

Hey @CJHwong fwiw, I added breeze asr support directly, it'll be out in the next version

Thanks for adding Breeze natively!

VirenMohindra · 2026-02-08T16:08:29Z

src-tauri/src/shortcut/mod.rs

+    settings::write_settings(&app, settings);
+
+    Ok(())
+}


bug: app breaks on restart when custom model was selected. if the user has a custom model selected and toggles this off, selected_model in settings still points to the custom model. on restart, discovery is skipped so the model doesn't exist → Failed to load model: Model not found and transcription is completely broken until the user manually selects another model.

suggested fix: when disabling, check if the currently selected model is custom. if so, reset selected_model to an empty string (which triggers auto_select_model_if_needed to pick a valid default on next startup):

if !enabled { let models = app.state::<ModelManager>().get_available_models(); if let Some(model) = models.iter().find(|m| m.id == settings.selected_model) { if model.is_custom { settings.selected_model = String::new(); } } }

Fixed. The toggle-off path now resets selected_model when it's a custom model. Also hardened auto_select_model_if_needed to clear any stale selection that doesn't exist in available_models, which covers the case where a custom model file is deleted from disk too.

VirenMohindra · 2026-02-08T16:08:29Z

src-tauri/src/shortcut/mod.rs

+    let mut settings = settings::get_settings(&app);
+    settings.custom_models_enabled = enabled;
+    settings::write_settings(&app, settings);
+


bug: custom models remain visible in UI after toggle-off (without restart). toggling off only persists the setting — it doesn't emit a model-state-changed event or remove custom models from the ModelManager's cached available_models. the frontend never refreshes.

suggested fix: after write_settings(), either:

remove custom models from the available_models mutex and emit model-state-changed so the frontend refreshes, or

at minimum emit the event and have get_available_models() filter based on the current custom_models_enabled setting

Fixed. Toggle now adds/removes custom models from the mutex and emits model-state-changed so the UI refreshes instantly.

when the toggle is on but no custom .bin files exist in the models directory, the "Custom Models" section is either empty or hidden entirely with no guidance

think we should add a small info box / empty state in ModelsSettings.tsx that tells the user where to place their custom model files (e.g. "place .bin whisper models in ~/Library/Application Support/com.pais.handy/models/") so they're not left wondering why nothing shows up

VirenMohindra · 2026-02-08T16:16:40Z

separately, @edwche10 wanna take a look at the custom models UI? this PR adds opt-in custom whisper model discovery (behind a debug toggle). a few design questions~

custom models are marked with a CUSTOM label inline with the model name in the dropdown. does this work or should we delineate differently?
section order is now Downloaded -> Custom -> Available to Download. custom models are local so they sit alongside downloaded models rather than available ones. does this work?
should we surface anything on the card to indicate user-sourced models? e.g. a reference link to where they got the model
custom models currently bypass the language filter (since we don't know what languages they support). is that fine for now, or should we eventually let users tag their custom models with metadata (languages, accuracy, etc) via some UI workflow? currently as it stands in the PR today we would need to drop the custom model directly inside the models/ directory which is not the best UI

and finally maybe premature but: should we add a new filter section on the models page to filter against custom models? there's only ever going to be a handful of custom models, a filter section feels like overkill for a debug feature. i'd prob say YAGNI for now

[why] Two bugs reported in PR review: (1) app breaks on restart when a custom model was selected and the toggle is disabled — selected_model still points to the missing model, causing "Model not found" error. (2) Custom models remain visible in the UI after toggle-off because the in-memory model list is never updated and no event is emitted. [how] - Add remove_custom_models() and add_custom_models() to ModelManager for runtime mutation of the available_models mutex - On disable: reset selected_model to empty if it's a custom model, then remove custom models from the in-memory list - On enable: run discover_custom_whisper_models against the mutex - Emit model-state-changed event so the frontend refreshes immediately

Toggle now takes effect immediately, so the restart sentence is inaccurate. Updated all 16 locales and README instructions.

[why] If a custom model file is deleted from disk while it's the selected model, the app gets stuck on "Loading..." forever on next launch because the model ID is not in available_models but auto_select_model_if_needed only checked for empty string. [how] Validate that selected_model exists in available_models before accepting it. If not found, clear the selection so auto-select picks a valid downloaded model.

VirenMohindra

LGTM! tested on MacOS and works well!

custom models with supported_languages: vec![] will get filtered out when the language filter is set to anything other than "all". the filteredModels memo filters by model.supported_languages.includes(languageFilter), which returns false for an empty array

this means if a user selects e.g. "English" in the filter, their custom models disappear. not a blocker, but worth noting custom models could arguably bypass the language filter since their capabilities are unknown

CJHwong added 2 commits January 20, 2026 10:20

CJHwong force-pushed the feat/custom-whisper-model-discovery branch from 1c99e66 to 294f455 Compare January 20, 2026 02:28

VirenMohindra reviewed Feb 8, 2026

View reviewed changes

CJHwong added 5 commits February 8, 2026 17:05

docs: fix custom models section reference in README

23a0353

Model selector → Models settings page.

VirenMohindra suggested changes Feb 8, 2026

View reviewed changes

CJHwong added 3 commits February 9, 2026 01:21

chore(i18n): remove "restart required" from custom model descriptions

e023fc8

Toggle now takes effect immediately, so the restart sentence is inaccurate. Updated all 16 locales and README instructions.

VirenMohindra approved these changes Feb 8, 2026

View reviewed changes

cjpais added 2 commits February 9, 2026 10:10

remove toggle and clean up

8ec52fc

format

251bbcb

cjpais merged commit a499916 into cjpais:main Feb 9, 2026
4 checks passed

Uh oh!

Conversation

CJHwong commented Jan 19, 2026

Before Submitting This PR

Human Written Description

Related Issues/Discussions

Community Feedback

Testing

Screenshots/Videos (if applicable)

AI Assistance

Uh oh!

cjpais commented Jan 20, 2026

Uh oh!

cjpais commented Feb 8, 2026

Uh oh!

VirenMohindra left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cjpais commented Feb 8, 2026

Uh oh!

CJHwong commented Feb 8, 2026

Uh oh!

CJHwong commented Feb 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VirenMohindra commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VirenMohindra left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

VirenMohindra left a comment •

edited

Loading

VirenMohindra commented Feb 8, 2026 •

edited

Loading

VirenMohindra left a comment •

edited

Loading