Skip to content

refactor(timm_mobilenetv3): migrate to Task SDK - #1308

Merged
yifeif-nv merged 2 commits into
NVIDIA:mainfrom
yifeif-nv:refactor/mobilenetv3-task-sdk
Sep 17, 2026
Merged

yifeif-nv merged 2 commits into
NVIDIA:mainfrom
yifeif-nv:refactor/mobilenetv3-task-sdk

Conversation

@yifeif-nv

@yifeif-nv yifeif-nv commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Background

Migrate timm MobileNetV3 to the semantic Task SDK introduced by #1226.
Every changed file is under families/timm_mobilenetv3/; the shared C ABI,
C++ wrapper, CLI, benchmark and CI discovery already support this Task.

Exit Criteria

  • The family implements and advertises image_to_class_scores, returning every
    original logit in checkpoint class order through the C and C++ SDKs.
  • Preserve the original graph, preprocessing and official timm top-1 oracle.
  • Family CPU and single-device E2E checks pass. Required Pre-merge checks must
    pass before merge; local results are not a remote Pre-merge claim.

Implementation

  • Implement IModel and IImageToClassScores; the family returns its own
    bind record with no optional runtime Config fields.
  • Update support, builder and the owned manifest to image_to_class_scores.
    Return complete owned logits, not a top-k subset or softmax probabilities.
  • Carry class count and checkpoint-provided labels/vocabulary identity in the
    family's runtime metadata. Missing identity stays empty; no hash, invented
    name or local checkpoint path is used as a cross-model identity.
  • Add CPU tests for binding, input shape, preprocessing, complete result ownership,
    known/unknown class identity and rejection of unsupported Config.
  • Add direct public C11 and C++17 consumers to the existing single-device E2E.
    Both read the result after releasing the model and must agree on every score
    for identical RGB pixels. The original JPEG CLI path and timm top-1 check
    remain unchanged; different JPEG decoders are not claimed bitwise equivalent.
  • Existing family CTest discovery builds the consumers through a target
    dependency. No shared registry, workflow or new environment variable is added.

Follow-up: family-owned benchmark timing

The migration moved native argmax outside the timed Task call, while the old
reference still timed argmax and summary generation. The owning family now
provides tests/performance.yaml and tests/performance_reference.py through
the existing reference-script mechanism. Model inference, full host float32
scores and synchronization are timed; classification and reporting are not.

The original entry ID, workload, FP16 policy, 3 warmups, 10 measurements, 5%
margin and top-class oracle are preserved. The existing reference policy still
excludes input preparation whereas the native Task includes family preprocessing;
this change does not claim to remove that pre-existing scope difference or
establish a performance improvement. No shared file changes.

Change categories

  • Model or runtime behavior
  • Public API
  • ABI
  • Bundle or artifact format
  • Dependencies
  • Documentation only
  • CI or developer tooling

Public C layouts and bundle format 1 do not change. The family's primary Task
and runtime metadata do change: rebuild old classification bundles. There is
no old Task alias, compatibility adapter or fallback execution path.

Validation

Commands and Results

The following original migration results belong to
8d317e3e86cef6dda577694ca15aa082610b900f, using the repository's native
dependencies and existing E2E environment. Follow-up validation is listed
separately below; these results are not new-head CI results.

  • cmake --build build --target trtmc trtmc_backend_trt test_timm_mobilenetv3_task_contract test_timm_mobilenetv3_image_preprocess:
    passed, including the public C/C++ consumer binaries.
  • ctest --test-dir build --output-on-failure -R '^timm_mobilenetv3_(task_contract|image_preprocess)$':
    passed, 2/2 CPU tests.
  • python -m pytest families/timm_mobilenetv3/tests -m 'not gpu and not trt and not e2e' -q:
    17 passed; the explicitly unselected real E2E case was skipped.
  • python -m pytest families/timm_mobilenetv3/tests/test_e2e.py --e2e-testcase mobilenetv3-large-100-ra-in1k -q:
    1 real single-device case passed. CLI/C/C++ retain the original top-1 oracle;
    the two SDK consumers agree on all 1000 logits and class metadata.
  • python3 -m tools.community_ci source-quality --base ed98d6affe9e61be98ead407c1fd77e4155d6b75:
    passed legal headers, inventory, complexity, formatting and 212 contract tests.
  • git diff --check: passed. Existing benchmark resolution selects the new
    Task and retains its classify operation without a central suite edit.

Additional checks for the timing follow-up:

  • python3 -m pytest families/timm_mobilenetv3/tests/test_performance_reference.py -q:
    15 passed, including timer ordering, receipt identity and invalid-output checks.
  • The complete family CPU selection above now passes 32 tests, with only the
    explicitly unselected real E2E case skipped.
  • The family reference executed the original single-GPU 3-warmup/10-measurement
    workload on the pinned checkpoint below: completed, 10 timing samples,
    1000 finite scores and top-class 656. This is reference execution evidence,
    not a new paired speedup or full-migration E2E claim.

Hardware, Environment, and Revisions

  • Base: ed98d6affe9e61be98ead407c1fd77e4155d6b75 (feat(api): add semantic Task C ABI and C++ SDK #1226).
  • Original migration E2E/native validation: 8d317e3e86cef6dda577694ca15aa082610b900f.
  • Timing follow-up CPU/reference validation: bd007e2fe3fb859c4244b5efc0fc1f3c6d89a585.
    Production runtime, builder and original E2E sources are unchanged by this follow-up.
  • Linux/aarch64, one NVIDIA GB300 GPU, CUDA toolkit 13.3.33, TensorRT 11.1.0.106,
    PyTorch 2.12.0+cu130 and timm 1.0.28; FP16 as in the original manifest.
  • Checkpoint: timm/mobilenetv3_large_100.ra_in1k revision
    96f46a1c52932f27492dff66c72378eb99b443a7; original checked-in test image.

Not Run / Remaining Gaps

  • No multi-device execution, full Nightly or all-checkpoint qualification.
  • No claim of a performance improvement or cross-decoder bitwise parity.
  • Required Pre-merge must pass on the follow-up head before merge; earlier-head
    results do not qualify this update.

Contributor Self-Review

  • I have completed a self-review of this change.

Reviewed the family-only diff, unchanged preprocessing/reference, complete
outputs and public evidence, with independent technical and publication review.

Notes For Future Readers

#1226 is already merged. This PR is independent of the PatchTSMixer migration
and other families. families/timm_mobilenetv3/README.md describes the new Task,
bundle rebuild and direct SDK validation commands. Runtime and family DSOs
upgrade together; obsolete shared execution interfaces are not removed here.

Risk level

  • Low
  • Medium
  • High

This changes one family's public Task identity and bundle metadata. The original
model/CLI oracle and additional complete SDK-output checks constrain regression.

Implement the image class-score Task with complete owned logits and family-owned metadata. Preserve preprocessing and the original reference oracle, and exercise direct C and C++ consumers in the existing single-device E2E.

Signed-off-by: yifeif-nv <277870278+yifeif-nv@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Sep 16, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 02c68fab-4e58-4c68-a4a8-4a6a32992d47

📥 Commits

Reviewing files that changed from the base of the PR and between 8d317e3 and bd007e2.

📒 Files selected for processing (4)
  • families/timm_mobilenetv3/README.md
  • families/timm_mobilenetv3/tests/performance.yaml
  • families/timm_mobilenetv3/tests/performance_reference.py
  • families/timm_mobilenetv3/tests/test_performance_reference.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • families/timm_mobilenetv3/README.md

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Summary

Summary

Migrates timm_mobilenetv3 from classification to image_to_class_scores.

The runtime returns all checkpoint logits in class order. It publishes class count, vocabulary ID, and labels as family-owned metadata. It validates input, output, and runtime configuration.

The manifest, builder, support declarations, documentation, benchmark reference, and tests now use the semantic task. Direct C11 and C++17 SDK consumers extend single-device E2E coverage. The graph, preprocessing, JPEG CLI path, and top-1 oracle remain unchanged.

Validation included CPU tests, non-GPU/non-TRT/non-E2E tests, one single-device E2E case, source-quality checks, benchmark-reference tests, and git diff --check. Multi-device, full Nightly, and all-checkpoint qualification were not run.

Architecture impact

  • Family-owned files: Model configuration, support declarations, runtime pipeline and plugin, manifest, documentation, benchmark reference, and family tests.
  • Changed shared surfaces: The runtime now implements IImageToClassScores and uses the public C and C++ image-to-class-scores APIs.
  • Dependency directions: The runtime depends on internal semantic task interfaces. The new SDK consumers depend on public task bindings.
  • Affected consumers: Existing classification bundles require rebuilding because task identity and runtime metadata changed.
  • Format impact: Public C layouts and bundle format 1 remain unchanged.
  • Blast-radius questions: Multi-device behavior, full Nightly coverage, and all-checkpoint compatibility remain unqualified.

Review outcome

HUMAN REVIEW REQUIRED

No current review findings were supplied, so severity counts are unavailable. Human review remains required because compatibility and blast-radius questions remain unresolved.

Walkthrough

The MobileNetV3 family now exposes image_to_class_scores inference. Bundles publish class metadata, runtimes return complete logits, and validation covers pipeline contracts, C/C++ SDK consumers, model builds, end-to-end parity, and performance timing.

Changes

MobileNetV3 image-to-class-scores

Layer / File(s) Summary
Task and metadata wiring
families/timm_mobilenetv3/model.py, families/timm_mobilenetv3/support.py, families/timm_mobilenetv3/runtime/plugin.cpp, families/timm_mobilenetv3/tests/manifests/...
The family uses image_to_class_scores. Bundle metadata includes class count, vocabulary ID, and labels. The runtime factory passes this metadata to the pipeline.
Pipeline task contract
families/timm_mobilenetv3/runtime/pipeline.h, families/timm_mobilenetv3/runtime/pipeline.cpp
The pipeline replaces classify with task bindings and run. It validates RGB float32 input, configuration, dimensions, logits, and class counts. It returns owned raw logits with score metadata.
Runtime contract and SDK validation
families/timm_mobilenetv3/runtime/CMakeLists.txt, families/timm_mobilenetv3/tests/cpp/*, families/timm_mobilenetv3/tests/sdk_consumer.*
C++ contract tests and C/C++ SDK consumers validate preprocessing, metadata, logits, configuration, resource handling, and public API execution.
Model, end-to-end, and documentation coverage
families/timm_mobilenetv3/tests/test_model.py, families/timm_mobilenetv3/tests/test_e2e.py, families/timm_mobilenetv3/README.md
Tests cover metadata validation, build output, SDK parity, and task changes. The README documents the runtime and SDK contracts.
Performance reference and timing validation
families/timm_mobilenetv3/tests/performance.yaml, families/timm_mobilenetv3/tests/performance_reference.py, families/timm_mobilenetv3/tests/test_performance_reference.py
The performance suite measures synchronized model-call latency and validates timing boundaries, output scores, protocol settings, and baseline receipts.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant SDKConsumer
  participant TimmMobilenetv3ImageClassificationPipeline
  participant ITrtModule
  SDKConsumer->>TimmMobilenetv3ImageClassificationPipeline: submit image_to_class_scores request
  TimmMobilenetv3ImageClassificationPipeline->>ITrtModule: preprocess input and run inference
  ITrtModule-->>TimmMobilenetv3ImageClassificationPipeline: return float32 logits
  TimmMobilenetv3ImageClassificationPipeline-->>SDKConsumer: return LabelScoresResult with logits and metadata
Loading

Merge Risk: ⚪ Minimal · up to bd007

The new benchmark executes and validates its family script independently of the inert backend label, so no current merge-blocking risk is identified.

🚥 Pre-merge checks | ✅ 7 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 6.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 79 functions across 12 files. (2 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
Benchmark Validation Integrity ⚠️ Warning The benchmark adds asymmetric output validation. The native candidate times TimmMobilenetv3ImageClassificationPipeline::run() inside public_task_call_wall; that call performs require_logits() va… Align output validation at the measurement boundary. Either move candidate-only output checks outside the timed task call, or add equivalent shape, dtype, class-count, and validity checks to the reference timed invocation before it returns.…
✅ Passed checks (7 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: migrating the timm MobileNetV3 family to the Task SDK.
Description check ✅ Passed The description completes the required sections with background, exit criteria, implementation details, change categories, validation results, environment revisions, remaining gaps, self-review, notes…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Family Ownership Boundary ✅ Passed PASS. The authoritative diff changes only families/timm_mobilenetv3/ (16 files); no other family, central registry, switch, source list, or strategy map changes. Changed code references only self-ow…
Shared Semantic Neutrality ✅ Passed PASS: The authoritative PR inventory contains only files under families/timm_mobilenetv3/. The changes are limited to the model/support Python, runtime implementation and CMake files, family tests a…
Shared Change Blast Radius ✅ Passed The authoritative PR diff changes only families/timm_mobilenetv3/**; it changes no shared code, contract, catalog, workflow, or repository-level tooling file. The added CMake targets, SDK consumers,…
Full details: Docstring Coverage

Explanation

Docstring coverage is 6.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 79 functions across 12 files. (2 skipped: 2 unsupported.)

Full details: Benchmark Validation Integrity

Explanation

The benchmark adds asymmetric output validation. The native candidate times TimmMobilenetv3ImageClassificationPipeline::run() inside public_task_call_wall; that call performs require_logits() validation and checks the configured class count before returning. The new reference times model execution and host-score conversion, then performs shape and finite-score validation in _summary() after the timer. The reference also coerces the output to float32 instead of validating the original output dtype. Therefore output validation is included in the timed region on the candidate side but excluded on the reference side.

Resolution

Align output validation at the measurement boundary. Either move candidate-only output checks outside the timed task call, or add equivalent shape, dtype, class-count, and validity checks to the reference timed invocation before it returns. Keep reduction, synchronization, host transfer, and serialization placement identical on both paths, then retain tests that verify the event ordering.


Comment @coderabbitai help to get the list of available commands.

Use the existing family-owned reference protocol for the migrated class-score Task. Time inference and complete host scores, then perform classification and reporting outside the timer. Preserve the existing workload, timing declarations and comparison criteria; the pre-existing preprocessing scope difference remains documented.

Signed-off-by: yifeif-nv <277870278+yifeif-nv@users.noreply.github.com>
@yifeif-nv

Copy link
Copy Markdown
Collaborator Author

Review follow-up for bd007e2fe3fb859c4244b5efc0fc1f3c6d89a585:

The family-owned reference now places argmax and reporting after timing, while
timing inference and the complete host float32 result. The original workload,
3/10 sampling, 5% margin and top-class oracle are unchanged.

The remaining warning is narrower than a failed inference/parity result:

  • Finite-value checks are outside timing on both paths: the native worker runs
    json_values in its observer, and the reference runs _summary after _measure.
  • Float32 conversion is intentional. The native builder also casts engine output
    to float32; requiring the reference's raw tensor dtype to equal that transport
    dtype would incorrectly conflate execution precision with the public result.
  • The reference builds the model with checkpoint num_classes and loads weights
    strictly, but does not repeat native's explicit output class-count guard.
    That additional defensive check is deferred as a non-blocking follow-up, not
    claimed implemented. The real reference run returned all 1000 finite scores
    and top class 656. No claim is made that metadata/ABI overhead is identical
    across implementations.

The existing preprocessing timing-scope difference remains documented in the
family README. No numerical criterion or required test has been waived; merge
readiness still requires the current head's protected Pre-merge gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant