Skip to content

refactor(vision): migrate Inception classifiers to Task SDK - #1334

Merged
yifeif-nv merged 1 commit into
NVIDIA:mainfrom
yifeif-nv:refactor/task-sdk-vision-inception
Sep 18, 2026
Merged

yifeif-nv merged 1 commit into
NVIDIA:mainfrom
yifeif-nv:refactor/task-sdk-vision-inception

Conversation

@yifeif-nv

Copy link
Copy Markdown
Collaborator

Background

Migrate Inception, Inception-v4 and Inception-ResNet to the existing image-class-score Task SDK.
These three independent families use the same contract, so they are grouped in
one reviewable migration PR. Every changed file belongs to one of:

  • families/timm_inception/**
  • families/timm_inception_v4/**
  • families/timm_inception_resnet/**

This follows the shared SDK in #1226 and the merged MobileNetV3 pattern in #1308.
It does not introduce a shared model implementation or a dependency between families.

Exit Criteria

  • Each family implements IModel and IImageToClassScores, with its own bind
    declaration, correct bundle primary Task, and complete owned logits.
  • Preserve each graph, weights, preprocessing, declared cases and original oracle;
    do not replace a family's existing acceptance rule with a weaker common check.
  • Pass local CPU and the selected official single-device E2Es. Current-head remote
    checks remain required before merge; this submission does not claim Pre-merge green.

Implementation

  • Switch support, builder metadata and owned manifests from classification to
    image_to_class_scores; no legacy alias or fallback execution.
  • Return all class logits in checkpoint order. Only checkpoint-provided labels
    and vocabulary identity are published; absent identity stays empty. No hashes
    or invented cross-model IDs.
  • Keep each model's computation and image preprocessing unchanged. Add family-local
    binding, input/output, metadata and result-lifetime CPU contracts.
  • Build public C11/C++17 consumers through the existing family CTest dependency
    mechanism. The existing E2Es call both SDKs with identical decoded RGB pixels,
    compare every score, and read results after model release. Original JPEG CLI
    and reference paths remain; different decoders are not claimed bitwise identical.
  • Reuse the existing family-owned performance declaration/script protocol.
    Preserve entry IDs, workloads, precision, 3/10 sampling and 5% margin.
    No shared header, registry, CLI, benchmark implementation, website or workflow edit.

Change categories

  • Model or runtime behavior
  • Public API
  • ABI
  • Bundle or artifact format
  • Dependencies
  • Documentation only
  • CI or developer tooling

Public C layouts and bundle format 1 are unchanged. Primary Task and family
runtime metadata change, so rebuild existing bundles for these family runtimes.

Validation

Commands and Results

With the existing native dependencies and E2E environment:

  • cmake --build build --target trtmc trtmc_backend_trt trtmc_benchmark_worker test_timm_inception_task_contract test_timm_inception_image_preprocess test_timm_inception_v4_task_contract test_timm_inception_v4_image_preprocess test_timm_inception_resnet_task_contract test_timm_inception_resnet_image_preprocess:
    passed, including the six public SDK consumer binaries.
  • ctest --test-dir build --output-on-failure -R '^(timm_inception|timm_inception_v4|timm_inception_resnet)_(task_contract|image_preprocess)$':
    6/6 CPU tests passed.
  • python -m pytest families/timm_inception/tests families/timm_inception_v4/tests families/timm_inception_resnet/tests -m 'not gpu and not trt and not e2e' -q:
    95 passed; 3 explicitly unselected E2E cases skipped.
  • python -m pytest families/timm_inception/tests/test_e2e.py families/timm_inception_v4/tests/test_e2e.py families/timm_inception_resnet/tests/test_e2e.py --e2e-testcase inception-v3-tv-in1k --e2e-testcase inception-v4-tf-in1k --e2e-testcase inception-resnet-v2-tf-in1k -q:
    3 passed (three actual checkpoint cases), including C/C++ calls.
  • python3 -m tools.community_ci source-quality --base 9d65d3bf0022255237df36727a3a2b3f6de3a5d4:
    legal, inventory, complexity, formatting and 212 contract tests passed.
  • git diff --check: passed.

The existing native worker and owned references executed all three original
3-warmup / 10-measurement workloads and passed the unchanged classification
output contracts. Each native/reference result retained all 1000 scores.
This is execution/output evidence, not a latency improvement claim.

Hardware, Environment, and Revisions

  • Base: 9d65d3bf0022255237df36727a3a2b3f6de3a5d4.
  • Tested branch contents: 9b4d709a810e2bf3f9ea9b396f24d17c4169e03e.
  • Linux/aarch64, single-device NVIDIA GB300 execution per case, CUDA toolkit
    13.3.33, TensorRT 11.1.0.106, PyTorch 2.12.0+cu130 and timm 1.0.28. Selected
    model cases use their existing FP16 manifest settings and checked-in images.
Selected checkpoint Resolved tested revision
timm/inception_v3.tv_in1k 393d84cc85c467d8fbc0dc81a65c04e87a32572c
timm/inception_v4.tf_in1k ad5d294cda312745a9afa433e45d9ccd956a1548
timm/inception_resnet_v2.tf_in1k 836b07d7d599da247de92f148c7a07fda18afced

Not Run / Remaining Gaps

  • No multi-device execution, full Nightly, complete package qualification or
    all-profile GPU qualification. Unselected cases remain declared.
  • No speedup claim. The existing reference timing policy excludes preprocessing,
    while native public Task timing includes family preprocessing; that scope
    distinction remains documented, not declared equivalent.
  • Remote Pre-merge has not yet passed when this PR is opened.

Contributor Self-Review

  • I have completed a self-review of this change.

Reviewed ownership, retained model behavior, test oracles and public evidence.
Independent reviewers checked the exact changes and publication text.

Notes For Future Readers

This branch starts directly from the shared mainline base, not another migration
batch. The three families do not import one another or MobileNetV3; intentional
local duplication preserves independent ownership. Use the existing
TRTMC_NATIVE_BUILD_DIR for development SDK consumer binaries. Core/backend/family
DSOs upgrade together, and old bundles must be rebuilt. Obsolete shared interfaces
will be removed separately after the remaining families migrate.

Risk level

  • Low
  • Medium
  • High

Three family Task identities and runtime metadata change, but model math and
preprocessing are retained and the selected real-model paths were exercised.

Implement family-owned image class-score Tasks, bindings and complete result metadata. Preserve each model graph, preprocessing and existing oracle while exercising public C/C++ consumers and owned benchmark references. No shared implementation or sibling-family dependency is added.

Signed-off-by: yifeif-nv <277870278+yifeif-nv@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Sep 17, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: dc967053-72bb-49a0-9161-8598327f9caa

📥 Commits

Reviewing files that changed from the base of the PR and between 4b74779 and 9b4d709.

📒 Files selected for processing (45)
  • families/timm_inception/model.py
  • families/timm_inception/runtime/CMakeLists.txt
  • families/timm_inception/runtime/pipeline.cpp
  • families/timm_inception/runtime/pipeline.h
  • families/timm_inception/runtime/plugin.cpp
  • families/timm_inception/support.py
  • families/timm_inception/tests/cpp/test_task_contract.cpp
  • families/timm_inception/tests/manifests/inception-v3-tv-in1k.json
  • families/timm_inception/tests/performance.yaml
  • families/timm_inception/tests/performance_reference.py
  • families/timm_inception/tests/sdk_consumer.c
  • families/timm_inception/tests/sdk_consumer.cpp
  • families/timm_inception/tests/test_e2e.py
  • families/timm_inception/tests/test_model.py
  • families/timm_inception/tests/test_performance_reference.py
  • families/timm_inception_resnet/model.py
  • families/timm_inception_resnet/runtime/CMakeLists.txt
  • families/timm_inception_resnet/runtime/pipeline.cpp
  • families/timm_inception_resnet/runtime/pipeline.h
  • families/timm_inception_resnet/runtime/plugin.cpp
  • families/timm_inception_resnet/support.py
  • families/timm_inception_resnet/tests/cpp/test_task_contract.cpp
  • families/timm_inception_resnet/tests/manifests/inception-resnet-v2-tf-in1k.json
  • families/timm_inception_resnet/tests/performance.yaml
  • families/timm_inception_resnet/tests/performance_reference.py
  • families/timm_inception_resnet/tests/sdk_consumer.c
  • families/timm_inception_resnet/tests/sdk_consumer.cpp
  • families/timm_inception_resnet/tests/test_e2e.py
  • families/timm_inception_resnet/tests/test_model.py
  • families/timm_inception_resnet/tests/test_performance_reference.py
  • families/timm_inception_v4/model.py
  • families/timm_inception_v4/runtime/CMakeLists.txt
  • families/timm_inception_v4/runtime/pipeline.cpp
  • families/timm_inception_v4/runtime/pipeline.h
  • families/timm_inception_v4/runtime/plugin.cpp
  • families/timm_inception_v4/support.py
  • families/timm_inception_v4/tests/cpp/test_task_contract.cpp
  • families/timm_inception_v4/tests/manifests/inception-v4-tf-in1k.json
  • families/timm_inception_v4/tests/performance.yaml
  • families/timm_inception_v4/tests/performance_reference.py
  • families/timm_inception_v4/tests/sdk_consumer.c
  • families/timm_inception_v4/tests/sdk_consumer.cpp
  • families/timm_inception_v4/tests/test_e2e.py
  • families/timm_inception_v4/tests/test_model.py
  • families/timm_inception_v4/tests/test_performance_reference.py

Included review availability: Your plan provides up to 12 included reviews per hour; 5 remain after this review.


📝 Summary

Summary

This change migrates the Inception, Inception-v4, and Inception-ResNet families from classification to image_to_class_scores.

Each family now:

  • Implements family-owned IModel and IImageToClassScores bindings.
  • Publishes complete checkpoint-order logits.
  • Publishes ScoreKind::Logit, vocabulary identity, and labels.
  • Validates class counts, labels, input layout, tensor type, output shape, and runtime configuration.
  • Preserves model graphs, preprocessing, declared cases, and acceptance oracles.
  • Updates support metadata, manifests, and bundle runtime metadata.

The change adds CPU task-contract tests, public C and C++ SDK consumers, E2E checks, and performance reference coverage. Performance workloads, sampling, margins, and timing boundaries remain unchanged.

Architecture impact

Family-owned files

The implementation remains within the three owning family directories:

  • families/timm_inception/
  • families/timm_inception_v4/
  • families/timm_inception_resnet/

The family directories contain model metadata, runtime pipelines, plugins, contract tests, SDK consumers, E2E tests, manifests, and performance references.

Shared surfaces

No shared implementation, registry, CLI, benchmark, website, or workflow changes are reported.

The new runtime code consumes existing internal task contracts. It does not add family-specific behavior to shared code.

Dependency directions

The dependency direction remains:

  • Family runtime → existing internal task interfaces.
  • Family tests → family runtime and public SDK APIs.
  • SDK consumers → public trtmc_c or C++ SDK APIs.
  • Performance references → model and benchmark interfaces.

No cross-family implementation or validation dependency is reported.

Affected consumers

The affected consumers include:

  • Bundle builders using the three family builders.
  • Runtime loaders using the three family plugins.
  • Manifest-driven tests.
  • Family C and C++ SDK consumers.
  • Family E2E tests.
  • Family performance references.

Existing consumers that request classification for these families require migration to image_to_class_scores.

Unresolved blast-radius questions

Existing bundles require rebuilding because the primary task and family runtime metadata changed.

The supplied evidence does not establish completion of multi-device execution, full Nightly, complete package qualification, all-profile GPU qualification, or remote Pre-merge checks.

Review finding counts are unavailable because no current review findings were supplied.

Validation

Reported validation includes:

  • Six CPU contract and preprocessing tests.
  • 95 non-GPU, non-TRT, non-E2E tests.
  • Three selected single-device E2E tests.
  • Source-quality checks.
  • git diff --check.

Outcome

HUMAN REVIEW REQUIRED

The implementation and family ownership evidence are clear, but full compatibility and qualification remain unresolved. Human review must confirm migration impact for existing classification consumers and complete the pending qualification runs.

Walkthrough

The three Timm image families now use the image_to_class_scores task. They persist class metadata, execute structured image requests, return raw logits with metadata, and add native contract, SDK, end-to-end, and performance validation.

Changes

Timm Inception

Layer / File(s) Summary
Task and bundle contract
families/timm_inception/model.py, families/timm_inception/support.py, families/timm_inception/tests/manifests/..., families/timm_inception/tests/test_model.py
Build and support contracts now use image_to_class_scores. Bundle metadata includes validated class count, vocabulary ID, and labels.
Request-based runtime execution
families/timm_inception/runtime/...
The pipeline now accepts structured image requests, rejects unsupported configuration and invalid logits, and returns copied logit scores with class metadata.
Contract, SDK, and performance validation
families/timm_inception/tests/...
Added native contract tests, C and C++ SDK consumers, end-to-end metadata checks, and performance reference validation.

Timm Inception-ResNet

Layer / File(s) Summary
Task and bundle contract
families/timm_inception_resnet/model.py, families/timm_inception_resnet/support.py, families/timm_inception_resnet/tests/...
Build and support contracts now use image_to_class_scores. Bundle metadata includes validated class count, vocabulary ID, and labels.
Request-based runtime execution
families/timm_inception_resnet/runtime/...
The pipeline now requires the named logits output, validates structured image requests, and returns logit scores with class metadata.
Contract, SDK, and performance validation
families/timm_inception_resnet/tests/...
Added native contract tests, C and C++ SDK consumers, end-to-end metadata checks, and performance reference validation.

Timm Inception-v4

Layer / File(s) Summary
Task and bundle contract
families/timm_inception_v4/model.py, families/timm_inception_v4/support.py, families/timm_inception_v4/tests/...
Build and support contracts now use image_to_class_scores. Bundle metadata includes validated class count, vocabulary ID, and labels.
Request-based runtime execution
families/timm_inception_v4/runtime/...
The pipeline now requires the named logits output, validates structured image requests, and returns logit scores with class metadata.
Contract, SDK, and performance validation
families/timm_inception_v4/tests/...
Added native contract tests, C and C++ SDK consumers, end-to-end metadata checks, and performance reference validation.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant SDK
  participant FamilyPlugin
  participant ImageToClassScoresPipeline
  participant TensorRT
  SDK->>FamilyPlugin: Load bundle and submit image request
  FamilyPlugin->>ImageToClassScoresPipeline: Provide runtime metadata
  ImageToClassScoresPipeline->>TensorRT: Execute preprocessed RGB tensor
  TensorRT-->>ImageToClassScoresPipeline: Return logits
  ImageToClassScoresPipeline-->>SDK: Return scores, vocabulary ID, and labels
Loading

Suggested reviewers: zhenshanx-nv

Merge Risk: ⚪ Minimal · up to 9b4d7

No concrete current-head issue remains; the migrated classification path emits the expected score payload.

🚥 Pre-merge checks | ✅ 7 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 6.49% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 231 functions across 36 files. (9 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
Benchmark Validation Integrity ⚠️ Warning The new benchmark comparisons do not measure equivalent regions. Each new performance_reference.py prepares the image once before timing (inputs = transform(...).to(device="cuda")), then times onl… Use one measurement contract for both paths. The simplest fix is to preload only the image asset, then perform the same resize/normalize and device transfer inside every timed reference invocation. Declare `timing_scope: task-pipeline-call-…
✅ Passed checks (7 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: migrating the three Inception classifier families to the Task SDK.
Description check ✅ Passed The description is complete and follows the required template. It covers background, exit criteria, implementation, change categories, validation results, environment and revisions, remaining gaps, se…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Family Ownership Boundary ✅ Passed No family-ownership violation was introduced. The authoritative diff changes only files under the three owning family roots, with no central registry or source-list file changed. Exact sibling-family …
Shared Semantic Neutrality ✅ Passed PASS — The pull request changes 45 files, and every changed path is under one of the three family-owned trees: families/timm_inception, families/timm_inception_v4, or `families/timm_inception_resn…
Shared Change Blast Radius ✅ Passed PASS. The authoritative diff changes 45 files, and every path is under families/timm_inception, families/timm_inception_v4, or families/timm_inception_resnet. It does not modify shared SDK heade…
Full details: Docstring Coverage

Explanation

Docstring coverage is 6.49% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 231 functions across 36 files. (9 skipped: 9 unsupported.)

Full details: Benchmark Validation Integrity

Explanation

The new benchmark comparisons do not measure equivalent regions. Each new performance_reference.py prepares the image once before timing (inputs = transform(...).to(device="cuda")), then times only model inference and host-logit materialization. The changed Inception pipelines perform preprocessing inside run, which the native worker measures inside public_task_call_wall. The PR description confirms this scope difference. The reference also validates shape and finiteness in _summary after timing, while each pipeline validates logits presence, dtype, and class count before returning the timed result. The new family-owned performance files activate these paths for timm_inception.classify, timm_inception_v4.classify, and timm_inception_resnet.classify; the base had only shared adapter entries for these IDs.

Resolution

Use one measurement contract for both paths. The simplest fix is to preload only the image asset, then perform the same resize/normalize and device transfer inside every timed reference invocation. Declare timing_scope: task-pipeline-call-wall and input_preparation_included: true for all three references. Align output validation at the timing boundary as well: validate the same logits presence, dtype, shape, and class-count conditions before the reference invocation ends, and keep reduction, finite/reporting checks, and JSON serialization outside timing (or move the equivalent checks to both implementations). Re-run the three benchmark references and candidate workloads after the contract change.


Comment @coderabbitai help to get the list of available commands.

@yifeif-nv yifeif-nv added the run-internal-ci Maintainer-approved dispatch to internal CI label Sep 17, 2026
@github-actions github-actions Bot removed the run-internal-ci Maintainer-approved dispatch to internal CI label Sep 17, 2026
@yifeif-nv
yifeif-nv merged commit 051bd97 into NVIDIA:main Sep 18, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant