Add on prem support to SDK by cnovel · Pull Request #308 · iTwin/reality-capture

cnovel · 2026-06-29T12:08:16Z

No description provided.

Copilot

Pull request overview

This PR adds an on‑premise (“job queue DB”) integration surface to the Python SDK, introducing managers/models for reading and mutating local JobQueue/Engine SQLite databases, plus a test suite and fixture data to validate behavior.

Changes:

Added on‑prem on-premise SDK modules (JobManager, EngineManager, models, and Result error handling) and corresponding tests/fixtures.
Centralized shared job enums/progress/execution models into reality_capture.common.job and reused them from the service job model.
Relaxed several specification field validations (removed ^bkt: regex patterns) to accommodate non-bucket/on‑prem paths.

Reviewed changes

Copilot reviewed 46 out of 48 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
python_sdk/tests/test_onprem_jobqueue.py	Adds JobManager integration tests (submit/get/progress/summary/pagination/cancel + error paths).
python_sdk/tests/test_onprem_job_validator.py	Validates on‑prem `Job` specification type dispatch for many job types.
python_sdk/tests/test_onprem_generic_manager.py	Tests manager init failures and cross-platform read-only folder behavior.
python_sdk/tests/test_onprem_engine.py	Adds EngineManager tests for read/list/get/send_signal + lock/read-only paths.
python_sdk/tests/data/DB_Jobs/JobQueue.db	Git LFS pointer for job queue DB fixture.
python_sdk/tests/data/DB_Jobs/Engines.db	Git LFS pointer for engines DB fixture.
python_sdk/tests/data/DB_Jobs/jobs/job_faa0be51-3f34-452b-a4ea-885ebe9015b3/settings.json	Job settings fixture (valid JSON).
python_sdk/tests/data/DB_Jobs/jobs/job_dec6d7f0-aa2a-4d96-9f26-2f1fa7faa5d5/settings.json	Job settings fixture (valid JSON).
python_sdk/tests/data/DB_Jobs/jobs/job_d4fc6fa0-8c99-4dff-a7cd-4deb930c4d78/settings.json	Job settings fixture (valid JSON).
python_sdk/tests/data/DB_Jobs/jobs/job_8aaaffdb-e1a2-451e-97de-9fb2cd5ef8b6/settings.json	Job settings fixture (valid JSON).
python_sdk/tests/data/DB_Jobs/jobs/job_6448e60c-a22a-44f7-819e-1a71068feebd/settings.json	Job settings fixture (valid JSON).
python_sdk/tests/data/DB_Jobs/jobs/job_12083864-05aa-4a5a-b183-6242fec8bfa5/settings.json	Job settings fixture (valid JSON).
python_sdk/tests/data/DB_Jobs_Corrupted/JobQueue.db	Git LFS pointer for corrupted job queue DB fixture.
python_sdk/tests/data/DB_Jobs_Corrupted/Engines.db	Git LFS pointer for corrupted engines DB fixture.
python_sdk/tests/data/DB_Jobs_Corrupted/jobs/job_faa0be51-3f34-452b-a4ea-885ebe9015b3/settings.json	Job settings fixture (schema-corrupted JSON).
python_sdk/tests/data/DB_Jobs_Corrupted/jobs/job_dec6d7f0-aa2a-4d96-9f26-2f1fa7faa5d5/settings.json	Job settings fixture (valid JSON for other corruption scenarios).
python_sdk/tests/data/DB_Jobs_Corrupted/jobs/job_d4fc6fa0-8c99-4dff-a7cd-4deb930c4d78/settings.json	Job settings fixture (valid JSON for other corruption scenarios).
python_sdk/tests/data/DB_Jobs_Corrupted/jobs/job_8aaaffdb-e1a2-451e-97de-9fb2cd5ef8b6/settings.json	Job settings fixture (non-JSON payload).
python_sdk/tests/data/DB_Jobs_Corrupted/jobs/job_12083864-05aa-4a5a-b183-6242fec8bfa5/settings.json	Job settings fixture (valid JSON for other corruption scenarios).
python_sdk/tests/data/DB_Engines/JobQueue.db	Git LFS pointer for engine DB fixture companion.
python_sdk/tests/data/DB_Engines/Engines.db	Git LFS pointer for engines DB fixture.
python_sdk/src/reality_capture/on_premise/_generic_manager.py	Adds shared DB initialization + file-locking utilities.
python_sdk/src/reality_capture/on_premise/engine_manager.py	Adds EngineManager for reading engines and sending signals.
python_sdk/src/reality_capture/on_premise/job.py	Adds on-prem Job/Progress/Summary models + filters/pagination model.
python_sdk/src/reality_capture/on_premise/job_manager.py	Adds JobManager for job CRUD-ish operations, paging, progress, and summary.
python_sdk/src/reality_capture/on_premise/result.py	Adds `Result[T]` and `ManagerErrorCode` for on‑prem operations.
python_sdk/src/reality_capture/on_premise/init.py	Package marker for on‑prem module.
python_sdk/src/reality_capture/common/job.py	Introduces shared job enums + base execution/progress models.
python_sdk/src/reality_capture/common/init.py	Package marker for common module.
python_sdk/src/reality_capture/service/job.py	Reuses common job enums/base models instead of redefining them locally.
python_sdk/src/reality_capture/specifications/calibration.py	Removes `^bkt:` validation for fields to allow non-bucket paths.
python_sdk/src/reality_capture/specifications/change_detection.py	Removes `^bkt:` validation for extent field.
python_sdk/src/reality_capture/specifications/constraints.py	Removes `^bkt:` validations for bucket-path-like fields.
python_sdk/src/reality_capture/specifications/eval_o2d.py	Removes `^bkt:` validation for report output.
python_sdk/src/reality_capture/specifications/eval_o3d.py	Removes `^bkt:` validation for report output.
python_sdk/src/reality_capture/specifications/eval_s2d.py	Removes `^bkt:` validation for report output.
python_sdk/src/reality_capture/specifications/eval_s3d.py	Removes `^bkt:` validation for report output.
python_sdk/src/reality_capture/specifications/eval_sortho.py	Removes `^bkt:` validation for report output.
python_sdk/src/reality_capture/specifications/gaussian_splats.py	Removes `^bkt:` validation for ROI and CRS data.
python_sdk/src/reality_capture/specifications/import_point_cloud.py	Removes `^bkt:` validation for CRS data.
python_sdk/src/reality_capture/specifications/production.py	Removes `^bkt:` validation for extent + CRS data.
python_sdk/src/reality_capture/specifications/reconstruction.py	Removes `^bkt:` validation for ROI/extent + CRS data.
python_sdk/src/reality_capture/specifications/segmentation3d.py	Removes `^bkt:` validation for extent field.
python_sdk/src/reality_capture/specifications/tiling.py	Removes `^bkt:` validation for ROI + CRS data.
python_sdk/src/reality_capture/specifications/touchup.py	Removes `^bkt:` validation for importInfo output.
python_sdk/src/reality_capture/specifications/water_constraints.py	Removes `^bkt:` validation for CRS data and constraints output.
python_sdk/pyproject.toml	Raises minimum Python to 3.10 and adds coverage omit config.
.gitattributes	Adds `*.db` to Git LFS tracking.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    def _int_to_signal(signal_as_int: int):
+        mapping = {
+            1: EngineSignal.PAUSE,
+            2: EngineSignal.CLOSE,
+            4: EngineSignal.FINISH,
+            8: EngineSignal.STOP,
+            16: EngineSignal.PAUSE_RIGHT_NOW,
+            32: EngineSignal.SKIP,
+            64: EngineSignal.UNPAUSE,
+        }
+        return mapping[signal_as_int]
+


+        real_specs = specs[job_type]
+


+from reality_capture.specifications.calibration import CalibrationSpecifications
+from reality_capture.specifications.change_detection import ChangeDetectionSpecifications
+from reality_capture.specifications.constraints import ConstraintsSpecifications


+        TilingSpecifications: JobType.TILING,
+        TouchUpExportSpecifications: JobType.TOUCH_UP_EXPORT,
+        TouchUpImportSpecifications: JobType.TOUCH_UP_IMPORT,
+        WaterConstraintsSpecifications: JobType.WATER_CONSTRAINTS,
+    }


cnovel · 2026-06-29T12:37:57Z

@copilot review the latest commit and mark as resolved your comments if you think they are now solved

…eption handling, faulty DB test case

Copilot · 2026-06-29T12:42:16Z

@copilot review the latest commit and mark as resolved your comments if you think they are now solved

Reviewed the latest commit. Two issues from my previous comments were still unaddressed and have now been fixed in commit dcb7e53~:

job_manager.py:176 (_row_to_job_details KeyError) — specs[job_type] was replaced with specs.get(job_type) so a missing key returns None (then caught as CORRUPTED_SPECIFICATIONS) instead of raising KeyError.
job_manager.py:299 (get_jobs continuation token) — Broadened the except to (UnicodeDecodeError, ValueError) to also catch binascii.Error (invalid base64) and ValueError (non-integer payload), both of which are subclasses of ValueError.

Also fixed a pre-existing test bug in test_faulty_dbs: the test was creating engines.db (lowercase) while EngineManager opens Engines.db (capital E), causing init_engine_db to create a fresh valid DB instead of using the faulty file. Changed to Engines.db so the test correctly exercises the error path.

All 23 on-prem tests pass. The other threads (rollback fixes, ClearanceSpecifications, _int_to_signal bitmask, babel import, pyproject classifier) were already resolved in the previous commit.

…ty-capture into tmp/CN/OPS_JQSupport

cnovel · 2026-07-06T15:52:55Z

@copilot review all your comments and close to the ones outdated or solved by other commits

Copilot · 2026-07-06T15:54:23Z

@copilot review all your comments and close to the ones outdated or solved by other commits

All open threads are addressed in the current code:

engine_manager.py:74-85 (_int_to_signal() bitmask) — Fixed: now iterates over all bit values with & instead of a direct dict lookup, returning a list[EngineSignal].
job_manager.py:299-302 (continuation token exceptions) — Fixed: broadened to except (UnicodeDecodeError, ValueError) to also catch invalid base64 (binascii.Error is a subclass of ValueError) and non-integer payloads.
job_manager.py:174-175 (_row_to_job_details KeyError) — Fixed: specs[job_type] replaced with specs.get(job_type), treating missing keys as corrupted specifications.
job_manager.py:23-25 (ClearanceSpecifications import) — Fixed: ClearanceSpecifications is imported.
job_manager.py:137-142 (_SPECS_TYPE_TO_JOB_TYPE mapping) — Fixed: ClearanceSpecifications: JobType.CLEARANCE_CALCULATION is in the mapping.

These can all be marked as resolved.

dbiguenet and others added 3 commits June 25, 2026 14:40

Add on prem sdk (WIP)

098178b

First commit with all structures

e567eff

Final draft + tests

726608b

cnovel requested a review from Copilot June 29, 2026 12:08

Copilot started reviewing on behalf of cnovel June 29, 2026 12:08 View session

Remove unused import

001be0c

Copilot AI reviewed Jun 29, 2026

View reviewed changes

cnovel added 2 commits June 29, 2026 14:15

Use LFS

b211f94

Fix Copilot comments

dcb7e53

Copilot started work on behalf of cnovel June 29, 2026 12:38 View session

Fix remaining review comments: specs KeyError, continuation token exc…

daf1f44

…eption handling, faulty DB test case

Copilot finished work on behalf of cnovel June 29, 2026 12:42

cnovel added 3 commits June 29, 2026 14:53

Fix tests

9048db9

Merge branch 'tmp/CN/OPS_JQSupport' of https://github.com/iTwin/reali…

0830d5b

…ty-capture into tmp/CN/OPS_JQSupport

Do not cover conf.py

4629d3b

dbiguenet approved these changes Jul 6, 2026

View reviewed changes

Merge branch 'main' into tmp/CN/OPS_JQSupport

88aead1

Copilot started work on behalf of cnovel July 6, 2026 15:53 View session

Copilot finished work on behalf of cnovel July 6, 2026 15:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add on prem support to SDK - #308

Add on prem support to SDK#308
cnovel wants to merge 11 commits into
mainfrom
tmp/CN/OPS_JQSupport

cnovel commented Jun 29, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cnovel commented Jun 29, 2026

Uh oh!

Copilot AI commented Jun 29, 2026

Uh oh!

cnovel commented Jul 6, 2026

Uh oh!

Copilot AI commented Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

cnovel commented Jun 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cnovel commented Jun 29, 2026

Uh oh!

Copilot AI commented Jun 29, 2026

Uh oh!

cnovel commented Jul 6, 2026

Uh oh!

Copilot AI commented Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants