feat: Registry v2 artifact searcher by BackendBits · Pull Request #805 · Netcracker/qubership-envgene

BackendBits · 2025-11-18T08:40:34Z

Summary

Add support for Registry V2 cloud registries in the Python artifact searcher and fix build_env tests so CI passes.

Issue

Internal need to:

Use AWS CodeArtifact and GCP Artifact Registry via RegDef V2.
Unblock the open-source CI pipeline, which was failing in scripts/build_env tests due to a [tests] package conflict.

Breaking Change?

Yes
No

No existing flows are changed; V2 support is used only when configured.

Scope / Project

python/artifact-searcher
scripts/build_env

Implementation Notes

Added CloudAuthHelper in artifact_searcher to:
- Resolve authConfig entries from RegDef V2.
- Resolve credentials from env_creds.
- Configure MavenArtifactSearcher for:
  - AWS CodeArtifact.
  - GCP Artifact Registry (service account).
Updated build_env RegDef V2 handling to:
- Detect version 2.0.
- Validate V2 RegDefs with the correct schema.
- Check authConfig references in V2 config sections.
Fixed build_env tests by:
- Adding [scripts/build_env/tests/init.py] so the local [tests] package is used in CI.
- Keeping imports as from tests.test_helpers import TestHelpers.

Tests / Evidence

Existing pytest suites for:
- python/envgene/envgenehelper
- scripts/build_env
run in the GitHub Actions [tests] job.
After the changes, the previous ModuleNotFoundError in scripts/build_env tests is resolved and the suite runs to completion (subject to any functional test failures).

Additional Notes

No new external dependencies beyond what artifact_searcher already requires.
[CloudAuthHelper] logs clear errors when configuration or credentials are incomplete, making misconfigurations easier to diagnose.

Detect version field in RegDef files and validate against V2 schema when version is 2.0. Use logger for validation messages and move schema paths to constants.

- Add AuthConfig model for V2 authentication configuration - Add version and auth_config fields to Registry model - Add V2 routing in check_artifact_async with fallback to V1 - Add CloudAuthHelper for AWS/GCP cloud registry authentication - Add environment credential loading for V2 cloud registries - Add V2 cloud registry dependencies (boto3, google-auth) - Add comprehensive tests for V2 models and routing Preserves all V1 functionality including: - Nexus detection and URL conversion - Snapshot version resolution - URL-based artifact search

- Simplified retry logic (2 retries, 5s fixed delay) - Reduced timeouts to reasonable values (60s search, 120s download) - Removed debug/diagnostic code - Removed unnecessary test files - Clean up code style

…ifacotry/nexus

Detect version field in RegDef files and validate against V2 schema when version is 2.0. Use logger for validation messages and move schema paths to constants.

…searcher

…istry_v2

…searcher

…istry_v2

…searcher

…istry_v2

…nto registry-v2-artifact-searcher

…istry_v2

…searcher

…t-searcher

…istry_v2

…searcher

miyamuraga · 2026-02-11T13:57:58Z

python/artifact-searcher/artifact_searcher/cloud_auth_helper.py

+
+        auth_ref = getattr(registry.maven_config, 'auth_config', None)
+        if not auth_ref:
+            return None


we quietly return None without log

miyamuraga · 2026-02-11T14:55:19Z

python/artifact-searcher/artifact_searcher/cloud_auth_helper.py

+try:
+    from qubership_pipelines_common_library.v1.maven_client import MavenArtifactSearcher
+except ImportError:
+    MavenArtifactSearcher = None


Better to raise error here instead of using None, so it’s clear library is missing. if we return None here, exception will still be raised later at line 81, but failing early would make error clearer.

miyamuraga · 2026-02-11T14:58:43Z

python/artifact-searcher/artifact_searcher/utils/models.py

    helm_config: Optional[HelmConfig] = None
    helm_app_config: Optional[HelmAppConfig] = None
+    version: Optional[str] = "1.0"
+    auth_config: Optional[dict[str, AuthConfig]] = None


I see you haven't made different models for reg def v2. do not mix 2 models with different schemas in one. use polymorphism

miyamuraga · 2026-02-11T15:06:04Z

python/artifact-searcher/artifact_searcher/cloud_auth_helper.py

+
+
+# Timeout for MavenArtifactSearcher: (connect_timeout, read_timeout)
+DEFAULT_SEARCHER_TIMEOUT = (30, 60)


specify all timeouts in one place with defaults. may be here artifact_searcher.utils.constants.
make it possible to set them from pipeline. ARTIFACT_SEARCHER_CONFIG: map
don't forget to put this new pipe param here and job parameters

miyamuraga · 2026-02-11T15:14:51Z

python/artifact-searcher/artifact_searcher/cloud_auth_helper.py

+        Returns AuthConfig or None.
+        """
+        if artifact_type != "maven":
+            return None


for what this condition? artifact_type: has default maven value always. what sense of artifact_type if it is not used anywhere else ?

miyamuraga · 2026-02-12T17:26:56Z

python/artifact-searcher/artifact_searcher/artifact.py

+    max_retries = 2
+    last_error = None
+    local_path = None
+    maven_url = None


miyamuraga · 2026-02-12T18:04:16Z

python/artifact-searcher/artifact_searcher/artifact.py

+                searcher = await loop.run_in_executor(None, CloudAuthHelper.create_maven_searcher, app.registry, env_creds)
+
+            urls = await asyncio.wait_for(
+                loop.run_in_executor(None, partial(searcher.find_artifact_urls, artifact=maven_artifact)),


why do we use third-party synchronous library to search for an artifact If we already have asynchronous method in our library for v1, we just need to add auth for other registers and that all. We need to reconsider the using of this library.

miyamuraga · 2026-02-12T18:09:59Z

python/artifact-searcher/artifact_searcher/artifact.py

+    maven_url = None
+
+    # Retry on transient errors (401, timeout, expired)
+    for attempt in range(max_retries):


is it necessary to add mechanism of retries for both v1 and v2

miyamuraga · 2026-02-12T18:39:37Z

python/artifact-searcher/artifact_searcher/artifact.py

+            local_path = os.path.join(create_app_artifacts_local_path(app.name, version), os.path.basename(maven_url))
+            os.makedirs(os.path.dirname(local_path), exist_ok=True)
+
+            download_success = await _v2_download_with_fallback(


no need to waste time downloading artifact when searching, in this method just need to find

you can check check_artifact_by_full_url_async for example

miyamuraga · 2026-02-12T18:45:20Z

python/artifact-searcher/artifact_searcher/artifact.py

+                return await _check_artifact_v1_async(app, artifact_extension, version, cred=None, classifier="")
+
+            # Try to extract useful HTTP error details for debugging
+            if hasattr(e, 'response'):


use same method for v1 and v2 for error handling

miyamuraga · 2026-02-12T19:04:32Z

python/artifact-searcher/artifact_searcher/artifact.py

+
+    try:
+        await asyncio.wait_for(
+            loop.run_in_executor(None, lambda: searcher.download_artifact(url, str(local_path))),


we already have method for downloading artifact, why do we use it from third-party library again?

miyamuraga · 2026-02-12T19:11:54Z

python/artifact-searcher/artifact_searcher/cloud_auth_helper.py

+        )
+
+    @staticmethod
+    def get_gcp_access_token(service_account_json: str) -> Optional[str]:


why do we use self-written authorization method if we have to use third-party library from qubership?

miyamuraga · 2026-02-12T19:18:39Z

python/artifact-searcher/artifact_searcher/artifact.py

+
+    try:
+        headers = {}
+        if auth_config.provider == "gcp":


why is there download implementation 3? we already have download either in our library or in third-paty one

miyamuraga · 2026-02-12T19:20:08Z

python/artifact-searcher/artifact_searcher/cloud_auth_helper.py

+            return None
+
+    @staticmethod
+    def get_gcp_credentials_from_registry(registry: Registry, env_creds: Optional[Dict[str, dict]]) -> Optional[str]:


why only for gcp? superfluous

miyamuraga · 2026-02-12T20:00:26Z

scripts/build_env/process_sd.py

-    if 'SNAPSHOT' in version:
-        raise ValueError("SNAPSHOT is not supported version of Solution Descriptor artifacts")
-    # TODO: check if job would fail without plugins
+def download_sd_by_appver(app_name: str, version: str, plugins: PluginEngine, env: Environment = None) -> dict[str, object]:


env: Environment = None not used

miyamuraga · 2026-02-12T20:04:20Z

scripts/build_env/process_sd.py

+
+    # V1 fallback path or non-V2 registry - need credentials for HTTP download
+    cred = None
+    if app_def.registry.credentials_id and env_creds:


remove fallback here also

miyamuraga · 2026-02-12T20:15:01Z

scripts/build_env/env_template/process_env_template.py

        if artifact_info:
-            template_url, _ = artifact_info
+            template_url, repo_info = artifact_info
+            # V2 optimization: artifact already downloaded by MavenArtifactSearcher


optimization not needed here

miyamuraga · 2026-02-12T20:16:11Z

scripts/build_env/env_template/process_env_template.py

-            template_url, _ = artifact_info
+            template_url, repo_info = artifact_info
+            # V2 optimization: artifact already downloaded by MavenArtifactSearcher
+            if isinstance(repo_info, tuple) and len(repo_info) == 2 and repo_info[0] == "v2_downloaded":


what is it v2_downloaded ?? repository name can't be v2_downloaded

miyamuraga · 2026-02-12T20:18:05Z

scripts/build_env/env_template/process_env_template.py

-            template_url, _ = artifact_info
+            template_url, repo_info = artifact_info
+            # V2 optimization: artifact already downloaded by MavenArtifactSearcher
+            if isinstance(repo_info, tuple) and len(repo_info) == 2 and repo_info[0] == "v2_downloaded":


isinstance(repo_info, tuple) and len(repo_info) == 2 needless

BackendBits requested review from GlimmerCape, andyroode, chethana-shastry-p and miyamuraga as code owners November 18, 2025 08:40

github-actions bot added bug labels Nov 18, 2025

BackendBits force-pushed the registry-v2-artifact-searcher branch 3 times, most recently from 7047995 to f4408c0 Compare November 21, 2025 07:22

popoveugene removed bug labels Nov 27, 2025

BackendBits force-pushed the registry-v2-artifact-searcher branch from f4408c0 to 4c67632 Compare November 30, 2025 13:01

github-actions bot added the enhancement label Nov 30, 2025

BackendBits force-pushed the registry-v2-artifact-searcher branch from 4c67632 to 43d5a27 Compare December 1, 2025 09:58

github-actions bot added the bug label Dec 1, 2025

BackendBits added 2 commits December 3, 2025 01:00

feat: Add RegDef V2 validation with version detection

e30aaa2

Detect version field in RegDef files and validate against V2 schema when version is 2.0. Use logger for validation messages and move schema paths to constants.

BackendBits force-pushed the registry-v2-artifact-searcher branch from 5ffc9b1 to f556b46 Compare December 2, 2025 23:57

github-actions bot added the refactor label Dec 3, 2025

BackendBits force-pushed the registry-v2-artifact-searcher branch from 0f21053 to e2bc8bf Compare December 3, 2025 06:31

BackendBits force-pushed the registry-v2-artifact-searcher branch from c4a4175 to 05467d1 Compare December 14, 2025 06:28

BackendBits requested a review from popoveugene as a code owner December 14, 2025 06:28

fix: V2 artifact searcher cleanup and improvements

186f166

- Simplified retry logic (2 retries, 5s fixed delay) - Reduced timeouts to reasonable values (60s search, 120s download) - Removed debug/diagnostic code - Removed unnecessary test files - Clean up code style

BackendBits force-pushed the registry-v2-artifact-searcher branch from 05467d1 to 186f166 Compare December 14, 2025 06:55

BackendBits added 6 commits December 14, 2025 12:46

fix: minor issues on v2

58f97af

fix: changes for v2 working with Auth_config as null or empty for Art…

e018414

…ifacotry/nexus

fix: Creds Fix

2b002fa

feat: Add RegDef V2 validation with version detection

f7086ac

Detect version field in RegDef files and validate against V2 schema when version is 2.0. Use logger for validation messages and move schema paths to constants.

Merge branch 'feature/support_registry_v2' into registry-v2-artifact-…

f3dbcc7

…searcher

Merge remote-tracking branch 'upstream/main' into feature/support_reg…

45b145d

…istry_v2

BackendBits added 18 commits February 5, 2026 17:02

fix: updated the Process_sd

5605a62

Merge remote-tracking branch 'upstream/main' into feature/support_reg…

0b11b51

…istry_v2

Merge branch 'feature/support_registry_v2' into registry-v2-artifact-…

0debfe7

…searcher

fix: Allow SNAPSHOT version resolution for SD artifacts

cc95304

Merge remote-tracking branch 'upstream/main' into feature/support_reg…

115f7e9

…istry_v2

Merge branch 'feature/support_registry_v2' into registry-v2-artifact-…

29c717a

…searcher

Merge remote-tracking branch 'upstream/main' into feature/support_reg…

23d1815

…istry_v2

fix: fixed issue with render

e0efde1

Merge remote-tracking branch 'upstream/main' into feature/support_reg…

b818147

…istry_v2

fix: fixed schema

363501e

Merge remote-tracking branch 'upstream/main' into feature/support_reg…

eb5de58

…istry_v2

Merge remote-tracking branch 'upstream/feature/support_registry_v2' i…

029bfe7

…nto registry-v2-artifact-searcher

fix: Review Comment for Duplicated method and Missing Test

34a902b

Merge remote-tracking branch 'upstream/main' into feature/support_reg…

c54d081

…istry_v2

Merge branch 'feature/support_registry_v2' into registry-v2-artifact-…

91a6e51

…searcher

Merge remote-tracking branch 'upstream/main' into registry-v2-artifac…

098dec1

…t-searcher

Merge remote-tracking branch 'upstream/main' into feature/support_reg…

0e17b1e

…istry_v2

Merge branch 'feature/support_registry_v2' into registry-v2-artifact-…

31d453f

…searcher

miyamuraga requested changes Feb 12, 2026

View reviewed changes

miyamuraga reviewed Feb 12, 2026

View reviewed changes

miyamuraga requested changes Feb 12, 2026

View reviewed changes

miyamuraga reviewed Feb 12, 2026

View reviewed changes

miyamuraga added the changes-requested label Feb 13, 2026

fix: SNAPSHOT FIX

a2b89df



		# Timeout for MavenArtifactSearcher: (connect_timeout, read_timeout)
		DEFAULT_SEARCHER_TIMEOUT = (30, 60)

Conversation

BackendBits commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Issue

Breaking Change?

Scope / Project

Implementation Notes

Tests / Evidence

Additional Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miyamuraga Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miyamuraga Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

BackendBits commented Nov 18, 2025 •

edited

Loading

miyamuraga Feb 12, 2026 •

edited

Loading

miyamuraga Feb 12, 2026 •

edited

Loading