-
Notifications
You must be signed in to change notification settings - Fork 6
set up base for migrating infield locations from APM to CDM #2159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
erikangelsen-cognite
wants to merge
10
commits into
main
Choose a base branch
from
infield/app-config-migration-base
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
3f3ffdd
set up base for migrating infield locations from APM to CDM
erikangelsen-cognite 97fca69
restucture files and do instanceSpace migration
erikangelsen-cognite d828d26
change type structure
erikangelsen-cognite ca4d32f
migrate data model to location filters
erikangelsen-cognite 708bd16
feature toggle migration
erikangelsen-cognite 9a583fb
comment todo for the remaining fields to migrate
erikangelsen-cognite 3b4e6d7
migrate asset
erikangelsen-cognite 50bf33f
disciplines and access management migration
erikangelsen-cognite 9a23117
data filters migration
erikangelsen-cognite 2563b6c
migrate data exploration config
erikangelsen-cognite File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
10 changes: 10 additions & 0 deletions
10
cognite_toolkit/_cdf_tk/commands/_migrate/infield_config/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| """InField V2 configuration migration module. | ||
|
|
||
| This module handles migration from old APM Config format to InField V2 configuration format. | ||
| The migration is split into separate modules for better organization and maintainability. | ||
| """ | ||
|
|
||
| from .migration import create_infield_v2_config | ||
|
|
||
| __all__ = ["create_infield_v2_config"] | ||
|
|
19 changes: 19 additions & 0 deletions
19
cognite_toolkit/_cdf_tk/commands/_migrate/infield_config/constants.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| """Constants for InField V2 config migration.""" | ||
|
|
||
| from cognite.client.data_classes.data_modeling import ViewId | ||
| from cognite.client.data_classes.data_modeling.ids import DataModelId | ||
|
|
||
| # View IDs for the new format | ||
| LOCATION_CONFIG_VIEW_ID = ViewId("infield_cdm_source_desc_sche_asset_file_ts", "InFieldLocationConfig", "v1") | ||
| DATA_EXPLORATION_CONFIG_VIEW_ID = ViewId("infield_cdm_source_desc_sche_asset_file_ts", "DataExplorationConfig", "v1") | ||
|
|
||
| # Target space for InFieldLocationConfig nodes | ||
| TARGET_SPACE = "APM_Config" | ||
|
|
||
| # Default data model for LocationFilter | ||
| DEFAULT_LOCATION_FILTER_DATA_MODEL = DataModelId( | ||
| space="infield_cdm_source_desc_sche_asset_file_ts", | ||
| external_id="InFieldOnCDM", | ||
| version="v1", | ||
| ) | ||
|
|
86 changes: 86 additions & 0 deletions
86
cognite_toolkit/_cdf_tk/commands/_migrate/infield_config/data_exploration_config.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| """Migration of DataExplorationConfig for InField V2 config migration. | ||
|
|
||
| This module handles the creation of a single DataExplorationConfig node per APM Config, | ||
| which is shared across all InFieldLocationConfig nodes via a direct relation. | ||
| """ | ||
|
|
||
| from typing import Any | ||
|
|
||
| from cognite.client.data_classes.data_modeling import NodeApply, NodeOrEdgeData | ||
|
|
||
| from .constants import DATA_EXPLORATION_CONFIG_VIEW_ID, TARGET_SPACE | ||
| from .types_new import DataExplorationConfigProperties | ||
|
|
||
|
|
||
| def create_data_exploration_config_node( | ||
| feature_configuration: dict[str, Any] | None, | ||
| config_external_id: str, | ||
| ) -> NodeApply | None: | ||
| """Create a DataExplorationConfig node from FeatureConfiguration. | ||
|
|
||
| Only one DataExplorationConfig is created per APM Config, shared across all locations. | ||
|
|
||
| Args: | ||
| feature_configuration: FeatureConfiguration dict from old APM Config | ||
| config_external_id: External ID of the APM Config (used to generate DataExplorationConfig external ID) | ||
|
|
||
| Returns: | ||
| NodeApply for DataExplorationConfig, or None if feature_configuration is missing or incomplete | ||
| """ | ||
| if not feature_configuration: | ||
| return None | ||
|
|
||
| # Extract properties from FeatureConfiguration | ||
| props: DataExplorationConfigProperties = {} | ||
|
|
||
| # Migrate observations | ||
| if observations := feature_configuration.get("observations"): | ||
| props["observations"] = observations | ||
|
|
||
| # Migrate activities | ||
| if activities := feature_configuration.get("activities"): | ||
| props["activities"] = activities | ||
|
|
||
| # Migrate documents | ||
| if documents := feature_configuration.get("documents"): | ||
| # Remove metadata. prefix from type and description if present | ||
| migrated_documents = documents.copy() | ||
| if "type" in migrated_documents and isinstance(migrated_documents["type"], str): | ||
| migrated_documents["type"] = migrated_documents["type"].removeprefix("metadata.") | ||
| if "description" in migrated_documents and isinstance(migrated_documents["description"], str): | ||
| migrated_documents["description"] = migrated_documents["description"].removeprefix("metadata.") | ||
| props["documents"] = migrated_documents | ||
|
|
||
| # Migrate notifications | ||
| if notifications := feature_configuration.get("notifications"): | ||
| props["notifications"] = notifications | ||
|
|
||
| # Migrate assets (from assetPageConfiguration) | ||
| if assets := feature_configuration.get("assetPageConfiguration"): | ||
| props["assets"] = assets | ||
|
|
||
| # Only create node if at least one property is present | ||
| if not props: | ||
| return None | ||
|
|
||
| # Generate external ID for DataExplorationConfig | ||
| data_exploration_external_id = f"data_exploration_{config_external_id}" | ||
|
|
||
| return NodeApply( | ||
| space=TARGET_SPACE, | ||
| external_id=data_exploration_external_id, | ||
| sources=[NodeOrEdgeData(source=DATA_EXPLORATION_CONFIG_VIEW_ID, properties=props)], | ||
| ) | ||
|
|
||
|
|
||
| def get_data_exploration_config_external_id(config_external_id: str) -> str: | ||
| """Generate external ID for DataExplorationConfig node. | ||
|
|
||
| Args: | ||
| config_external_id: External ID of the APM Config | ||
|
|
||
| Returns: | ||
| External ID for the DataExplorationConfig node | ||
| """ | ||
| return f"data_exploration_{config_external_id}" | ||
|
|
6 changes: 6 additions & 0 deletions
6
cognite_toolkit/_cdf_tk/commands/_migrate/infield_config/location_config/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| """InFieldLocationConfig field migration module.""" | ||
|
|
||
| from .fields import apply_location_config_fields | ||
|
|
||
| __all__ = ["apply_location_config_fields"] | ||
|
|
34 changes: 34 additions & 0 deletions
34
...ite_toolkit/_cdf_tk/commands/_migrate/infield_config/location_config/access_management.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| """Migration of accessManagement field for InFieldLocationConfig.""" | ||
|
|
||
| from typing import Any | ||
|
|
||
| from ..types_new import AccessManagement | ||
|
|
||
|
|
||
| def migrate_access_management(location_dict: dict[str, Any]) -> AccessManagement | None: | ||
| """Migrate accessManagement from old configuration. | ||
|
|
||
| Extracts templateAdmins and checklistAdmins from the old location configuration | ||
| and creates an AccessManagement dict. | ||
|
|
||
| Args: | ||
| location_dict: Location configuration dict (from dump(camel_case=True)) | ||
|
|
||
| Returns: | ||
| AccessManagement dict, or None if neither templateAdmins nor checklistAdmins are present | ||
| """ | ||
| template_admins = location_dict.get("templateAdmins") | ||
| checklist_admins = location_dict.get("checklistAdmins") | ||
|
|
||
| # Only create accessManagement if at least one of the fields is present | ||
| if not template_admins and not checklist_admins: | ||
| return None | ||
|
|
||
| access_management: AccessManagement = {} | ||
| if template_admins: | ||
| access_management["templateAdmins"] = template_admins | ||
| if checklist_admins: | ||
| access_management["checklistAdmins"] = checklist_admins | ||
|
|
||
| return access_management | ||
|
|
24 changes: 24 additions & 0 deletions
24
...te_toolkit/_cdf_tk/commands/_migrate/infield_config/location_config/app_instance_space.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| """Migration of appInstanceSpace field for InFieldLocationConfig.""" | ||
|
|
||
| from typing import Any | ||
|
|
||
|
|
||
| def migrate_app_instance_space(location_dict: dict[str, Any]) -> str | None: | ||
| """Migrate appInstanceSpace from appDataInstanceSpace. | ||
|
|
||
| Extracts the appDataInstanceSpace from the old configuration and returns it | ||
| as appInstanceSpace. If appDataInstanceSpace is not present or is an empty string, | ||
| returns None. | ||
|
|
||
| Args: | ||
| location_dict: Location configuration dict (from dump(camel_case=True)) | ||
|
|
||
| Returns: | ||
| App instance space string, or None if not present or empty | ||
| """ | ||
| app_instance_space = location_dict.get("appDataInstanceSpace") | ||
| if not app_instance_space: # None or empty string | ||
| return None | ||
|
|
||
| return app_instance_space | ||
|
|
27 changes: 27 additions & 0 deletions
27
cognite_toolkit/_cdf_tk/commands/_migrate/infield_config/location_config/data_filters.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| """Migration of dataFilters field for InFieldLocationConfig.""" | ||
|
|
||
| from typing import Any | ||
|
|
||
| from ..types_new import RootLocationDataFilters | ||
|
|
||
|
|
||
| def migrate_data_filters(location_dict: dict[str, Any]) -> RootLocationDataFilters | None: | ||
| """Migrate dataFilters from old configuration. | ||
|
|
||
| Extracts the dataFilters dictionary from the old location configuration | ||
| and returns it as a RootLocationDataFilters dict. The structure matches | ||
| between old and new format, so it can be returned as-is. | ||
|
|
||
| Args: | ||
| location_dict: Location configuration dict (from dump(camel_case=True)) | ||
|
|
||
| Returns: | ||
| RootLocationDataFilters dict, or None if dataFilters is not present | ||
| """ | ||
| data_filters = location_dict.get("dataFilters") | ||
| if not data_filters: | ||
| return None | ||
|
|
||
| # Return the data filters as-is (already in the correct format) | ||
| return data_filters | ||
|
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this true, do you expect APM_Config to turn into location filter(s) and a special Infield v2 configuration?