diff --git a/docs/analisys/colly-to-envgene-mapping.md b/docs/analisys/colly-to-envgene-mapping.md new file mode 100644 index 00000000..8b03272b --- /dev/null +++ b/docs/analisys/colly-to-envgene-mapping.md @@ -0,0 +1,92 @@ +# Mapping of Colly attributes to EnvGene attributes + +- [Mapping of Colly attributes to EnvGene attributes](#mapping-of-colly-attributes-to-envgene-attributes) + - [Description](#description) + - [Mapping table](#mapping-table) + - [`env_definition.yml` example](#env_definitionyml-example) + - [To Discuss](#to-discuss) + - [To Implement](#to-implement) + +## Description + +This document details how Colly attributes are persisted within the EnvGene Instance repository, specifying in which files and sections each attribute is stored. + +## Mapping table + +| Colly Object | Colly Attribute | Attribute Type in Colly | EnvGene Repository | Location in EnvGene | Description | +|--------------|-------------------------|---------------------------------------------------------------------------------------------------|--------------------|-------------------------------------------------|-------------------------------------------------------------------------------------------------------------| +| Environment | `id` | string | instance | `env_definition.metadata.id` | Unique identifier of the Environment in Colly and EnvGene, matches the name | +| Environment | `name` | string | instance | `env_definition.metadata.name` | Environment name; same as the identifier | +| Environment | `owners` | array of string | instance | `env_definition.metadata.owners` | Users responsible for the environment (Owners in Colly) | +| Environment | `teams` | array of string | instance | `env_definition.metadata.teams` | List of teams assigned to the environment | +| Environment | `status` | enum [`IN_USE`, `RESERVED`, `FREE`, `MIGRATING`, `DEPRECATED`] | instance | `env_definition.metadata.status` | Current status of the environment: in use, free, reserved, etc | +| Environment | `expirationDate` | string (LocalDate "yyyy-MM-dd") | instance | `env_definition.metadata.expirationDate` | Date until which the environment is allocated (expirationDate) | +| Environment | `type` | enum [`ENVIRONMENT`, `CSE_TOOLSET`, `DESIGN_TIME`, `APP_DEPLOYER`, `INFRASTRUCTURE`, `UNDEFINED`] | instance | `env_definition.metadata.type` | Technical category of the environment; type of environment | +| Environment | `role` | string (the list of allowed values is set via deployment parameter) | instance | `env_definition.metadata.role` | Usage role of the environment (e.g. Dev, QA), set by parameters | +| Environment | `labels` | array of string | instance | `env_definition.metadata.labels` | User-defined labels/tags for the environment | +| Environment | `description` | string | instance | `env_definition.metadata.description` | Free-form environment description | +| Environment | `region` | string | instance | `env_definition.metadata.region` | Geographical region associated with the environment; set by the user | +| Environment | `accessGroups` | array of string | instance | `env_definition.metadata.accessGroups` | List of user groups that can work with the environment | +| Environment | `effectiveAccessGroups` | array of string | instance | `env_definition.metadata.effectiveAccessGroups` | Resolved full list of user groups (contains groups and their descendants). resolved based on `accessGroups` | +| Cluster | `name` | string | TBD | TBD | Name / Identifier of the cluster (cluster name), unique within the instance context | +| Cluster | `id` | string | TBD | TBD | Unique identifier of the cluster (same as name); retained for compatibility | +| Cluster | `description` | string | TBD | TBD | Free-form cluster description | + +### `env_definition.yml` example + +```yaml +metadata: + owners: + - "user1" + - "user2" + teams: + - "team-a" + - "team-b" + status: "IN_USE" + expirationDate: "2024-12-31" + type: "ENVIRONMENT" + role: "Dev" + labels: + - "prod" + - "priority-high" + description: "very important env" + region: cm +``` + +## To Discuss + +- [x] Is `ticketLinks` required? + - The attribute is not needed by users, so it was decided to remove it + +- [x] Checkov [linting errors](https://github.com/Netcracker/qubership-envgene/actions/runs/18886399036/job/53902750553) + - Decided to ignore + +- [x] Add the `deployPostfix` attribute to the Namespace? + - No + +- [x] Remove the `deploymentVersion` attribute from the Environment? + - Yes + +- [x] What is `deploymentStatus`? + - it is removed + +- [x] Do we keep `cleanInstallationDate`? Is it computed based on SD_VERSIONS? + - it will be removed at [MS2](https://github.com/Netcracker/qubership-colly/issues/153) + +- [x] How do we uniquely identify cluster, environment, and namespace in both services? + +- [x] How do we separate the two services? + - Both services should return a unified schema for cluster, environment, and namespace, with all fields, but fill in only their own data. For example, for Environment: + - inventory-service: owners, teams, status, type, role, labels, description, expirationDate + - operational-service: cleanInstallationDate, monitoringData, deploymentStatus, lastSDDeploymentOperation, lastDeployedSDsByType + - ~~Two different models~~ + +- [ ] Where to store the `description` of a Cluster + +- [ ] Using the OpenAPI specification as documentation + - How are descriptions and examples added to the OpenAPI spec? + +## To Implement + +- [ ] Change the formation of the macros `current_env.description` and `current_env.owners` taking into account the metadata section and migration +- [ ] Extend EnvGene `env_definition.yaml` JSON schema diff --git a/docs/analisys/inventory-service-openapi-splitted-v2.yaml b/docs/analisys/inventory-service-openapi-splitted-v2.yaml new file mode 100644 index 00000000..59e42f1a --- /dev/null +++ b/docs/analisys/inventory-service-openapi-splitted-v2.yaml @@ -0,0 +1,204 @@ +--- +openapi: 3.1.0 +info: + title: envgene-inventory-service API + version: 1.0.0-SNAPSHOT +paths: + /colly/v2/inventory-service/auth-status: + get: + responses: + "200": + description: OK + summary: Get Auth Status + tags: + - Envgene Inventory Service Rest + /colly/v2/inventory-service/clusters: + get: + responses: + "200": + description: OK + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/ClusterDto" + summary: Get Clusters + tags: + - Envgene Inventory Service Rest + /colly/v2/inventory-service/environments: + get: + responses: + "200": + description: OK + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/EnvironmentDto" + summary: Get Environments + tags: + - Envgene Inventory Service Rest + /colly/v2/inventory-service/environments/{environmentId}: + put: + parameters: + - name: environmentId + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + $ref: "#/components/schemas/EnvironmentDto" + required: true + responses: + "200": + description: OK + summary: Update Environment + tags: + - Envgene Inventory Service Rest + /colly/v2/inventory-service/internal/cluster-infos: + get: + responses: + "200": + description: OK + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/InternalClusterInfoDto" + summary: Get Internal Cluster Info + tags: + - Envgene Inventory Service Rest + /colly/v2/inventory-service/manual-sync: + post: + responses: + "201": + description: Created + summary: Sync Environments With Git + tags: + - Envgene Inventory Service Rest +components: + schemas: + ClusterDto: + type: object + properties: + id: + type: string + name: + type: string + EnvironmentDto: + type: object + properties: + id: + type: string + name: + type: string + description: + type: string + namespaces: + type: array + items: + $ref: "#/components/schemas/NamespaceDto" + cluster: + $ref: "#/components/schemas/ClusterDto" + owners: + type: array + items: + type: string + labels: + type: array + items: + type: string + teams: + type: array + items: + type: string + status: + $ref: "#/components/schemas/EnvironmentStatus" + expirationDate: + $ref: "#/components/schemas/LocalDate" + type: + $ref: "#/components/schemas/EnvironmentType" + role: + type: string + EnvironmentStatus: + type: string + enum: + - IN_USE + - RESERVED + - FREE + - MIGRATING + EnvironmentType: + type: string + enum: + - ENVIRONMENT + - CSE_TOOLSET + - DESIGN_TIME + - APP_DEPLOYER + - INFRASTRUCTURE + - PORTAL + - UNDEFINED + InternalClusterInfoDto: + type: object + properties: + id: + type: string + name: + type: string + token: + type: string + cloudApiHost: + type: string + environments: + type: array + items: + $ref: "#/components/schemas/LightEnvironmentDto" + monitoringUrl: + type: string + LightEnvironmentDto: + type: object + properties: + id: + type: string + name: + type: string + namespaces: + type: array + items: + $ref: "#/components/schemas/NamespaceDto" + LocalDate: + type: string + format: date + examples: + - 2022-03-10 + NamespaceDto: + type: object + properties: + id: + type: string + name: + type: string + ProjectDto: + type: object + properties: + id: + type: string + description: Project ID. unique within the Colly instance. + name: + type: string + description: + type: string + environments: + type: array + items: + $ref: "#/components/schemas/EnvironmentDto" + securitySchemes: + SecurityScheme: + type: openIdConnect + description: Authentication + openIdConnectUrl: http://host.docker.internal:8180/realms/quarkus/.well-known/openid-configuration diff --git a/docs/analisys/operational-service-openapi-splitted-v2.yaml b/docs/analisys/operational-service-openapi-splitted-v2.yaml new file mode 100644 index 00000000..15af59b6 --- /dev/null +++ b/docs/analisys/operational-service-openapi-splitted-v2.yaml @@ -0,0 +1,120 @@ +--- +openapi: 3.1.0 +info: + title: environment-operational-service API + version: 1.0.0-SNAPSHOT +paths: + /colly/v2/operational-service/auth-status: + get: + responses: + "200": + description: OK + summary: Get Auth Status + tags: + - Cluster Resources Rest + /colly/v2/operational-service/clusters: + get: + responses: + "200": + description: OK + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/ClusterDTO" + summary: Get Clusters + tags: + - Cluster Resources Rest + /colly/v2/operational-service/environments: + get: + responses: + "200": + description: OK + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/EnvironmentDTO" + summary: Get Environments + tags: + - Cluster Resources Rest + /colly/v2/operational-service/manual-sync: + post: + responses: + "201": + description: Created + summary: Load Environments Manually + tags: + - Cluster Resources Rest + /colly/v2/operational-service/metadata: + get: + responses: + "200": + description: OK + content: + application/json: + schema: + $ref: "#/components/schemas/ApplicationMetadata" + summary: Get Metadata + tags: + - Cluster Resources Rest +components: + schemas: + ApplicationMetadata: + type: object + properties: + monitoringColumns: + type: array + items: + type: string + ClusterDTO: + type: object + properties: + id: + type: string + name: + type: string + synced: + type: boolean + EnvironmentDTO: + type: object + properties: + id: + type: string + name: + type: string + namespaces: + type: array + items: + $ref: "#/components/schemas/NamespaceDTO" + cluster: + $ref: "#/components/schemas/ClusterDTO" + deploymentVersion: + type: string + cleanInstallationDate: + $ref: "#/components/schemas/Instant" + monitoringData: + type: object + additionalProperties: + type: string + Instant: + type: string + format: date-time + examples: + - 2022-03-10T16:15:50Z + NamespaceDTO: + type: object + properties: + uid: + type: string + name: + type: string + existsInK8s: + type: boolean + securitySchemes: + SecurityScheme: + type: openIdConnect + description: Authentication + openIdConnectUrl: http://host.docker.internal:8180/realms/quarkus/.well-known/openid-configuration diff --git a/docs/analisys/operational-service-openapi-v1.yaml b/docs/analisys/operational-service-openapi-v1.yaml new file mode 100644 index 00000000..1c3d1249 --- /dev/null +++ b/docs/analisys/operational-service-openapi-v1.yaml @@ -0,0 +1,269 @@ +--- +openapi: 3.1.0 +paths: + /colly/environment-operational-service/auth-status: + get: + responses: + "200": + description: OK + summary: Get Auth Status + tags: + - Cluster Resources Rest + /colly/environment-operational-service/clusters: + get: + responses: + "200": + description: OK + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/ClusterDTO" + summary: Get Clusters + tags: + - Cluster Resources Rest + /colly/environment-operational-service/clusters/{clusterName}: + post: + parameters: + - name: clusterName + in: path + required: true + schema: + type: string + requestBody: + content: + application/x-www-form-urlencoded: + schema: + type: object + properties: + description: + type: string + required: true + responses: + "201": + description: Created + "401": + description: Not Authorized + "403": + description: Not Allowed + summary: Save Cluster + tags: + - Cluster Resources Rest + security: + - SecurityScheme: + - admin + /colly/environment-operational-service/environments: + get: + responses: + "200": + description: OK + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/EnvironmentDTO" + summary: Get Environments + tags: + - Cluster Resources Rest + /colly/environment-operational-service/environments/{envId}: + post: + parameters: + - name: envId + in: path + required: true + schema: + type: string + requestBody: + content: + application/x-www-form-urlencoded: + schema: + type: object + properties: + name: + type: string + owner: + type: string + description: + type: string + status: + type: string + labels: + type: array + items: + type: string + type: + type: string + team: + type: string + deploymentStatus: + type: string + ticketLinks: + type: string + expirationDate: + type: string + required: true + responses: + "201": + description: Created + "401": + description: Not Authorized + "403": + description: Not Allowed + summary: Save Environment + tags: + - Cluster Resources Rest + security: + - SecurityScheme: + - admin + delete: + parameters: + - name: envId + in: path + required: true + schema: + type: string + responses: + "204": + description: No Content + "401": + description: Not Authorized + "403": + description: Not Allowed + summary: Delete Environment + tags: + - Cluster Resources Rest + security: + - SecurityScheme: + - admin + /colly/environment-operational-service/metadata: + get: + responses: + "200": + description: OK + content: + application/json: + schema: + $ref: "#/components/schemas/ApplicationMetadata" + summary: Get Metadata + tags: + - Cluster Resources Rest + /colly/environment-operational-service/tick: + post: + responses: + "201": + description: Created + summary: Load Environments Manually + tags: + - Cluster Resources Rest +components: + schemas: + ApplicationMetadata: + type: object + properties: + monitoringColumns: + type: array + items: + type: string + ClusterDTO: + type: object + properties: + name: + type: string + description: + type: string + synced: + type: boolean + DeploymentStatus: + type: string + enum: + - DEPLOYED + - FAILED + - IN_PROGRESS + - NOT_STARTED + EnvironmentDTO: + type: object + properties: + id: + type: string + name: + type: string + namespaces: + type: array + items: + $ref: "#/components/schemas/NamespaceDTO" + cluster: + $ref: "#/components/schemas/ClusterDTO" + owner: + type: string + team: + type: string + status: + $ref: "#/components/schemas/EnvironmentStatus" + expirationDate: + $ref: "#/components/schemas/LocalDate" + type: + $ref: "#/components/schemas/EnvironmentType" + labels: + type: array + items: + type: string + description: + type: string + deploymentVersion: + type: string + cleanInstallationDate: + $ref: "#/components/schemas/Instant" + monitoringData: + type: object + additionalProperties: + type: string + deploymentStatus: + $ref: "#/components/schemas/DeploymentStatus" + ticketLinks: + type: string + EnvironmentStatus: + type: string + enum: + - IN_USE + - RESERVED + - FREE + - MIGRATING + EnvironmentType: + type: string + enum: + - ENVIRONMENT + - CSE_TOOLSET + - DESIGN_TIME + - APP_DEPLOYER + - INFRASTRUCTURE + - UNDEFINED + Instant: + type: string + format: date-time + examples: + - 2022-03-10T16:15:50Z + LocalDate: + type: string + format: date + examples: + - 2022-03-10 + NamespaceDTO: + type: object + properties: + uid: + type: string + name: + type: string + existsInK8s: + type: boolean + securitySchemes: + SecurityScheme: + type: openIdConnect + description: Authentication + openIdConnectUrl: >- + http://host.docker.internal:8180/realms/quarkus/.well-known/openid-configuration +info: + title: environment-operational-service API + version: 1.0.0-SNAPSHOT diff --git a/docs/analisys/operational-service-openapi-v2.yaml b/docs/analisys/operational-service-openapi-v2.yaml new file mode 100644 index 00000000..02674766 --- /dev/null +++ b/docs/analisys/operational-service-openapi-v2.yaml @@ -0,0 +1,486 @@ +--- +openapi: 3.1.0 +info: + title: operational-service API + version: 2.0.0 +paths: + /colly/operational-service/v2/auth-status: + get: + responses: + "200": + description: OK + headers: + X-API-Version: + schema: + type: string + summary: Get Auth Status + tags: + - Cluster Resources Rest + /colly/operational-service/v2/clusters: + get: + responses: + "200": + description: OK + headers: + X-API-Version: + schema: + type: string + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/ClusterDTO" + summary: Get Clusters + tags: + - Cluster Resources Rest + /colly/operational-service/v2/clusters/{clusterName}: + post: + parameters: + - name: clusterName + in: path + required: true + schema: + type: string + requestBody: + content: + application/x-www-form-urlencoded: + schema: + type: object + properties: + description: + type: string + required: true + responses: + "201": + description: Created + headers: + X-API-Version: + schema: + type: string + "401": + description: Not Authorized + headers: + X-API-Version: + schema: + type: string + "403": + description: Not Allowed + headers: + X-API-Version: + schema: + type: string + summary: Save Cluster + tags: + - Cluster Resources Rest + security: + - SecurityScheme: + - admin + /colly/operational-service/v2/environments: + get: + responses: + "200": + description: OK + headers: + X-API-Version: + schema: + type: string + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/EnvironmentDTO" + summary: Get Environments + tags: + - Cluster Resources Rest + /colly/operational-service/v2/environments/{envId}: + post: + parameters: + - name: envId + in: path + required: true + schema: + type: string + requestBody: + content: + application/x-www-form-urlencoded: + schema: + type: object + properties: + name: + type: string + owners: + type: array + items: + type: string + description: + type: string + status: + type: string + labels: + type: array + items: + type: string + type: + type: string + teams: + type: array + items: + type: string + expirationDate: + type: string + role: + type: string + required: true + responses: + "201": + description: Created + headers: + X-API-Version: + schema: + type: string + "401": + description: Not Authorized + headers: + X-API-Version: + schema: + type: string + "403": + description: Not Allowed + headers: + X-API-Version: + schema: + type: string + summary: Save Environment + tags: + - Cluster Resources Rest + security: + - SecurityScheme: + - admin + delete: + parameters: + - name: envId + in: path + required: true + schema: + type: string + responses: + "204": + description: No Content + headers: + X-API-Version: + schema: + type: string + "401": + description: Not Authorized + headers: + X-API-Version: + schema: + type: string + "403": + description: Not Allowed + headers: + X-API-Version: + schema: + type: string + summary: Delete Environment + tags: + - Cluster Resources Rest + security: + - SecurityScheme: + - admin + /colly/operational-service/v2/metadata: + get: + responses: + "200": + description: OK + headers: + X-API-Version: + schema: + type: string + content: + application/json: + schema: + $ref: "#/components/schemas/ApplicationMetadata" + summary: Get Metadata + tags: + - Cluster Resources Rest + /colly/operational-service/v2/tick: + post: + responses: + "201": + description: Created + headers: + X-API-Version: + schema: + type: string + summary: Load Environments Manually + tags: + - Cluster Resources Rest +components: + schemas: + ApplicationMetadata: + type: object + properties: + monitoringColumns: + type: array + items: + type: string + clusterInfoUpdateInterval: + description: "Period of updating information from the cluster" + $ref: "#/components/schemas/Duration" + environmentRoles: + type: array + items: + type: string + description: list of available Environment roles configured via deployment parameters. + ClusterDTO: + type: object + properties: + name: + type: string + description: + type: string + clusterInfoUpdateStatus: + type: object + properties: + lastResult: + type: boolean + description: >- + Determines the information update status for cluster: if at + least one object from the cluster was successfully read, + the update is considered successful + lastCompletedAt: + description: >- + Time of the last successful update information from the cluster + $ref: "#/components/schemas/Instant" + EnvironmentDTO: + type: object + properties: + id: + type: string + description: >- + Unique identifier of the Environment in the service instance. + Generated automatically upon creation. + readOnly: true + name: + type: string + description: >- + Human-readable Environment name. For example, env-1 + namespaces: + type: array + description: >- + List of namespace objects included in the Environment + items: + $ref: "#/components/schemas/NamespaceDTO" + cluster: + $ref: "#/components/schemas/ClusterDTO" + owners: + type: array + description: > + Users responsible for the Environment. This attribute is user-defined. + items: + type: string + default: [] + example: + - Alice + teams: + type: array + description: > + Teams assigned to the Environment. This attribute is user-defined. + items: + type: string + default: [] + example: + - QA alpha team + status: + $ref: "#/components/schemas/EnvironmentStatus" + description: >- + Current status of the Environment. Values: PLANNED, FREE, IN_USE, RESERVED, DEPRECATED. + Indicates the environment's lifecycle state, e.g. planned but not yet deployed, + ready for use, in use by a team, reserved, or marked for deprecation. + expirationDate: + $ref: "#/components/schemas/LocalDate" + description: >- + Date until which Environment is allocated. This attribute is user-defined. + default: "" + example: + - 2025-12-31 + type: + $ref: "#/components/schemas/EnvironmentType" + description: >- + Defines the technical category of the Environment. This attribute is user-defined. + example: + - INFRASTRUCTURE + role: + type: string + description: >- + Defines the usage role of the Environment within the project. + The list of allowed values is configured via deployment parameters. + The list of available roles can be retrieved via /colly/operational-service/v2/metadata + default: "" + labels: + type: array + description: >- + Custom labels for the Environment. This attribute is user-defined. + items: + type: string + default: [] + description: + type: string + description: >- + Free-form Environment description. This attribute is user-defined. + default: "" + monitoringData: + type: object + additionalProperties: + type: string + region: + type: string + description: >- + Geographical region associated with the Environment. This attribute is user-defined. + deploymentOperations: + $ref: "#/components/schemas/DeploymentOperations" + required: + - id + - name + - namespaces + - cluster + EnvironmentStatus: + type: string + enum: + - PLANNED + - FREE + - IN_USE + - RESERVED + - DEPRECATED + default: FREE + EnvironmentType: + type: string + enum: + - ENVIRONMENT + - CSE_TOOLSET + - DESIGN_TIME + - APP_DEPLOYER + - INFRASTRUCTURE + - UNDEFINED + default: ENVIRONMENT + Instant: + type: string + format: date-time + examples: + - 2022-03-10T16:15:50Z + DeploymentOperations: + type: array + description: >- + Array of deployment operations for all namespaces related to the Environment. + Each element represents a separate deployment operation that may involve one or more + namespaces and consists of multiple SD (Solution Descriptor) deployments. + items: + type: object + properties: + completedAt: + type: string + format: date-time + description: >- + Time when the deployment operation finished. This operation may involve one or more + namespaces that are part of the Environment. The time reflects when all deployment + items in this operation completed, regardless of their individual outcomes. + example: 2022-03-10T16:15:50Z + deploymentItems: + type: array + description: >- + List of SD (Solution Descriptor) deployments that are part of this deployment operation. + Each entry represents a single SD deployment with its details. + items: + type: object + properties: + name: + type: string + description: >- + Name and version of the SD in format ":". + example: product-sd:1.2.3 + type: + type: string + description: >- + Type of SD (e.g., "product", "project", etc.). + example: product + mode: + type: string + enum: + - CLEAN_INSTALL + - ROLLING_UPDATE + description: >- + Deployment mode for the SD: + - CLEAN_INSTALL: remove all existing resources before deployment + - ROLLING_UPDATE: deploy on top of existing resources + example: CLEAN_INSTALL + status: + type: string + enum: + - SUCCESS + - FAILURE + description: >- + Result of the deployment operation for this SD. + example: SUCCESS + required: + - name + - type + - mode + - status + required: + - completedAt + - status + - deploymentItems + example: + - completedAt: 2022-03-10T16:15:50Z + deploymentItems: + - name: product-sd:1.2.3 + type: product + mode: ROLLING_UPDATE + status: SUCCESS + - name: project-sd:2.1.0 + type: project + mode: ROLLING_UPDATE + status: FAILURE + - completedAt: 2022-03-10T17:16:40Z + deploymentItems: + - name: project-sd:2.1.0 + type: project + status: SUCCESS + LocalDate: + type: string + format: date + examples: + - 2022-03-10 + NamespaceDTO: + type: object + properties: + uid: + type: string + name: + type: string + existsInK8s: + type: boolean + deployPostfix: + type: string + description: >- + Deploy postfix, namespace role in solution + Duration: + type: string + format: duration + description: >- + Duration in ISO 8601 format (e.g., 'PT30M' for 30 minutes, 'PT2H' for 2 + hours, 'P1D' for 1 day) + examples: + - "PT30M" + - "PT2H" + - "P1D" + - "PT1H30M" + securitySchemes: + SecurityScheme: + type: openIdConnect + description: Authentication + openIdConnectUrl: >- + http://host.docker.internal:8180/realms/quarkus/.well-known/openid-configuration diff --git a/docs/analisys/project-repository.md b/docs/analisys/project-repository.md new file mode 100644 index 00000000..bdc128bc --- /dev/null +++ b/docs/analisys/project-repository.md @@ -0,0 +1,263 @@ +# Project Repository + +- [Project Repository](#project-repository) + - [Description](#description) + - [Repository Structure](#repository-structure) + - [Defaults](#defaults) + - [\[Defaults\] `parameters.yaml`](#defaults-parametersyaml) + - [\[Defaults\] `credentials.yaml`](#defaults-credentialsyaml) + - [Projects](#projects) + - [\[Projects\] `parameters.yaml`](#projects-parametersyaml) + - [\[Projects\] `credentials.yaml`](#projects-credentialsyaml) + - [Example](#example) + - [To discuss](#to-discuss) + +## Description + +This document describes the structure and contents of the Project repository. + +## Repository Structure + +```text +├── defaults +| ├── parameters.yaml|yml +| └── credentials.yaml|yml +└── clusters +| ├── parameters.yaml|yml +| └── credentials.yaml|yml +└── projects + └── + ├── parameters.yaml|yml + └── credentials.yaml|yml +``` + +### Defaults + +#### [Defaults] `parameters.yaml` + + + +```yaml +# Optional +# Global list of users with access permissions for all projects +accessGroups: + # Mandatory + # User name + name: string + # Optional + # User permissions + permissions: enum[RO, RW] +``` + +#### [Defaults] `credentials.yaml` + +Currently, this file has no contents + +### Projects + +#### [Projects] `parameters.yaml` + +```yaml +# Mandatory +# Name of the customer +customerName: string +# Mandatory +# Name of the project +name: string +# Mandatory +# Type of the project +type: enum[ project, product ] +# Optional +# List of groups with RW access rights to objects of this project +accessGroups: + - string +# Optional +# Platform type for clusters in this project +# "ocp" stands for OpenShift, "k8s" for generic Kubernetes +clustersPlatform: enum[ ocp, k8s ] +mavenRepoName: string +# Mandatory +repositories: + - # Mandatory + # Assumption: repository with type envgeneTemplate is only one per project + type: enum[ envgeneInstance, envgeneTemplate, clusterProvision, envProvision, solutionDeploy ] + # Mandatory + url: string + # Mandatory + # Token for repository access + # Pointer to Credential in credentials.yaml + # In MS1, Colly will get access to the repository using a technical user, parameters for the user will be passed as a deployment parameter + token: creds.get('').secret + # Optional + # If not set, the "default" branch is used (as in GitLab/GitHub) + branch: string + # Optional + # Geographical region associated with the Environment. This attribute is user-defined + # Used in cases where specific `pipeline` repositories need to be used for certain environments + region: string + # Optional + # Only for type: envgeneTemplate + # TODO: discover from repository + envgeneArtifact: + # Mandatory + # EnvGene environment template artifact name (application from the application:version notation) + # For example platform:20251215.113905-22 + name: string + # Optional + # List of template names inside the artifact + templateDescriptorNames: list of strings + # Optional + # Template name that is used by default when creating project environments. + # Should be included in templateDescriptorNames + defaultTemplateDescriptorName: string +``` + +#### [Projects] `credentials.yaml` + +Contains [Credential](https://github.com/Netcracker/qubership-envgene/blob/main/docs/envgene-objects.md#credential) objects + +```yaml +: + type: usernamePassword + data: + username: string + password: string +: + type: secret + data: + secret: string +``` + +Example: + +```yaml +customerName: ACME +projectName: ACME-bss +repositories: + - type: envgeneInstance + url: https://git.acme.com/instance + token: instance-cred + - region: offsite-cn + type: pipeline + url: https://git.acme.com/pipeline + token: offsite-cn-pipeline-cred + - region: offsite-mb + type: pipeline + url: https://git.acmemb.com/pipelines + token: offsite-mb-pipeline-cred + - type: envgeneTemplate + url: https://git.acme.com/template + token: template-cred + branches: + - r25.3 + - r25.4 +envgeneTemplates: + envgene-acme: + - main + - dt + - dm +``` + +```yaml +instance-cred: + type: secret + data: + secret: "MGE3MjYwNTQtZGE4My00MTlkLWIzN2MtZjU5YTg3NDA2Yzk0MzlmZmViZGUtYWY4_PF84_ba" +template-cred: + type: secret + data: + secret: "MGE3MjYwNTQtZGE4My00MTlkLWIzN2MtZjU5YTg3NDA2Yzk0MzlmZmViZGUtYWY4_PF84_bb" +offsite-cn-pipeline-cred: + type: secret + data: + secret: "MGE3MjYwNTQtZGE4My00MTlkLWIzN2MtZjU5YTg3NDA2Yzk0MzlmZmViZGUtYWY4_PF84_bb" +offsite-mb-pipeline-cred: + type: secret + data: + secret: "MGE3MjYwNTQtZGE4My00MTlkLWIzN2MtZjU5YTg3NDA2Yzk0MzlmZmViZGUtYWY4_PF84_bb" +``` + +#### Example + +Minimal project + +```yaml +# parameters.yaml +customerName: ACME Corp +name: ACME BSS +type: project +repositories: + - type: envgeneInstance + url: https://gitlab.example.com/acme/instance.git + token: creds.get('gitlab-token').secret + +# credentials.yaml +gitlab-token: + type: secret + data: + secret: dummy-token-value +``` + +Full project + +```yaml +# parameters.yaml +customerName: ACME Corp +name: ACME OSS +type: project +accessGroups: + - devops-team + - developers + - qa +clustersPlatform: k8s +repositories: + - type: envgeneInstance + url: https://gitlab.example.com/acme/instance.git + token: creds.get('gitlab-token').secret + defaultBranch: release/26.1 + region: cn + +# credentials.yaml +gitlab-token: + type: secret + data: + secret: dummy-token-value +``` + +## To discuss + +- [x] Use case for Colly using its own project repository: + 1. Read all projects and extract the URL, token, and branches from the `envgeneInstance` repositories in order to display the environments from these projects. + +- [x] Does each project currently have a different `accessGroups`? + - Yes, it is different + +- [x] Is creating Project post/patch via the Colly API planned? + - Not right now, maybe in the future + +- [x] `projectId` = customerAbbr + projectAbbr + +- [x] Should the Project repository be used as the Maintenance inventory? + - yes + +- [x] Need a mapping from environment to project. For example, an environment attribute `project`. + - Use case: find the DCL pipeline repository by environment + - Requestor - The Customer + +- [ ] global configuration + +- [ ] `pipeline` is too generic, we need to specify the exact type of pipeline + +- [ ] `accessGroups` is a list of user groups for "clusters" or for Colly? + - list of groups that can work with the project => used for access control in SSP: + - who can see/edit which project/cluster/env + +- [ ] `jiraCustomerName` - is it needed? + +- [x] `envgeneArtifact` + - Short term: + - In the Repository, `envgeneArtifact.name` is set manually in Git + - Long term: + - need to decide whether Colly can discover: + - `envgeneArtifact.name` + - `envgeneArtifact.templateDescriptorNames` diff --git a/docs/analisys/the-customer-requirnments.md b/docs/analisys/the-customer-requirnments.md new file mode 100644 index 00000000..b1c43bab --- /dev/null +++ b/docs/analisys/the-customer-requirnments.md @@ -0,0 +1,336 @@ +# API Enhancement according to the Customer requirements + +- [API Enhancement according to the Customer requirements](#api-enhancement-according-to-the-customer-requirements) + - [Description](#description) + - [Environment](#environment) + - [Namespace](#namespace) + - [Cluster](#cluster) + - [Colly instance](#colly-instance) + - [To discuss](#to-discuss) + - [To implement](#to-implement) + +## Description + +This document describes the requirements for Colly from The Customer and the changes they introduce to Colly. + +This is not the full list of attributes for these objects, but only those that will be processed as per The Customer’s requirements. + +## Environment + +| Colly Attribute | Attribute Type | Description | +|---------------------------------------------------|---------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `name` | string | Environment name, cannot be changed after creation | +| `status` | enum [`FREE`, `IN_USE`, `RESERVED`, `DEPRECATED`] | Current status of the Environment | +| `role` | string (the valid values are configured via a deployment parameter) | Defines the usage role of the Environment within the project. The list is configured via a deployment parameter and can be extended. | +| `teams` | list of strings | Teams assigned to the Environment. If there are multiple teams, their names are separated by commas. | +| `owners` | list of strings | People responsible for the Environment. If there are multiple, their names are separated by commas. | +| `deploymentOperations` | list of objects | Array of deployment operations for all namespaces related to the Environment. Each element represents a separate deployment operation that may involve one or more namespaces and consists of multiple SD (Solution Descriptor) deployments. | +| `deploymentOperations[].completedAt` | string, date-time | Time when the deployment operation finished. This operation may involve one or more namespaces that are part of the Environment. The time reflects when all deployment items in this operation completed, regardless of their individual outcomes. | +| `deploymentOperations[].deploymentItems` | list of objects | List of SD (Solution Descriptor) deployments that are part of this deployment operation. Each entry represents a single SD deployment with its details. | +| `deploymentOperations[].deploymentItems[].name` | string | Name and version of the SD in format `name:version`. | +| `deploymentOperations[].deploymentItems[].type` | string | Type of SD (e.g., "product", "project", etc.). | +| `deploymentOperations[].deploymentItems[].mode` | enum [`CLEAN_INSTALL`, `ROLLING_UPDATE`] | Deployment mode for the SD: CLEAN_INSTALL (remove all existing resources before deployment) or ROLLING_UPDATE (deploy on top of existing resources). | +| `accessGroups` | string | List of user groups that can work with the environment | +| `effectiveAccessGroups` | string | Resolved full list of user groups (contains groups and their descendants). resolved based on `accessGroups` | +| `description` | string | Free-form Environment description | +| `namespaces` | list of [Namespace](#namespace) objects | List of associated namespaces | +| `cluster` | [Cluster](#cluster) object | Associated cluster | +| `monitoringData.lastIdpLoginDate` | string, date-time | Time of the last successful login to the IDP associated with the Environment | +| `region` | string | Geographical region associated with the Environment. This attribute is user-defined | + +## Namespace + +| Colly Attribute | Attribute Type | Description | +|-----------------------------------------|----------------|----------------------------| +| `name` | string | Namespace name | + +## Cluster + +| Colly Attribute | Attribute Type | Description | +|-----------------------------------------|------------------|-------------------------------------------------------------------------------------------------------| +| `name` | string | Cluster name, cannot be changed after creation | +| `clusterInfoUpdateStatus.lastSuccessAt` | string | Time of the last successful update information from the cluster | + +## Colly instance + +| Colly Attribute | Attribute Type | Description | +|-----------------------------------------|----------------- |---------------------------------------------------------------| +| `clusterInfoUpdateInterval` | string, duration | Period of synchronization with the cluster in ISO 8601 format | + +## To discuss + +- [x] `lastIdpLoginDate` + - Implement as in the old Colly + +- [x] `ticketLinks` + - This is the last deploy ticket ID + - Not needed + +- [x] `type` + + - `type` was previously determined based on labels set by the Cloud Passport Discovery CLI, but this functionality is being removed. + - We will keep the attribute with the ability for users to set it manually. + - Open Questions (OQ): + 1. Should this attribute be computed by Colly (and if so, based on what criteria), or should it be user-defined? + 1. It should be user-defined, selected from a predefined list without ability to extend + 2. Does the customer need this type-based environment categorization? + 1. No, it is not required. + +- [x] `lastDeployedSDsByType` ?? + + - Determines the latest SD of a given type that was successfully deployed to one of the namespaces included in the environment. + - This attribute is used for CI environments. + - Shows the scope of the last successful deployment operation. + - Single field with a complex object: + + ```yaml + : + ``` + + for example + + ```yaml + product: product-sd:1.2.3 + project: my-project:1.2.3 + ``` + + - Mapping of SD type to SD name is specified in the Colly deployment parameters: + + ```yaml + solutionDescriptors: + : + - + ``` + + - Default value: + + ```yaml + solutionDescriptors: + product: + - (?i)product + project: + - (?i)project + ``` + +- [x] `deploymentOperations` + - This attribute contains information obtained from the configMap `sd_versions`. + - This attribute is temporary. In the future, the information source for deployment operations will be replaced by a dedicated service, which will require a change in the parameter model. + - DD deployment is out of scope. + +- [x] `status` + - Propose ![env-state-machine.drawio.png](/docs/images/env-state-machine.drawio.png) + 1. `PLANNED` Planned for deployment, but not yet deployed. It exists only as a record in Colly for planning purposes. + 2. `FREE` The Environment is successfully deployed in the cloud but is not used by anyone; it is ready for use and not scheduled for usage. + 3. `IN_USE` The Environment is successfully deployed in the cloud and is being used by a specific team for specific purposes. + 4. `RESERVED` The Environment is successfully deployed in the cloud and reserved for use by a specific team, but is not currently in use. + 5. `DEPRECATED` The Environment is not used by anyone, and a decision has been made to delete it. + - OQ: + 1. What are the cases? + 2. Should it be extendable? + 1. NO + 3. What is `to be deprecated`? Why do we not have `deprecated`, `deleted`, or `not used` states? + 4. Do we need `MIGRATING` (meaning the upgrade is in progress)? + +- [x] `clusterInfoUpdateInterval`, `clusterInfoUpdateStatus.lastSuccessAt` + 1. What are the main scenarios? + 1. The user wants to check that the cluster status shown in Colly is up to date and matches the real state of the cluster, so they can make decisions. + 1. Solution: + 1. Cluster attribute `clusterInfoUpdateStatus.lastSuccessAt` shows the time of the last successful sync + 2. Colly instance (operational service) attribute `clusterInfoUpdateInterval` shows the sync frequency + 2. The user thinks the cluster status data is outdated and wants to trigger a sync manually. + 1. Solution: We do NOT allow users to trigger the sync manually. + 3. An external system integrated with Colly thinks the data is outdated and wants to trigger sync. + 1. Solution: Use the API `/colly/v2/inventory-service/manual-sync` + 2. Should Colly support a forced `clusterInfoUpdate`, not by schedule but by user request? + 1. The customer will call this API as part of their workflow, but users in the UI will not have access to this. + 3. What is the recommended sync frequency with the cluster? + 1. At least once every 30 minutes. + +- [x] `role` + + - Should it be extendable? + - Currently, `role` values are extended via deployment parameters + - A separate interface to provide the list of roles is needed + - Challenge the predefined list of roles + - [`Dev`, `QA`, `Project CI`, ~~`SaaS`~~, `Joint CI`, ~~`Other`~~, `Pre Sale`] - set via deployment parameters + +- [x] `team` or `teams`? `owner` or `owners`? + - `owners`, `teams` are lists + +- [x] Each POST in the API will result in a separate commit + +- [x] `id` is `uuid`; `name` is `` + +- [x] The mediation layer composes the API between the inventory and operational services + +- [x] It should be possible to get a list of projects + - `/colly/v2/inventory-service/projects` + - returns a summary view: + - projectId + - projectName + - `/colly/v2/inventory-service/projects/Id` + - returns detail view + - The potential problem of a project with two instance repositories will be addressed when it arises + +- [x] It should be possible to get a list of clusters + - `/colly/v2/inventory-service/clusters` + - returns a summary view: + - clusterId + - projectName + - `/colly/v2/inventory-service/projects/Id` + - returns detail view + +- [x] Add the `deployPostfix` attribute to the Namespace? + - No + +- [x] Is it correct to say that a single physical business solution instance, consisting of product and project applications, can be modeled with two EnvGene environments - one for product, one for project? + - No, there is only one Environment. + +- [x] Before creating a cluster, the environment queries Colly to check if there is already a cluster with the same name. + +- [x] If Colly discovers the instance repo and receives a cluster that already exists in Colly (by the `name` attribute), another one is created with the same name but a unique ID. + +- [ ] What should be the scope of synchronization between Colly and the cluster + 1. All clusters of the instance + 2. Individual clusters + 3. Individual environments within a cluster + +- [ ] Lock + - The lock must answer the following questions: + - Status: locked or not locked + - Who locked it: free-form string + - Reason for locking: free-form string + - When it was locked: timestamp + - When it will be unlocked: date + - Required interfaces: + - Set/remove lock on the environment + - Force sync lock status from Git (can be implemented later) + - Only a Colly admin can lock through the UI; users cannot + - OQ: + 1. Should locks be defined by inventory backend? + 2. Who, when and why lock/unlock + 3. What are the cases from SSP? + +- [ ] What does the TheCustomer -> Colly -> EnvGene integration look like when creating an Environment? + +- [ ] How to aggregate DeploymentOperations across the namespaces of the environment? + +- [ ] The SaaS instance of Colly must support: + - How do we roll out a new version of Colly (canary deployment)? + - The mediation layer finds Colly via service mesh + +- [ ] It should be possible to get a list of environments per project + - Implement by adding a search by `projectId` parameter to `/colly/v2/inventory-service/environments` + - It is necessary to introduce the `projectId` attribute to the environment object + - The Customer does NOT require information about which `customerName` an environment belongs to, nor obtaining lists of environments by `customerName` + +- [ ] Add "default values" to Project attributes, for example: + - `app:ver` of the env template artifact + - Template name in the artifact + - Short term: + - Defaults are set manually in Git + Colly returns them as a project attribute + - Long term: + - ability to set them via API appears + - "Default values" are optional for the project + - There should be an ability to set "default values" at the global level. + - Who merges global into project?: + - Colly? + - Who else (fast click)? + - OQ: + - Do you need templateDescriptorNames or defaultTemplateDescriptorName? or both? + +- [ ] Environment groups + - `accessGroups` + - List of groups that can work with the project => used for access control in SSP: + - Who can see/edit which env + - Attribute in env inventory + - There should be an ability to set this attribute: + - When creating env (`env_inventory_generation`) + - `effectiveAccessGroups` + - Resolved full list of groups (contains groups and their descendants) + - Attribute in env inventory + - Need to display the resolved list at the time of env creation => need to save it somewhere + - OQ: + - Who computes `effectiveAccessGroups` + - not Colly + - Where is the `effectiveAccessGroups` cache stored + - in Git + - Only read or also write `effectiveAccessGroups` and `accessGroups` via Colly? + +- [ ] Is information about synchronization with git repositories (inventory service) needed: + + ```yaml + # colly operational service + projectGitInfoUpdateStatus.lastSuccessAt: + # cluster + clusterInfoUpdateStatus.lastSuccessAt: + # env + gitInfoUpdateStatus.lastSuccessAt: + ``` + +- [ ] deploymentOperations + +1. как определяем CLEAN_INSTALL (achka не отдает этот аттрибут) +2. переименовать mode CLEAN_INSTALL -> "" ROLLING_UPDATE -> "" + +- делаем группировку по deployment_session_id - получаем мапу deploymentSessionId -> список апликейшенов которые деплоились за эту сессию (возможно с нескольких СД) + - что делаем если ставились отдельно ДД? **просто игнорим** +- для каждой сессии достаем список аппликейшенов и группируем их в мапу sd_version -> список аппликейшенов +- для каждого списка аппликейшенов вычисляем + - самую раннюю (позднюю) дату деплоя + - статус - если хотя бы один апп упал, значит всё failed + - по имени СД вычисляем ее тип - PRODUCT/PROJECT + - mode - всегда rolling update - clean install определять по дате создания versions конфиг мапы?? + +- [ ] `/colly/inventory-service/v2/projectDefaults` + +этот интерфейс отдает то что лежит в `/defaults/parameters.yaml` в проектном гите + +- [ ] `MAVEN_REPO_NAME` + +- добавляем mavenRepoName аттрибут проекта +- нужно ли MAVEN_REPO_URL? если да, то он один на проект? + +```text +MAVEN_REPO_URL: https://artifactorycn.netcracker.com/ +MAVEN_REPO_NAME: pd.saas-global.mvn.group +``` + +## To implement + +- [x] Change environment attributes + 1. `team`(string) -> `teams`(list of strings) + 2. `owner`(string) -> `owners`(list of strings) +- [ ] `role` + - [x] Add `role` attribute on Environment + - [ ] Add deployment parameter to extend `role` valid values + - [x] Remove default value for `role` + - [ ] Implement an interface (`/colly/operational-service/v2/metadata`) that returns the list of `role` valid values (Low priority) +- [x] `type` + - [x] Remove the functionality for auto-generating the `type` attribute. Users should be able to set this value themselves by selecting from a list of allowed values. The list of values should be specified as a deployment parameter. +- [ ] Include the current Colly API version in the X-API-Version HTTP response header for every API response (Low priority) +- [x] Add `lastIdpLoginDate` attribute (via configurable monitoringData) +- [x] Add deployment parameter for `monitoringData` extension +- [x] Remove `ticketLinks` attribute +- [ ] Add `deploymentOperations` ( will do via ACHKA in 26.1 ) **P2** + - [ ] Remove `cleanInstallationDate` +- [x] Add `region` attribute +- [x] Add `clusterInfoUpdateInterval`, `clusterInfoUpdateStatus.lastSuccessAt` + remove `synced` **2.5.0** +- [x] Ability to sync per project, not per Colly instance **2.4.0** +- [x] Add `accessGroups` to Project **2.4.0** +- [x] Add `clustersPlatform` to Project +- [x] Add `envgeneArtifact` to Project +- [x] Create default configuration for `monitoringData` +- [ ] Support cred macro +- [x] `numberOfNodes` **2.5.0** +- [ ] The configuration for `monitoringData` is currently shared across all environments - it needs to be made more granular +- [x] Add `accessGroups`, `effectiveAccessGroups` to Environment. **2.3.0** +- [ ] add handling of Redis failure - if it fails, need to re-sync with Git and clusters +- [ ] add Redis probe - completed discovery +- [ ] add anti-affinity rules +- [x] add `deployPostfix` **2.4.0** +- [ ] add BG support for `deployPostfix` +- [ ] Add `owners` to Cluster. Agree where to store +- [ ] Support `inventory.cloudPassport` during Cluster "discovery". Priority is unclear, not doing it yet +- [x] support branch for instance repo **2.5.0** +- [ ] Add `ACHKA_URL` finding logic diff --git a/docs/analisys/values.yaml b/docs/analisys/values.yaml new file mode 100644 index 00000000..19f230f8 --- /dev/null +++ b/docs/analisys/values.yaml @@ -0,0 +1,50 @@ +--- +CLOUD_PUBLIC_HOST: +NAMESPACE: qubership-colly +colly: + global: + envgeneInventoryService: + image: ghcr.io/netcracker/envgene-inventory-service:latest + serviceName: envgene-inventory-service + instancesRepo: + gitToken: + cronSchedule: 0 0/1 * * * ? + idp: + url: + clientId: envgene-inventory-service + clientSecret: + redis: + hosts: redis://redis:6379 + username: "" + password: "" + ports: + http: 8080 + environmentOperationalService: + image: ghcr.io/netcracker/environment-operational-service:latest + serviceName: environment-operational-service + cronSchedule: 0 0/1 * * * ? + idp: + url: + clientId: environment-operational-service + clientSecret: + redis: + hosts: redis://redis:6379 + username: "" + password: "" + ports: + http: 8080 + ingress: + className: alb + http: + annotations: + alb.ingress.kubernetes.io/scheme: internet-facing + alb.ingress.kubernetes.io/target-type: ip + environmentRoles: + - DEV + - QA + - PROJECT_CI + - JOINT_CI + monitoringData: + lastIdpLoginDate: + name: Last IDP Login Date + query: count(kube_pod_status_phase{namespace=~"{namespace}",phase="Running"}) \ No newline at end of file diff --git a/docs/images/colly-model.png b/docs/images/colly-model.png new file mode 100644 index 00000000..ed9163e4 Binary files /dev/null and b/docs/images/colly-model.png differ diff --git a/docs/images/env-state-machine.drawio.png b/docs/images/env-state-machine.drawio.png new file mode 100644 index 00000000..a420d40c Binary files /dev/null and b/docs/images/env-state-machine.drawio.png differ