Skip to content

Commit

Permalink
PMM-10994 New integration setup (#535)
Browse files Browse the repository at this point in the history
* Reads command line arguments

* Splits setting up database

* Returns help line

* POC for node js integration testing

* remove unused file

* removes unused file

* returns file to original folder

* adds suport for shell script integrations

* moved pgsql_pgss_setup to pgsql folder

* correct location of yaml file

* adds pgsql_pgsm_setup

* lists folder

* change location of yaml files

* try different format for ansible-palybook

* Moves necessary files

* Fix location for playbooks

* Moving files for pgss setup

* adds missing steps for vacuum

* escaping characters

* adds new random variables

* tests with logs

* fixes commands

* Fixes wrong variable name

* adds javascript loop

* adds remaining command to the javascript

* Adds remaining command to the javascript

* fixes wrong variable name

* Removes unused variables

* Moves function to separate array, refactors

* Removes wrong branch

* Adds psmdb setup

* change psmdb setup location to mongo folder

* Fix location for yaml file

* PMM-10994: Adds test flags to the integration setup

* set github action env from node js

* Change file location

* remove pushd

* switches ansible install to ubuntu variant

* List folder

* changes location of yaml file.

* Remove logged data

* comment erroring lines

* Refactors

* comments out issues

* Fix import

* Uncoment step for regular load

* moves mongodb create users file

* Log just an error

* move ansible file to separate command

* Location of ansible file for pgss

* comment out pushd

* Fix file location for pgsm

* command line refactoring

* Log for pgsql vacuum setup

* removes catching command line errors

* Higher limit for vacuum

* Debug values

* change pgsql version

* Log pgsql version

* log error

* Better logging

* PMM-10994: Setting up pmm client from tarball

* pmm client in container added comments

* PGSM integration in typescript

* adds required env variables

* refactoring

* log output of ansible exectution

* log response from ansible command

* change command for ansible

* Instal ansible

* Log steps in node js

* set python intepreter

* execute comand and log response

* refactor execute ansible playbook

* return to the shell script

* Adds previously removed file.

* Adds previously removed file.

* Adds previously removed file.

* Adds previously removed file.

* moves yaml files

* Adds previously removed file.

* Adds previously removed file.

* Adds previously removed file.

* PMM-10994 Add PMM PS Integration (#537)

* prototype of pmm ps integration

* Log query source

* Fix env variable assigment

* added python to ansible

* change log

* Refactor

* Change command execution

* Added missing shell file

* Echo tarball

* Fixes ps tarball variable

* fix for ps tarbal in ansible

* fix for the tarball

* Refactors

* remove stoutd for ps tarball

* echo log file

* Pmm 10994 add haproxy setup (#543)

* adds haproxy setup

* changes python intepreter

* refactors

* Refactors

* Refactors

* refactors

* change location of setup script

* Fix Typo

* Refactors

* Refactors

* Moves ansible command to the javascript

* Fixes yaml formatting

* Uncomment another step in haproxy setup

* Uncomments more steps

* comments out steps

* install pmm client

* do not log script file

* comments out

* uncoment

* uncomment

* Pmm 10994 add external service (#544)

* Test for external service

* fixes typo

* fixes exporter version

* changes to move external setup to the typescript

* refactors

* PMM-10994 Add github pipeline (#542)

* adds poc of the pipeline

* Changes the name of the workflow

* fixes start pmm server command

* Fixes network in docker run command

* Fixes docker run typo

* Creates docker network

* Changes ports

* change directory

* lists directory

* adds npm install

* install ansible in nodejs

* CI flag

* Env variable for pmm client step

* echo env variable

* different package

* env variable for the script

* change env variable

* log all variables

* objects of env variables to string

* env variable different setup

* correct location for env file

* read file in correct folder

* read env file

* fix for the reading of the variable

* correct setting up env variable and job name

* ACTIONS_ALLOW_UNSECURE_COMMANDS

* execute pgsm setup as root

* refactors

* execute ansible as root

* install pmm client

* Add correct network to the containers

* List all containers.

* Fixes network flag

* Change pmm server address

* container already in the network

* Fix network name

* Refactors

* change tests to the correct format.

* adds pgsql version to the name of the container

* refactors

* refactors pgss and adds new test report

* log correct data

* Fix pgsm and pgss ansible playbooks

* correct pgsm container name

* fixes containers names

* log env variables

* correct name of the method

* Refactors

* echo env variables

* echo env variables

* console log which variable is set

* print all env variables

* Change default env variable

* log when ansible command starts

* refactors and change pgsql version

* pass variable to yaml file

* set default version for versions

* refactors

* fix variable type in yaml

* Change default pgsql version

* addd flag for pgsql version

* refactor generation of the test

* add token to creating reports

* Refactors pmm ps integration setup

* adds ps version as env variable

* fix ps setup yaml

* fix setting up default value for ps version

* change in command to run ansible playbook

* refactor ps setup yaml

* Fixes typo

* refactors psmdb setup

* refactor psmdb yaml setup

* list running containers for debugging

* Adds python interpreter for psmdb setup

* Fixes typo in psmdb ansible playbook

* git list pmm admin for ps setup

* change container name for integration testing

* remove ps debug command

* fixes in psmdb setup yaml file

* adds parameters to psmdb setup

* newer version of nodejs

* no warnings for mongodb and change command to setup users

* return node version to the older one

* fix calling pmm server haproxy config

* Run haproxy wit elevated privileges

* remove installing of ansible from haproxy setup

* uncomment rest of the haproxy setup

* install debian package for exporter

* docker pmm client

* refactors docker run for pmm client

* run pmm client in detached mode

* custom version of pmm client

* pass custom pmm client version into pgss setup

* Fixes typos

* adds available args for pmm client version

* client version for pgsm and ps setup

* custom ps version

* add ps version to available args

* adds mongodb replica for backup

* Log output of command line

* change flag for mongo replica for backup

* Change in external service config

* run pmm client connect as root

* connect to the docker pmm client for external service

* Remove test parallel

* Refactors

* console log of output for ansible

* Fix psmdb setup

* handle empty psmdb tarball url

* set default value for psmdb tarball url

* Fixes wrong location of version file

* Refactors

* Removes pmm-qa network from pgss setup

* mongo version and mongo setup type

* add variable for mo-setup

* Fixes Typo

* removes unused pgsm and pgss configs

* Remove autorun of the job

* Refactors

* Correct action name

* print node js version

* refactors the pipeline

* Comment to explain validation of args

* Removed not working external setup and added readme
  • Loading branch information
peterSirotnak authored Jan 7, 2023
1 parent cff6871 commit b86f53c
Show file tree
Hide file tree
Showing 37 changed files with 7,461 additions and 23 deletions.
153 changes: 153 additions & 0 deletions .github/workflows/pmm2-integration-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
---
name: pmm2-integration-tests

on:
workflow_dispatch:
inputs:
pmm_ui_tests_branch:
description: 'Target branch for pmm-ui-tests repository'
default: 'main'
required: true
pmm_qa_branch:
description: 'Target branch for pmm-qa repository'
default: 'main'
required: true
integration_setup:
description: 'Add flag to test selected integration setup'
default: '--setup-pmm-pgsm-integration'
required: true
pmm_server_version:
description: 'Provide version of pmm server'
default: 'dev-latest'
required: true
pmm_client_version:
description: 'Provide version of pmm server'
default: 'dev-latest'
required: true
pgsql_version:
description: 'Provide version of PostgreSQL'
default: '15'
required: true
ps_version:
description: 'Provide version of Percona Server for MySQL'
default: '8.0'
required: true
mongo_version:
description: 'Provide version of Mongo DB'
default: '4.4'
required: true
mongo_setup:
description: 'Provide setup type for Mongo DB'
type: choice
options:
- regular
- replica
- sharded
default: 'regular'
required: true

jobs:
integration-tests:
name: Integration Tests
runs-on: ubuntu-latest
timeout-minutes: 60
env:
SHA: ${{ github.event.inputs.sha || 'null' }}
ACTIONS_ALLOW_UNSECURE_COMMANDS: true

UI_TESTS_BRANCH: ${{ github.event.inputs.pmm_ui_tests_branch }}
QA_TESTS_BRANCH: ${{ github.event.inputs.pmm_qa_branch }}
INTEGRATION_SETUP: ${{ github.event.inputs.integration_setup }}
PMM_SERVER_VERSION: ${{ github.event.inputs.pmm_server_version }}
PMM_CLIENT_VERSION: ${{ github.event.inputs.pmm_client_version }}

PGSQL_VERSION: ${{ github.event.inputs.pgsql_version }}
PS_VERSION: ${{ github.event.inputs.ps_version }}
MONGO_VERSION: ${{ github.event.inputs.mongo_version }}
MONGO_SETUP: ${{ github.event.inputs.mongo_setup }}



steps:
- name: Tests of pmm server version ${{ github.event.inputs.pmm_server_version }} for integration flag ${{ github.event.inputs.integration_setup }}
if: ${{ github.event_name == 'workflow_dispatch' && env.SHA != 'null' }}
uses: Sibz/github-status-action@v1
with:
context: 'pmm2-ui-tests'
description: 'Tests execution has been started'
state: 'pending'
repository: ${{ github.event.inputs.repo }}
target_url: 'https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}'
sha: ${{ env.SHA }}

- name: Install NodeJS v16
uses: percona-platform/setup-node@v2
with:
node-version: 16.14.1

- name: Checkout UI tests
uses: percona-platform/checkout@v2
with:
ref: ${{ env.UI_TESTS_BRANCH }}
repository: percona/pmm-ui-tests
path: ./pmm-ui-tests

- name: Checkout pmm-qa Repo
uses: percona-platform/checkout@v2
with:
repository: percona/pmm-qa
path: ./pmm-qa
ref: ${{ env.QA_TESTS_BRANCH }}

- name: Start PMM-Sever
run: |
docker network create pmm-integration-network
docker run -d --restart always -e PERCONA_TEST_PLATFORM_ADDRESS=https://check-dev.percona.com:443 --network="pmm-integration-network" --publish 80:80 --publish 443:443 --name pmm-integration-server perconalab/pmm-server:${{ env.PMM_SERVER_VERSION }}
- name: Create env file
working-directory: pmm-qa/pmm-integration
run: |
touch .env
echo CI=true >> .env
- name: Setup Docker client for PMM-Server
run: |
docker run -d --name pmm-client --network="pmm-integration-network" -e PMM_AGENT_SERVER_ADDRESS=pmm-integration-server -e PMM_AGENT_SERVER_USERNAME=admin -e PMM_AGENT_SERVER_PASSWORD=admin -e PMM_AGENT_SERVER_INSECURE_TLS=1 -e PMM_AGENT_SETUP=1 -e PMM_AGENT_CONFIG_FILE=config/pmm-agent.yaml perconalab/pmm-client:${{ env.PMM_CLIENT_VERSION }}
- name: Setup Databases for PMM-Server
working-directory: pmm-qa/pmm-integration
run: |
sudo npm install
sudo npx ts-node ./integration-setup.ts --pmm-client-version=${{ env.PMM_CLIENT_VERSION }} --pgsql-version=${{ env.PGSQL_VERSION }} --ps-version=${{ env.PS_VERSION }} --mo-version=${{ env.MONGO_VERSION }} --mo-setup=${{ env.MONGO_SETUP }} ${{ env.INTEGRATION_SETUP }}
shell: bash

- name: Execute Integration Tests
working-directory: pmm-ui-tests
id: pmm-ui-tests
run: |
sudo docker ps -a
node -v
npm install
./node_modules/.bin/codeceptjs run --debug --steps --reporter mochawesome -c pr.codecept.js --grep '${{ env.INTEGRATION_FLAG }}'
- name: Generating the report
if: ${{ always() && (steps.pmm-ui-tests-all.outcome != 'skipped' || steps.pmm-ui-tests-tagged.outcome != 'skipped' ) }}
working-directory: pmm-ui-tests
run: |
npx mochawesome-merge tests/output/*/*.json > tests/output/output.json
- name: Create the report
uses: phoenix-actions/test-reporting@v10
if: success() || failure()
with:
token: ${{ secrets.ROBOT_TOKEN }}
name: Integration Tests
path: pmm-ui-tests/tests/output/output.json
reporter: mocha-json

- name: Attaching the report
if: ${{ always() && (steps.pmm-ui-tests-all.outcome != 'skipped' || steps.pmm-ui-tests-tagged.outcome != 'skipped' ) }}
uses: percona-platform/upload-artifact@v2
with:
name: pmm-ui-tests-report
path: pmm-ui-tests/tests/output
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
node_modules/*
pmm-integration/node_modules/*
tests/output/*
.idea
pmm-tests/backup/.vagrant
Expand Down
6 changes: 6 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

99 changes: 99 additions & 0 deletions pmm-integration/.eslintrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
{
"extends": ["react-app", "airbnb", "prettier"],
"env": {
"jest/globals": true,
"browser": true
},
"root": true,
"ignorePatterns": "src/core/gen/**/*",
"settings": {
"react": {
"version": "detect"
},
"import/resolver": {
"node": {
"extensions": [".ts", ".tsx"]
}
}
},
"plugins": ["react-hooks", "jest", "eslint-plugin-no-inline-styles", "prettier"],
"rules": {
"prettier/prettier": "off",
"no-console": ["warn", { "allow": ["warn", "error"] }],
"semi": ["error", "always"],
"comma-dangle": [
"error",
{
"arrays": "always-multiline",
"objects": "always-multiline",
"imports": "always-multiline",
"exports": "always-multiline",
"functions": "always-multiline"
}
],
"no-restricted-imports": "error",
"no-param-reassign": ["error", { "ignorePropertyModificationsForRegex": ["^acc$"] }],
"react/jsx-curly-brace-presence": ["error"],
"react/jsx-boolean-value": ["error"],
"react/jsx-filename-extension": ["off"],
// It's a bad practice, but in certain use cases we have no choice
"react/jsx-props-no-spreading": ["off"],
"react/prop-types": ["off"],
"react/display-name": ["off"],
"react/no-array-index-key": ["error"],
"max-len": ["error", { "code": 110, "ignoreStrings": true, "ignoreRegExpLiterals": true }],
"quotes": ["error", "single", { "avoidEscape": true }],
"import/extensions": ["off"],
"import/no-extraneous-dependencies": [
"error",
{ "devDependencies": true, "optionalDependencies": false, "peerDependencies": false }
],
"import/prefer-default-export": ["off"],
"import/no-unresolved": ["off"],
// no-unused-vars doesn't work with typescript
"no-unused-vars": "off",
"@typescript-eslint/no-unused-vars": ["error"],
// We use a lot of snake case variables in code interacting with API
"camelcase": ["off"],
// It's more convinient to declare big helpers functions below the main method
"no-use-before-define": ["off"],
"jsx-a11y": ["off"],
"jsx-a11y/anchor-is-valid": ["off"],
"jsx-a11y/click-events-have-key-events": ["off"],
"jsx-a11y/no-static-element-interactions": ["off"],
"jsx-a11y/no-noninteractive-element-interactions": ["error"],
"jsx-a11y/label-has-associated-control": ["error"],
"no-dupe-args": ["error"],
"no-new-object": ["error"],
"padding-line-between-statements": [
"error",
{ "blankLine": "always", "prev": "*", "next": "return" },
{ "blankLine": "always", "prev": "import", "next": "*" },
{ "blankLine": "any", "prev": "import", "next": "import" },
{ "blankLine": "always", "prev": "function", "next": "*" },
{ "blankLine": "always", "prev": "if", "next": "*" },
{ "blankLine": "always", "prev": "for", "next": "*" },
{ "blankLine": "always", "prev": ["const", "let", "var"], "next": "*" },
{ "blankLine": "any", "prev": ["const", "let", "var"], "next": ["const", "let", "var"] }
],
"no-inline-styles/no-inline-styles": ["warn"],
"max-classes-per-file": ["error"],
"no-shadow": ["off"],
"@typescript-eslint/no-shadow": ["error"],
"no-unused-expressions": ["error"],
"arrow-parens": ["error"],
"eol-last": ["error"]
},
"overrides": [
{
"files": ["**/*.test.ts", "**/*.test.tsx"],
"env": {
"jest": true
},
"plugins": ["jest"],
"rules": {
"no-magic-numbers": ["off"]
}
}
]
}
15 changes: 15 additions & 0 deletions pmm-integration/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
New setup that will create start and integrate PMM2 with selected databases. Can be used locally or in CI.

Basic how to use integration setup:

sudo npx ts-node ./integration-setup.ts --selected-flag

for example:

sudo npx ts-node ./integration-setup.ts --setup-pmm-ps-integration

If you left all other flags like database version or pmm version empty the latest or dev-latest version fill be used.

All available flags are present in file:

availableArgs.ts
111 changes: 111 additions & 0 deletions pmm-integration/availableArgs.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
import { executeAnsiblePlaybook, executeCommand, setEnvVariable } from "./helpers/commandLine";
import SetupParameters from "./helpers/setupParameters.interface";
import pgsqlVacuumSetup from "./postgres/pgsql-vacuum-setup";
import * as core from '@actions/core';
import installDockerCompose from "./otherConfigs/installDockerCompose";
import clearAllSetups from "./otherConfigs/clearAllSetups";

export interface SetupsInterface {
arg: string;
description: string;
function: (parameters: SetupParameters) => Promise<void>;
}

export const availableSetups: SetupsInterface[] = [
{
arg: '--setup-pgsql-vacuum',
description: 'Use this do setup postgres for vacuum monitoring tests',
function: async (parameters: SetupParameters) => {
await pgsqlVacuumSetup(parameters);
await setEnvVariable('INTEGRATION_FLAG', '@pgsql_vacuum');
core.exportVariable('INTEGRATION_FLAG', '@pgsql_vacuum');
},
},
{
arg: '--setup-pmm-pgsm-integration',
description: 'Use this option to setup PMM-Client with PGSM for integration testing',
function: async (parameters: SetupParameters) => {
await executeAnsiblePlaybook(`sudo ansible-playbook --connection=local --inventory 127.0.0.1, --limit 127.0.0.1 ./postgres/pgsql_pgsm_setup/pgsql_pgsm_setup.yml -e="PGSQL_VERSION=${parameters.pgsqlVersion} CLIENT_VERSION=${parameters.pmmClientVersion}"`)
core.exportVariable("INTEGRATION_FLAG", "@pgsm-pmm-integration");
},
},
{
arg: '--setup-pmm-pgss-integration',
description: 'Use this option to setup PMM-Client with PG Stat Statements for Integration Testing',
function: async (parameters: SetupParameters) => {
await executeAnsiblePlaybook(`sudo ansible-playbook --connection=local --inventory 127.0.0.1, --limit 127.0.0.1 ./postgres/pgsql_pgss_setup/pgsql_pgss_setup.yml -e="PGSQL_VERSION=${parameters.pgsqlVersion} CLIENT_VERSION=${parameters.pmmClientVersion}"`)
core.exportVariable("INTEGRATION_FLAG", "@pgss-pmm-integration");
},
},
{
arg: '--setup-pmm-psmdb-integration',
description: 'Use this option for Percona MongoDB setup with PMM2',
function: async (parameters: SetupParameters) => {
await executeCommand('chmod +x ./mongoDb/mongo_psmdb_setup/setup_pmm_psmdb_integration.sh');
await executeCommand('./mongoDb/mongo_psmdb_setup/setup_pmm_psmdb_integration.sh');
await executeCommand(`sudo ansible-playbook --connection=local --inventory 127.0.0.1, --limit 127.0.0.1 ./mongoDb/mongo_psmdb_setup/psmdb_setup.yml -e="CLIENT_VERSION=${parameters.pmmClientVersion} PSMDB_TARBALL=${parameters.psmdbTarballURL} PSMDB_VERSION=${parameters.moVersion} PSMDB_SETUP=${parameters.moSetup}"`);
await setEnvVariable("INTEGRATION_FLAG", "@pmm-psmdb-integration");
},
},
{
arg: '--setup-pmm-haproxy-integration',
description: 'Use this option for Haproxy setup with PMM2',
function: async (parameters: SetupParameters) => {
await executeCommand('chmod +x ./otherConfigs/haproxy/pmm-haproxy-setup.sh');
console.log(await executeCommand('./otherConfigs/haproxy/pmm-haproxy-setup.sh'));
console.log(await executeCommand('sudo ansible-playbook --connection=local --inventory 127.0.0.1, --limit 127.0.0.1 ./otherConfigs/haproxy/haproxy_setup.yml'));
await setEnvVariable('INTEGRATION_FLAG', '@pmm-haproxy-integration');
core.exportVariable('INTEGRATION_FLAG', '@pmm-haproxy-integration');
}
},
{
arg: '--setup-pmm-ps-integration',
description: 'Use this option for percona-server and PMM using dbdeployer',
function: async (parameters: SetupParameters) => {
await executeCommand('chmod +x ./mysql/pmm_ps_integration/pmm_ps_integration.sh');
await executeCommand('./mysql/pmm_ps_integration/pmm_ps_integration.sh');
await executeAnsiblePlaybook(`sudo ansible-playbook --connection=local --inventory 127.0.0.1, --limit 127.0.0.1 ./mysql/pmm_ps_integration/ps_pmm_setup.yml -e="PS_VERSION=${parameters.psVersion} CLIENT_VERSION=${parameters.pmmClientVersion}"`);
await setEnvVariable("INTEGRATION_FLAG", "@pmm-ps-integration");
},
},
{
arg: '--mongo-replica-for-backup',
description: 'Use this option to setup MongoDB Replica Set and PBM for each replica member on client node',
function: async (parameters: SetupParameters) => {
await installDockerCompose();
await executeCommand('chmod +x ./mongoDb/mongo_replica_for_backup/setup_mongo_replica_for_backup.sh');
await executeCommand('./mongoDb/mongo_replica_for_backup/setup_mongo_replica_for_backup.sh');
await setEnvVariable("INTEGRATION_FLAG", "@fb");
},
},
{
arg: '--clear-all-setups',
description: 'Use this to clear your local env of any integration setups.',
function: async (parameters: SetupParameters) => {
await clearAllSetups();
await setEnvVariable('INTEGRATION_FLAG', '@pgsql_vacuum');
core.exportVariable('INTEGRATION_FLAG', '@pgsql_vacuum');
},
}
];

export const availableConstMap = new Map<string, string>([
['--pgsql-version', 'Pass Postgre SQL server version Info'],
['--mo-version', 'Pass MongoDB Server version info'],
['--mo-setup', 'Pass MongoDB Server type info'],
['--ps-version', 'Pass Percona Server version info'],
['--setup-pmm-client-tarball', 'Sets up pmm client from provided tarball'],
['--pmm-client-version', 'Version of pmm client to use, default dev-latest']
]);

export const availableSetupMap = new Map(
availableSetups.map((object) => {
return [object.arg, object.description];
})
);

export const availableArgsMap = new Map<string, string>([...availableConstMap, ...availableSetupMap]);

const availableCommandsLineArgs: string[] = Array.from(availableArgsMap.keys());

export default availableCommandsLineArgs;
Loading

0 comments on commit b86f53c

Please sign in to comment.