diff --git a/README.md b/README.md index 8a4c840..187c3d4 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,9 @@ -# Ping Application Example Pipeline +# Ping Application Example Pipeline This repository is intended to present a simplified reference demonstrating how a management and deployment pipeline might work for applications that depend on services managed by a central IAM platform team. As such, it is a complement to the [infrastructure](https://github.com/pingidentity/pipeline-example-infrastructure) and [platform](https://github.com/pingidentity/pipeline-example-platform) example pipeline repositories. -> NOTE: This repository directly depends on a completed set-up of the [pipeline-example-application](https://github.com/pingidentity/pipeline-example-platform?tab=readme-ov-file#deploy-prod-and-qa). Please ensure you have completed the steps for confguration leading up to and including the previous link. +> [!IMPORTANT] +> This repository directly depends on a completed setup of the [pipeline-example-platform](https://github.com/pingidentity/pipeline-example-platform?tab=readme-ov-file#deploy-prod-and-qa). Please ensure you have completed the steps for configuration leading up to and including the previous link, where a `prod` and `qa` environment have been deployed. **Infrastructure** - Components dealing with deploying software onto self-managed Kubernetes infrastructure and any configuration that must be delivered directly via the filesystem. @@ -26,17 +27,43 @@ In this repository, the processes and features shown in a GitOps process of deve To be successful in recreating the use cases supported by this pipeline, there are initial steps that should be completed prior to configuring this repository: -- Completion of all pre-requisites and confiuration steps leading to [Feature Development](https://github.com/pingidentity/pipeline-example-platform?tab=readme-ov-file#feature-development) in the example-pipeline-platform. -- [Docker](https://docs.docker.com/engine/install/) - used to deploy a local sample application. +- Completion of all pre-requisites and configuration steps leading to [Feature Development](https://github.com/pingidentity/pipeline-example-platform?tab=readme-ov-file#feature-development) from the example-pipeline-platform repository +- [Docker](https://docs.docker.com/engine/install/) - used to deploy the UI for a sample interface +- [tflint](https://github.com/terraform-linters/tflint) - for Terraform linting +- [dvlint](https://github.com/pingidentity/dvlint) - for Davinci flow linting +- [trivy](https://github.com/aquasecurity/trivy) - for security scanning + +> [!TIP] +> The last three tools are used by the pipeline in Github, and the pipeline will fail if these tests and configuration checks do not pass. Installing these tools locally and running `make devcheck` before committing changes should ensure that the pipeline will pass when changes are pushed. -> Note - For PingOne, meeting these requirements means you should have credentials for a worker app residing in the "Administrators" environment that has organization-level scoped roles. For DaVinci, you should have credentials for a user in a non-"Administrators" environment that is part of a group specifically intended to be used by command-line tools or APIs with environment-level scoped roles. +> [!IMPORTANT] +> For PingOne, meeting these requirements means you should have credentials for a worker app residing in the "Administrators" environment that has organization-level scoped roles. For DaVinci, you should have credentials for a user in a non-"Administrators" environment that is part of a group specifically intended to be used by command-line tools or APIs with environment-level scoped roles. + +### Development Environment + +If you have not created a static development environment in your PingOne account, you can do so in the local copy of the *pipeline-example-platform* repository by running the following commands to instantiate one matching prod and qa: + +```bash +git checkout prod +git pull origin prod +git checkout -b dev +git push origin dev +``` + +Capture the environment ID for the development environment for use later. + +> [!NOTE] +> The platform team may support ephemeral development environments rather than the static environment mentioned here. For purposes of this guide, we will assume that the development environment is shared and will be used by multiple developers. + +![PingOne Environments](./img/pingOneEnvs.png "PingOne Environments") ### Repository Setup Click the **Use this template** button at the top right of this page to create your own repository. After the repository is created, clone it to your local machine to continue. The rest of this guide will assume you are working from the root of the cloned repository. -> Note - A pipeline will run and fail when the repository is created. This result is expected as the pipeline is attempting to deploy and the necessary configuration has not been performed. +> [!NOTE] +> A pipeline will run and fail when the repository is created. This result is expected as the pipeline is attempting to deploy the application and the necessary configuration has not yet been completed. ## Development Lifecycle Diagram @@ -46,102 +73,207 @@ The use cases in this repository follow a flow similar to this diagram: ## Before You Start -There are a few items to configure before you can use this repository effectively. +There are a few items to configure before you can successfully use this repository. + +### PingOne Environments + +> [!IMPORTANT] +> The configurations in this sample repository rely on environments created from [pipeline-example-platform](https://github.com/pingidentity/pipeline-example-platform). For the `PINGONE_TARGET_ENVIRONMENT_ID_PROD` and `PINGONE_TARGET_ENVIRONMENT_ID_QA` variables needed down below, get the Environment ID for the `prod` and `qa` environments. The Environment ID can be found from the output at the end of a terraform apply (whether from the Github Actions pipeline, or local) or directly from the PingOne console. + +### Github CLI + +The Github cli: `gh` will need to be configured for your repository. Run the command **gh auth login** and follow the prompts. You will need an access token for your Github account and the repository created from this template: + +```bash +gh auth login + +? What account do you want to log into? GitHub.com +? You're already logged into github.com. Do you want to re-authenticate? Yes +? What is your preferred protocol for Git operations? HTTPS +? Authenticate Git with your GitHub credentials? Yes +? How would you like to authenticate GitHub CLI? Paste an authentication token +Tip: you can generate a Personal Access Token here https://github.com/settings/tokens +The minimum required scopes are 'repo', 'read:org', 'workflow'. +? Paste your authentication token: **************************************** +- gh config set -h github.com git_protocol https +✓ Configured git protocol +✓ Logged in as +``` ### Github Actions Secrets -The Github pipeline actions will depend on sourcing some secrets as ephemeral environment variables. To prepare the secrets in the repository: +The Github pipeline actions depend on sourcing secrets as ephemeral environment variables. To prepare the secrets in the repository: ```bash cp secretstemplate localsecrets ``` -> Note, `secretstemplate` is intended to be a template file, `localsecrets` is a file that contains credentials but is part of .gitignore and should never be committed into the repository. **`secretstemplate`** is committed to the repository, do not edit it directly! +> [!CAUTION] +> `secretstemplate` is a template file while `localsecrets` contains credentials. `localsecrets` is part of *.gitignore* and should never be committed into the repository. **`secretstemplate`** is committed to the repository, so ensure that you do not edit it directly or you risk exposing your secrets. -Fill in `localsecrets` accordingly. The configurations in this repository rely on environments created from [pipeline-example-platform](https://github.com/pingidentity/pipeline-example-platform). For the `PINGONE_TARGET_ENVIRONMENT_ID_PROD` and `PINGONE_TARGET_ENVIRONMENT_ID_QA` variables, get the Environment ID for the `prod` and `qa` environments. The Environment ID can be found from the output at the end of a terraform apply (whether from the Github Actions pipeline, or local) or from the PingOne console directly. +Fill in `localsecrets` accordingly, referring to the comments in the file for guidance. Many of the values needed for this file can be found in the corresponding localsecrets file from the platform repository. -If you have not created a static development environment in your PingOne account, you can do so in the *pipeline-example-platform* local repository by running the following command to instantiate one matching prod and qa: +After updating the file, run the following commands to upload **localsecrets** to Github: ```bash -git checkout prod -git pull origin prod -git checkout -b dev -git push origin dev +_secrets="$(/usr/bin/base64 -i localsecrets)" +gh secret set --app actions TERRAFORM_ENV_BASE64 --body $_secrets +unset _secrets ``` -Enter the corresponding Environment ID into localsecrets in your application repository for `PINGONE_TARGET_ENVIRONMENT_ID_DEV`. This action will leave you with three persistent environments `prod`, `qa` and `dev`, each with a corresponding environment variable definition pair in the `localsecrets` file. +## Development Example Overview -![PingOne Environments](./img/pingOneEnvs.png "PingOne Environments") +To experience the developer's perspective, a walkthrough follows. The demonstration will simulate the use case of modifying a Davinci flow and promoting the change. To simplify the demonstration, a starting pre-configured flow will be created using Terraform. The UI components will be built into a Docker image and launched on your local machine. After you have deployed the flow, you will be able to make the changes necessary in the PingOne UI, export the configuration, and promote the change to the QA and Prod environments. + +## Feature Development + +Now that the repository and pipeline are configured, the typical git flow can be followed. You will follow steps similar to those documented in the [pipeline-example-platform "Feature Development"](https://github.com/pingidentity/pipeline-example-platform/tree/prod?tab=readme-ov-file#feature-development) section. + +A notable difference between this repository and the platform example is that the application pipeline does NOT deploy "development" feature configurations. Unlike QA and Prod, feature configuration *deployment* only takes place from the local machine and the development environment ID is not stored in the repository. In a following step when the `./scripts/local_feature_deploy.sh` script runs, you will be prompted for the environment ID. + +This deployment format accounts for static and ephemeral development environment setups, and further helps avoid the possibility that a developer changes a shared variable that impacts another engineer's work. For reliability, the developer must provide the environment ID for development and initial testing from the local machine, but the pipeline will handle deploying changes to QA and Prod, as those are common across teams and can be defined universally. + +### Feature Development Walkthrough + +1. To align with a typical developer experience, use the **Feature request** template in Github to create a GitHub Issue in the UI. GitHub Issue templates help ensure the requestor provides appropriate information on the issue. Provide a description and save the issue by clicking the "Submit New Issue" button. -Run the following command to upload **localsecrets** to Github: +![Create a new issue](./img/githubissuerequest.png "Create a new issue") + +2. Select the issue and click "Create a branch" and choose "Checkout Locally" from the right-hand navigation menu. + +![Create a branch](./img/createabranch.png "Create a branch") + +3. Deploy the sample Davinci flow to the development environment by running the following commands. You will be prompted for the development environment ID: ```bash -_secrets="$(base64 -i localsecrets)" -gh secret set --app actions TERRAFORM_ENV_BASE64 --body $_secrets -unset _secrets +source localsecrets +./scripts/local_feature_deploy.sh +``` + +> Note - If you want to see what Terraform will do without actually deploying, provide the `--dry-run` flag to the script. This flag generates the Terraform configuration without applying it by running `terraform plan`. + +4. Confirm the deployment by examining the Davinci flow in the PingOne console in the development environment matching the ID you provided. Click on the Davinci link from the PingOne console to open the DaVinci console and select **Flows** from the left navigation panel. Click on the **PingOne DaVinci Registration Example** flow to view the configuration. + +5. The Terraform configuration also deployed a sample client application in a local docker container that can be used to try out the flow by navigating to [https://127.0.0.1:8080](https://127.0.0.1:8080). You will be presented a simple progressive profiling style form to enter an email address. If the email address is not found, you will be prompted to register the user. + +6. On the next panel, you are told to provide the email and password. There are password rules in place, but you are not informed when prompted. Try using a simple password such as `password`. The form does not indicate there is a problem, but refuses to accept the password and continue. The password must be at least 8 characters long and contain at least one uppercase letter, one lowercase letter, one number, and one special character. + +7. Create a valid password. After registering the user, you will be redirected to login. + +8. To improve the flow, you will add a small prompt on the registration page to indicate that the password must meet the requirements. To do so, select the **Registration Window** node in the Davinci flow editor. Replace the text in the HTML Template editor with the following. The only change in this block from what is provided is the addition of the password requirements notification and some descriptive comments. + +```html +
+

We did not locate that account. Sign up now!

+ + + + + + + + + + Password must be at least 8 characters and contain 1 uppercase, 1 lowercase, 1 digit, and 1 symbol + + + + +
``` -> Note - On a Mac, if you have installed the **base64** application using brew, there will be a file content failure in the pipeline stemming from the first command shown above. Use the default version of base64 by specifying the path explicitly: `_secrets="$(/usr/bin/base64 -i localsecrets)"` +9. Click **Apply** to save the changes, then click **Deploy** to update the flow in the development environment. -### Deploy Prod and QA +10. Next, test your change from the client application. Browse to [https://127.0.0.1:8080](https://127.0.0.1:8080) again and provide a new email address. Notice on the registration page that you are presented the password requirements message. Completing the new user registration is optional, as you can already see the change has been applied in the interface. -The final step before creating new features is to deploy application configuration for `prod` and `qa`. +11. To capture the changes for inclusion in your code, export the flow. You can do so by selecting the three dots at the top right of the DaVinci flow editor UI and clicking **Download Flow JSON**. Ensure to select **Include Variable Values** when you export. -Under the **Actions** section in Github, locate the failed **Initial commit** workflow run from the creation of the repository. Click "Re-run jobs" and choose "Re-run all jobs". If your secrets are configured correctly, this should result in the successful deployment of new configuration to the "prod" environment in your PingOne account. +![Export Menu](./img/pingOneEnvs.png "Export Menu") -![re-run all jobs](./img/rerunalljobs.png "Re-run All Jobs") +12. For the sake of brevity, assume that testing has been done, and you are ready to proceed. After the application is "tested", the new configuration must be added to the Terraform configuration. This addition will happen in a few steps: -![Prod deployed](./img/proddeployed.png "Prod Deployed") + a. Copy the contents of the downloaded JSON file and use them to replace the `terraform/davinci-flows/davinci-widget-reg-authn-flow.json` file contents. If you examine the changes, you will see that it involves the company ID, metadata about the file and the changes you made to the node in the flow. -To deploy the `qa` environment, simply create and push a new branch from prod with the name `qa`: + b. Run the deploy script again: ```bash -git checkout prod -git pull origin prod -git checkout -b qa -git push origin qa +source localsecrets +./scripts/local_feature_deploy.sh ``` -![QA deployed](./img/qadeployed.png "QA Deployed") + c. If you examine the output, you will see that the change to the JSON object caused Terraform to detect a delta between the state stored in S3 (with the old JSON code) and what is present in the new code: + +```bash +Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the +following symbols: + ~ update in-place -## Feature Development -Now that the repository and pipeline are configured, the standard git flow can be followed. To experience the developer's perspective, follow steps similar to those documented within the [pipeline-example-platform "Feature Development"](https://github.com/pingidentity/pipeline-example-platform/tree/prod?tab=readme-ov-file#feature-development) section. A notable difference in this repository is that there will be no deployment of configuration to feature environments from the pipeline. Feature/Development envirionment configuration deployment will occur from local machines only. A "shared" application development environment will be used to connect to for deploying the applications managed in this repository. +Terraform will perform the following actions: + # davinci_flow.registration_flow will be updated in-place + ~ resource "davinci_flow" "registration_flow" { + ~ flow_configuration_json = (sensitive value) + ~ flow_export_json = (sensitive value) + ~ flow_json = (sensitive value) + id = "a6d551f1d7aa2612f2bf6c371b0026e1" + name = "PingOne DaVinci Registration Example" + # (4 unchanged attributes hidden) -To experience the developer's perspective, a demo walk through of the steps follows. The demo will simulated the use case of updating the client application and promoting the change into production. + # (3 unchanged blocks hidden) + } -1. In the **pipeline-example-platform** repository, create a GitHub Issue for a new feature request via the UI. Name the issue "application-development". This GitHub issue name will be used to create the PingOne environment. +Plan: 0 to add, 1 to change, 0 to destroy. +``` -![Create a new issue](./img/githubissuerequestapp.png "Create a new issue") + d. Applying the change will update and 'redeploy' the flow (even though it is exactly the same as what is there, the update to the file creates a delta for Terraform to resolve) and afterward, the state will reflect the new configuration. Type `yes` to apply the changes. -2. Click "Create a branch" and choose "Checkout Locally" for GitHub to create a development branch and PingOne environment on your behalf. + e. Run the script again to confirm that the state matches the configuration: -![Create a branch](./img/createabranch.png "Create a branch") +```bash +No changes. Your infrastructure matches the configuration. +Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are +needed. -3. The end of the Github Actions deploy step will output the environment id, capture it for use in this repository. +Apply complete! Resources: 0 added, 0 changed, 0 destroyed. +``` + +13. If you want to go one step further, you can modify the HTML code that is placed in the nginx image. To do so, you can modify `./terraform/sample-app/index.html` in some manner. When the script is ran again, it will detect the change to the file, rebuild the Docker image, and launch a replacement container for the UI. +14. Before committing and pushing the changes, run a devcheck against the code to ensure the formatting and syntax are correct, ignoring any warnings or informational messages: + +```bash +make devcheck ``` -Apply complete! Resources: 15 added, 0 changed, 0 destroyed. -Outputs: +15. Commit and push the changes to the repository: -pingone_environment_id = "b1de511a-f8af-4c91-93da-79c212d7ed14" +```bash +git add . +git commit -m "Adding password requirements to registration page" +git push ``` -4. In this **pipeline-example-application** repository, create a GitHub Issue for a new feature request via the UI. GitHub Issue Templates help ensure the requestor provides appropriate information on the issue. +16. The push will fire a pipeline that runs the same checks as you did locally. However, since this is a development branch, Terraform deployment will be skipped. + +17. Create a pull request in the repository from your branch to `qa`. Creation of the pull request will trigger a pipeline of checks and allow a "reviewer" to validate the changes. Merge the pull request to trigger the deployment workflow against the **qa** environment in your PingOne account. You may choose to confirm the flow exists in your **qa** environment and has your change. -5. Click "Create a branch" and choose "Checkout Locally" for GitHub to create a development branch and PingOne environment on your behalf. `cd` into the local root of this repository and enter the steps presented from "Checkout Locally". +18. Finally, you can create a pull request from `qa` to `prod`. Follow the same review process as for qa. Upon merge of the pull request, the pipeline will validate the changes and deploy the flow to the **prod** environment in your PingOne account. -6. Run the deploy command to initialize the application +## Cleanup + +When you are finished with the demonstration, you can clean up the resources from the PingOne `dev` environment by running the following commands. The `--destroy` flag will destroy the resources in the development environment: ```bash -./scripts/local_feature_deploy.sh +source localsecrets +./scripts/local_feature_deploy.sh --destroy ``` -Review the planned changes to see what will be deployed then enter "yes" to accept the plan. +After the development resources are destroyed: -7. There should now be an application running at http://localhost:8080. It will include a simple login screen, test the connection to PingOne by submitting an email and password. If the deployment is successful, a page will be returned showing that the user has been created in PingOne. +- delete the branch from the local and remote repositories +- resolve the issue in the Github UI +- delete the feature folder from the `application-state/dev/` in the S3 bucket -8. Next, edit the sample application code. A simple edit can be represented is updating the `

` block with text `Welcome!` within `terraform/sample-app/index.html`. +## Conclusion -9. \ No newline at end of file +This repository demonstrates a simplified example of how a pipeline can be used to manage the development and deployment of an application that relies on services managed by a central IAM platform team. The pipeline is designed to be flexible and extensible, allowing for the addition of new features and services as needed. By following the steps outlined in this repository, you can gain a better understanding of how to implement a similar pipeline in your own environment. diff --git a/img/githubissuerequest.png b/img/githubissuerequest.png new file mode 100644 index 0000000..f069575 Binary files /dev/null and b/img/githubissuerequest.png differ diff --git a/img/githubissuerequestapp.png b/img/githubissuerequestapp.png deleted file mode 100644 index ea4577b..0000000 Binary files a/img/githubissuerequestapp.png and /dev/null differ diff --git a/scripts/local_feature_deploy.sh b/scripts/local_feature_deploy.sh index cea867e..570d6ff 100755 --- a/scripts/local_feature_deploy.sh +++ b/scripts/local_feature_deploy.sh @@ -12,14 +12,16 @@ usage () { cat <