Passwords not set up correctly

[chrislovecnm]

Summary

When using vault the passwords where not set correctly and the git operator is not starting.

Steps to reproduce the behavior

Install via terraform and use vault.

My main.tf

module "eks-jx" {
  source = "jenkins-x/eks-jx/aws"
  region       = var.region
  use_vault    = var.use_vault
  use_asm      = var.use_asm
  cluster_name = var.cluster_name
  is_jx2       = var.is_jx2
  create_eks   = var.create_eks
  create_vpc   = var.create_vpc
  create_nginx = var.create_nginx
  jx_git_url   = var.jx_git_url
  apex_domain  = var.apex_domain
  subdomain    = var.subdomain
  tls_email    = var.tls_email
  use_kms_s3   = var.use_kms_s3
  registry     = var.registry

  nginx_chart_version = var.nginx_chart_version
  cluster_version     = var.cluster_version
  enable_backup       = var.enable_backup
  jx_bot_username     = var.jx_bot_username
  jx_bot_token        = var.jx_bot_token
  enable_external_dns = var.enable_external_dns

  jx_git_operator_values = var.jx_git_operator_values
  production_letsencrypt = var.production_letsencrypt

  create_and_configure_subdomain = var.create_and_configure_subdomain
}

Expected behavior

The system is set up to use vault

Actual behavior

jx secret verify returns this

jx-production/tekton-container-registry-auth key secret/data/tekton/container/registry/auth missing properties: .dockerconfigjson
jx-staging/tekton-container-registry-auth    key secret/data/tekton/container/registry/auth missing properties: .dockerconfigjson
jx/jenkins-maven-settings                    key secret/data/jx/mavenSettings missing properties: settingsXml, securityXml
jx/jenkins-x-chartmuseum                     key secret/data/jx/adminUser missing properties: password, username
jx/jx-basic-auth-htpasswd                    key secret/data/jx/basic/auth/htpasswd missing properties: auth
jx/jx-basic-auth-user-password               key secret/data/jx/basic/auth/user missing properties: password
jx/jx-basic-auth-user-password               key secret/data/jx/basic/auth/user/password missing properties: username
jx/lighthouse-hmac-token                     key secret/data/lighthouse/hmac missing properties: token
jx/lighthouse-oauth-token                    key secret/data/lighthouse/oauth missing properties: token
jx/nexus                                     key secret/data/nexus missing properties: password
jx/tekton-container-registry-auth            key secret/data/tekton/container/registry/auth missing properties: .dockerconfigjson
jx/tekton-git                                key secret/data/jx/pipelineUser missing properties: token, username

The majority of the secrets are in vault. I checked.

$ k -n jx-git-operator get po
NAME                                                 READY   STATUS    RESTARTS   AGE
jx-boot-db5ff8d2-8bff-4771-9734-daa9b248c475-7fs7z   0/1     Error     0          20h
jx-boot-db5ff8d2-8bff-4771-9734-daa9b248c475-995lv   0/1     Error     0          20h
jx-boot-db5ff8d2-8bff-4771-9734-daa9b248c475-c7vgn   0/1     Error     0          20h
jx-boot-db5ff8d2-8bff-4771-9734-daa9b248c475-hfmzr   0/1     Error     0          20h
jx-boot-db5ff8d2-8bff-4771-9734-daa9b248c475-t982z   0/1     Error     0          20h
jx-git-operator-7bc44fc4c-26bm7                      1/1     Running   1          20h

I found logs somewhere to run jx devops update. Which I am getting working on the m1.

Terraform version

The output of terraform version is:

See other bugs I have filed.

Module version

Current release

Operating system

Linux in container

[ankitm123]

Since the boot job failed, the secret generation never worked I think. Once the boot jobs works, try this guide: https://jenkins-x.io/v3/admin/troubleshooting/install/#issues-with-secret-generation

[chrislovecnm]

jx secret populate

Did not work previously. But I will try it again. Just got jx gitops update to work on the m1. I am getting:

jenkins-x-chartmuseum: key secret/data/jx/adminUser missing properties: password, username
jx-basic-auth-user-password: key secret/data/jx/basic/auth/user missing properties: password, key secret/data/jx/basic/auth/user/password missing properties: username
lighthouse-hmac-token: key secret/data/lighthouse/hmac missing properties: token
lighthouse-oauth-token: key secret/data/lighthouse/oauth missing properties: token
nexus: key secret/data/nexus missing properties: password
tekton-container-registry-auth: key secret/data/tekton/container/registry/auth missing properties: .dockerconfigjson
tekton-git: key secret/data/jx/pipelineUser missing properties: token, username

In the operator logs.

Hopefully the secrets command will work.

[ankitm123]

You dont need to run jx secret populate manually, the boot job runs it for you. If you do a direct push to master, it will run the step where it re-generates the secrets. But first you need to fix the boot job (unless that was fixed) 🤔

[chrislovecnm]

VAULT_ADDR=https://vault.jx-vault:8200 VAULT_NAMESPACE=jx-vault EXTERNAL_VAULT=false jx secret populate --secret-namespace jx-vault

Ran so let me read the docs you gave me and see if they help.

[chrislovecnm]

Yah the docs did not help. How are the secrets set in vault? Because that evidently failed during the terraform setup.

[ankitm123]

Did a direct push to master fail? Can you post the logs from the bootjob?

[chrislovecnm]

Can we chat on slack?

[chrislovecnm]

@ankitm123 I am @chrislovecnm on k8s slack. I have meetings. Ping me and I will respond when I can

[ankitm123]

Join the #jenkins-x-dev channel in the k8s slack, that's where most of the devs are. My slack handle is @ankit

[chrislovecnm]

Closing this. This was an issue with vault, and most likely an older version of k8s.

[cnukwas]

@chrislovecnm and @ankitm123 posting here since we're running into a similar issue. I am new to Jenkins-x and having trouble with the installation on EKS using Secrets Manager with Terraform installation approach. Seeing these errors. I don't see any references in the document where and how these passwords need to be set up. Any help is really appreciated.
I did make a dummy commit to our internal bitbucket cluster repo(jx3-eks-asm) and still seeing same errors.

jx secret verify
SECRET                                       STATUS
jx-production/tekton-container-registry-auth key tekton-container-registry-auth missing properties: .dockerconfigjson
jx-staging/tekton-container-registry-auth    key tekton-container-registry-auth missing properties: .dockerconfigjson
jx/jenkins-maven-settings                    key jx-maven-settings missing properties: settingsXml, securityXml
jx/jenkins-x-chartmuseum                     valid: jx-admin-user/BASIC_AUTH_PASS, jx-admin-user/BASIC_AUTH_USER
jx/jx-basic-auth-htpasswd                    key jx-basic-auth-htpasswd missing properties: token
jx/jx-basic-auth-user-password               valid: jx-basic-auth-user/password, jx-basic-auth-user/username
jx/lighthouse-oauth-token                    key lighthouse-oauth missing properties: token
jx/nexus                                     valid: jx-admin-user/password
jx/tekton-container-registry-auth            key tekton-container-registry-auth missing properties: .dockerconfigjson
jx/tekton-git                                key jx-pipeline-user missing properties: token, username