ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed

[jaruji]

Official Helm Chart version

1.12.0 (latest released)

Apache Airflow version

2.8.1

Kubernetes Version

1.26.6

Helm Chart configuration

config:
    webserver:
      expose_config: 'True'
    logging:
      remote_logging: 'True'
      remote_base_log_folder: wasb-airflow/logs
      remote_wasb_log_container: airflow
      remote_log_conn_id: wasb_default
  images:
    airflow:
      # define custom airflow image here (with PyPi packages installed)
      repository: org.azurecr.io/internal-airflow
      # CHANGE THIS when updating
      tag: "0.2.0"

  executor: KubernetesExecutor
  fernetKeySecretName: airflow-fernet-secret
  webserverSecretKeySecretName: airflow-webserver-secret
  createUserJob:
    useHelmHooks: false
    applyCustomEnv: false

  migrateDatabaseJob:
    enabled: true  
    useHelmHooks: false
    applyCustomEnv: false
    jobAnnotations:
        "argocd.argoproj.io/hook": Sync
  useStandardNaming: true

  dags:
    gitSync:
      enabled: true
      repo: [email protected]:ORG/custom-airflow.git
      branch: master
      subPath: "dags"
      sshKeySecret: airflow-ssh-secret
  ingress:
    web:
      enabled: true
      annotations:
        cert-manager.io/cluster-issuer: "letsencrypt"

      # The path for the web Ingress
      path: "/"

      # The pathType for the above path (used only with Kubernetes v1.19 and above)
      pathType: "ImplementationSpecific"

      # The hostnames or hosts configuration for the web Ingress
      # Set in argoCD application yaml
      hosts: []
      #   # The hostname for the web Ingress (can be templated)
      # - name: ""
      #   # configs for web Ingress TLS
      #   tls:
      #     # Enable TLS termination for the web Ingress
      #     enabled: false
      #     # the name of a pre-created Secret containing a TLS private key and certificate
      #     secretName: ""

      # The Ingress Class for the web Ingress (used only with Kubernetes v1.19 and above)
      ingressClassName: "nginx"

Docker Image customizations

#### Use the specified Apache Airflow image as a base
FROM apache/airflow:2.8.1

##### Install dependencies required for building pymssql
USER root
RUN apt-get update && apt-get install -y \
    freetds-dev \
    build-essential \
    && rm -rf /var/lib/apt/lists/*
RUN apt-get update -y \
    && apt-get install -y \
    libglib2.0-0 \
    libnss3 \
    libnspr4 \
    libdbus-1-3 \
    libatk1.0-0 \
    libatk-bridge2.0-0 \
    libcups2 \
    libdrm2 \
    libxkbcommon0 \
    libatspi2.0-0 \
    libxcomposite1 \
    libxdamage1 \
    libxext6 \
    libxfixes3 \
    libxrandr2 \
    libgbm1 \
    libpango-1.0-0 \
    libcairo2 \
    libasound2 \
    && rm -rf /var/lib/apt/lists/*  
#### Copy the requirements file into the container
COPY requirements.txt /
COPY .env /

#### Switch back to the airflow user
USER airflow

#### Install the requirements, including Apache Airflow
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt
RUN pip install python-dotenv
#install azure provider for airflow, needed for remote logging to azure blob
RUN pip install apache-airflow-providers-microsoft-azure 
RUN playwright install

What happened

When deploying the chart using ArgoCD, everything seems to more or less work correctly. But after a while, the argoCD app reverts to OutOfSync and Missing statuses. This persists until the chart is re-synced. When the sync if successfully finished, everything displays correctly - synced, etc... but it always reverts back to OutOfSync and the health status goes to Missing, which can be confusing. I have not found out what the cause of this is yet.

What you think should happen instead

After the successful helm chart deploy through ArgoCD, correctly display that the app is Healthy and Synced.

How to reproduce

I assume using the provided configurations to deploy the chart to a kubernetes cluster should be enough to reproduce - you also need to add the argoCD app config for airflow into your argoCD apps.
The argocd airflow app config:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: airflow
  namespace: {{ .Values.argocdNamespace }}
  finalizers:
    - resources-finalizer.argocd.argoproj.io  
spec:
  project: {{ .Values.projects.infra }}
  source:
    repoURL: {{ .Values.repoUrl }}
    targetRevision: {{ .Values.targetRevision }}
    path: infrastructure/airflow
    helm:
      valuesObject:
          ingress:
            web:
              hosts: 
                - name: "airflow.{{ .Values.ingress.host }}"
                  tls:
                    enabled: true
                    secretName: "airflow-ingress-secret"
  destination:
    server: https://kubernetes.default.svc
    namespace: "airflow"
  syncPolicy:
    syncOptions:
        - CreateNamespace=true

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Have you opened the issue to ArgoCD? What do they say ? It could be caused by some issues that we already know and plan to release in 1.12.1 #37399 or #37356 - can you please check with Argo CD support?

[jaruji]

I can open an issue there tomorrow. I did some more research, and enabled automated prune and selfHeal in the argoCD app like so:

 syncPolicy:
    automated:
        prune: true
        selfHeal: true

The argo airflow app is now re-syncing every couple of minutes to keep the Healthy and Synced status. Specifically, the run-airflow-migrations and create-user jobs are executing every couple of minutes in this case. If the automated selfHeal is turned off, the app just shifts to Missing and OutOfSync after a couple of minutes - but it keeps working just fine. Just wanted to give a better description of what is happening!

[potiuk]

Specifically, the run-airflow-migrations and create-user jobs are executing every couple of minutes in this case. If the automated selfHeal is turned off, the app just shifts to Missing and OutOfSync after a couple of minute but it keeps working just fine. Just wanted to give a better description of what is happening!

Just bear with us that for most people here who do not use ArgoCD - none of this - syncing, out-of-syncing, prune. self-healing has any significant meaning, and for people who voluntarily maintain Ariflow, they won't spend awfully lot of time digging in the details of Argo.

For us we deliver Helm Chart - which is a K8S Helm Chart and we generally deploy and test it in plain K8S. If there is anything that Argo Does which is specific to Argo, it's pretty much up to you to see what's happening and translate it into "what really happens".

By looking at logs etc. and trying to figure out thibgs - while Argo is popular, it's still pretty much 3rd-party technology that we have little to none expert knowedge there and we have to rely on our users - who are part of the community and actually have the skills and incentive to investigate things like that. If you can diagnose it and translate it to what happens in the "non-argo" terms but - generally what happens with resources, memory, networking and other things - then we can try to see how we can help with that.

Maybe other Argo users might chime in and provide some input - but from maintainers point of view it's like that .

Just to set some expectations here.

[jaruji]

I'm mainly opening the issue to see whether somebody else encountered something similar and managed to fix it. I do not have any expectations regarding you doing the necessary research and figuring it out for me. Guess this can be converted into a discussion if you feel like an issue is not appropriate in this case.

[potiuk]

Yep I think discussion is better.

[potiuk]

Also - what I could recommend maybe - try to install the Chart from main. The two issues mentioned above are already fixed in main and we will soon release 1.12.1 - but maybe it's one of the two that causes the behaviour you observe - this is by far the fastest way to asses is - both on our and your side

[ldacey]

Did you resolve this by chance? I noticed that my Airflow app constantly just runs the create user job as well (also deployed with argocd and chart 1.13.1)

[redisded]

Hi everybody, we had the same problem today, that we were able to solve adding moving the createUserJob definition from :

createUserJob:
        useHelmHooks: false
        applyCustomEnv: false

to :

createUserJob:
    useHelmHooks: false
    applyCustomEnv: false
    jobAnnotations:
        "argocd.argoproj.io/hook": Sync