Move CNCF Kubernetes to new provider structure (apache#46436)

* Move CNCF Kubernetes to new provider structure

* Fix doc include path and k8s test

* Fix taskflow tutorial

* Fix test_project_structure

* Strip src. prefix instead of replacing all src.

Co-authored-by: Kalyan R <[email protected]>

* Merge fix get_classes_from_file apache#46454

* Fix TestCncfProviderProjectStructure
- rename PROVIDER from "cncf" to "cncf/kubernetes"
- remove MISSING_EXAMPLES_FOR_CLASSES

* Fix k8s CI requirements

* fixup! Fix k8s CI requirements

---------

Co-authored-by: Kalyan R <[email protected]>
Co-authored-by: Jarek Potiuk <[email protected]>

Author: 3 people

.github/boring-cyborg.yml

@@ -102,14 +102,7 @@ labelPRBasedOnFilePath:
     - providers/cloudant/**
 
   provider:cncf-kubernetes:
-    - airflow/example_dags/example_kubernetes_executor.py
-    - airflow/example_dags/example_local_kubernetes_executor.py
-    - providers/src/airflow/providers/cncf/kubernetes/**/*
-    - providers/celery/src/airflow/providers/celery/executors/celery_kubernetes_executor.py
-    - docs/apache-airflow-providers-cncf-kubernetes/**/*
-    - kubernetes_tests/**/*
-    - providers/tests/cncf/kubernetes/**/*
-    - providers/tests/system/cncf/kubernetes/**/*
+    - providers/cncf/kubernetes/**
 
   provider:cohere:
     - providers/cohere/**

.pre-commit-config.yaml

@@ -665,7 +665,7 @@ repos:
           ^providers/fab/docs/auth-manager/webserver-authentication.rst$|
           ^providers/google/docs/operators/cloud/kubernetes_engine.rst$|
           ^docs/apache-airflow-providers-microsoft-azure/connections/azure_cosmos.rst$|
-          ^docs/apache-airflow-providers-cncf-kubernetes/operators.rst$|
+          ^providers/cncf/kubernetes/docs/operators.rst$|
           ^docs/conf.py$|
           ^docs/exts/removemarktransform.py$|
           ^newsfragments/41761.significant.rst$|

dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py

@@ -601,7 +601,7 @@ def _rebuild_k8s_image(
 
 COPY --chown=airflow:0 airflow/example_dags/ /opt/airflow/dags/
 
-COPY --chown=airflow:0 providers/src/airflow/providers/cncf/kubernetes/kubernetes_executor_templates/ /opt/airflow/pod_templates/
+COPY --chown=airflow:0 providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/kubernetes_executor_templates/ /opt/airflow/pod_templates/
 
 ENV GUNICORN_CMD_ARGS='--preload' AIRFLOW__WEBSERVER__WORKER_REFRESH_INTERVAL=0
 """

docs/.gitignore

@@ -19,6 +19,7 @@ apache-airflow-providers-asana
 apache-airflow-providers-atlassian-jira
 apache-airflow-providers-celery
 apache-airflow-providers-cloudant
+apache-airflow-providers-cncf-kubernetes
 apache-airflow-providers-cohere
 apache-airflow-providers-common-compat
 apache-airflow-providers-common-io

docs/apache-airflow-providers-cncf-kubernetes/changelog.rst

@@ -338,7 +338,7 @@ Below is an example of using the ``@task.kubernetes`` decorator to run a Python
 
 .. _taskflow/kubernetes_example:
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes_decorator.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_decorator.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_kubernetes]

docs/apache-airflow/tutorial/taskflow.rst

@@ -0,0 +1,67 @@
+
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+ .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+ .. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+    `PROVIDER_README_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+
+
+Package ``apache-airflow-providers-cncf-kubernetes``
+
+Release: ``10.1.0``
+
+
+`Kubernetes <https://kubernetes.io/>`__
+
+
+Provider package
+----------------
+
+This is a provider package for ``cncf.kubernetes`` provider. All classes for this provider package
+are in ``airflow.providers.cncf.kubernetes`` python package.
+
+You can find package information and changelog for the provider
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.1.0/>`_.
+
+Installation
+------------
+
+You can install this package on top of an existing Airflow 2 installation (see ``Requirements`` below
+for the minimum Airflow version supported) via
+``pip install apache-airflow-providers-cncf-kubernetes``
+
+The package supports the following python versions: 3.9,3.10,3.11,3.12
+
+Requirements
+------------
+
+======================  =====================
+PIP package             Version required
+======================  =====================
+``aiofiles``            ``>=23.2.0``
+``apache-airflow``      ``>=2.9.0``
+``asgiref``             ``>=3.5.2``
+``cryptography``        ``>=41.0.0``
+``kubernetes``          ``>=29.0.0,<=31.0.0``
+``kubernetes_asyncio``  ``>=29.0.0,<=31.0.0``
+``google-re2``          ``>=1.0``
+======================  =====================
+
+The changelog for the provider package can be found in the
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.1.0/changelog.html>`_.

providers/cncf/kubernetes/README.rst

@@ -108,21 +108,21 @@ With these requirements in mind, here are some examples of basic ``pod_template_
 
 Storing DAGs in the image:
 
-.. literalinclude:: /../../providers/src/airflow/providers/cncf/kubernetes/pod_template_file_examples/dags_in_image_template.yaml
+.. literalinclude:: /../../providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/pod_template_file_examples/dags_in_image_template.yaml
     :language: yaml
     :start-after: [START template_with_dags_in_image]
     :end-before: [END template_with_dags_in_image]
 
 Storing DAGs in a ``persistentVolume``:
 
-.. literalinclude:: /../../providers/src/airflow/providers/cncf/kubernetes/pod_template_file_examples/dags_in_volume_template.yaml
+.. literalinclude:: /../../providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/pod_template_file_examples/dags_in_volume_template.yaml
     :language: yaml
     :start-after: [START template_with_dags_in_volume]
     :end-before: [END template_with_dags_in_volume]
 
 Pulling DAGs from ``git``:
 
-.. literalinclude:: /../../providers/src/airflow/providers/cncf/kubernetes/pod_template_file_examples/git_sync_template.yaml
+.. literalinclude:: /../../providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/pod_template_file_examples/git_sync_template.yaml
     :language: yaml
     :start-after:  [START git_sync_template]
     :end-before: [END git_sync_template]

providers/src/airflow/providers/cncf/kubernetes/.latest-doc-only-change.txt providers/cncf/kubernetes/docs/.latest-doc-only-change.txt

@@ -102,7 +102,7 @@ Using this method will ensure correctness
 and type safety. While we have removed almost all Kubernetes convenience classes, we have kept the
 :class:`~airflow.providers.cncf.kubernetes.secret.Secret` class to simplify the process of generating secret volumes/env variables.
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes.py
     :language: python
     :start-after: [START howto_operator_k8s_cluster_resources]
     :end-before: [END howto_operator_k8s_cluster_resources]
@@ -135,21 +135,21 @@ Create the Secret using ``kubectl``:
 
 Then use it in your pod like so:
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes.py
     :language: python
     :start-after: [START howto_operator_k8s_private_image]
     :end-before: [END howto_operator_k8s_private_image]
 
 Also for this action you can use operator in the deferrable mode:
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes_async.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_async.py
     :language: python
     :start-after: [START howto_operator_k8s_private_image_async]
     :end-before: [END howto_operator_k8s_private_image_async]
 
 Example to fetch and display container log periodically
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes_async.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_async.py
     :language: python
     :start-after: [START howto_operator_async_log]
     :end-before: [END howto_operator_async_log]
@@ -168,7 +168,7 @@ alongside the Pod. The Pod must write the XCom value into this location at the `
 
 See the following example on how this occurs:
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes.py
     :language: python
     :start-after: [START howto_operator_k8s_write_xcom]
     :end-before: [END howto_operator_k8s_write_xcom]
@@ -177,7 +177,7 @@ See the following example on how this occurs:
 
 Also for this action you can use operator in the deferrable mode:
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes_async.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_async.py
     :language: python
     :start-after: [START howto_operator_k8s_write_xcom_async]
     :end-before: [END howto_operator_k8s_write_xcom_async]
@@ -621,15 +621,15 @@ request that dynamically launches this Job.
 Users can specify a kubeconfig file using the ``config_file`` parameter, otherwise the operator will default
 to ``~/.kube/config``. It also allows users to supply a template YAML file using the ``job_template_file`` parameter.
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes_job.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_job.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_k8s_job]
     :end-before: [END howto_operator_k8s_job]
 
 The :class:`~airflow.providers.cncf.kubernetes.operators.job.KubernetesJobOperator` also supports deferrable mode:
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes_job.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_job.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_k8s_job_deferrable]
@@ -656,7 +656,7 @@ KubernetesDeleteJobOperator
 The :class:`~airflow.providers.cncf.kubernetes.operators.job.KubernetesDeleteJobOperator` allows
 you to delete Jobs on a Kubernetes cluster.
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes_job.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_job.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_delete_k8s_job]
@@ -671,7 +671,7 @@ KubernetesPatchJobOperator
 The :class:`~airflow.providers.cncf.kubernetes.operators.job.KubernetesPatchJobOperator` allows
 you to update Jobs on a Kubernetes cluster.
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes_job.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_job.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_update_job]
@@ -686,7 +686,7 @@ KubernetesInstallKueueOperator
 The :class:`~airflow.providers.cncf.kubernetes.operators.kueue.KubernetesInstallKueueOperator` allows
 you to install the Kueue component in a Kubernetes cluster
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes_kueue.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_kueue.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_k8s_kueue_install]
@@ -709,7 +709,7 @@ KubernetesStartKueueJobOperator
 The :class:`~airflow.providers.cncf.kubernetes.operators.kueue.KubernetesStartKueueJobOperator` allows
 you to start a Kueue job in a Kubernetes cluster
 
-.. exampleinclude:: /../../providers/tests/system/cncf/kubernetes/example_kubernetes_kueue.py
+.. exampleinclude:: /../../providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_kueue.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_k8s_install_kueue]

providers/src/airflow/providers/cncf/kubernetes/CHANGELOG.rst providers/cncf/kubernetes/docs/changelog.rst

@@ -95,38 +95,16 @@ versions:
   - 1.0.1
   - 1.0.0
 
-dependencies:
-  - aiofiles>=23.2.0
-  - apache-airflow>=2.9.0
-  - asgiref>=3.5.2
-  - cryptography>=41.0.0
-  # The Kubernetes API is known to introduce problems when upgraded to a MAJOR version. Airflow Core
-  # Uses Kubernetes for Kubernetes executor, and we also know that Kubernetes Python client follows SemVer
-  # (https://github.com/kubernetes-client/python#compatibility). This is a crucial component of Airflow
-  # So we should limit it to the next MAJOR version and only deliberately bump the version when we
-  # tested it, and we know it can be bumped. Bumping this version should also be connected with
-  # limiting minimum airflow version supported in cncf.kubernetes provider, due to the
-  # potential breaking changes in Airflow Core as well (kubernetes is added as extra, so Airflow
-  # core is not hard-limited via install-requirements, only by extra).
-  - kubernetes>=29.0.0,<=31.0.0
-  # The Kubernetes_asyncio package is used for providing Asynchronous (AsyncIO) client library for
-  # standard Kubernetes API. The version is limited by minimum 18.20.1 because of introducing the ability to
-  # load kubernetes config file from dictionary in that release and is limited to the next MAJOR version
-  # (started from current 24.2.2 version) to prevent introducing some problems that could be due to some
-  #  major changes in the package.
-  - kubernetes_asyncio>=29.0.0,<=31.0.0
-  - google-re2>=1.0
-
 integrations:
   - integration-name: Kubernetes
     external-doc-url: https://kubernetes.io/
     how-to-guide:
       - /docs/apache-airflow-providers-cncf-kubernetes/operators.rst
-    logo: /integration-logos/kubernetes/Kubernetes.png
+    logo: /docs/integration-logos/Kubernetes.png
     tags: [software]
   - integration-name: Spark on Kubernetes
     external-doc-url: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
-    logo: /integration-logos/kubernetes/Spark-On-Kubernetes.png
+    logo: /docs/integration-logos/Spark-On-Kubernetes.png
     tags: [software]
 
 operators: