refactor(providers/databricks): move databricks provider to new structure

Author: josix

.github/boring-cyborg.yml

@@ -135,10 +135,7 @@ labelPRBasedOnFilePath:
     - providers/standard/**
 
   provider:databricks:
-    - providers/src/airflow/providers/databricks/**/*
-    - docs/apache-airflow-providers-databricks/**/*
-    - providers/tests/databricks/**/*
-    - providers/tests/system/databricks/**/*
+    - providers/databricks/**
 
   provider:datadog:
     - providers/datadog/**

dev/moving_providers/move_providers.py

@@ -362,10 +362,13 @@ def move_provider_yaml(provider_id: str) -> tuple[list[str], list[str], list[str
     dependencies = []
     optional_dependencies = []
     devel_dependencies = []
+    copied_logo = set()
     for line in original_content:
         if line.startswith("    logo: "):
             logo_path = line[len("    logo: ") :]
             logo_name = logo_path.split("/")[-1]
+            if logo_path in copied_logo:
+                continue
             new_logo_dir = (
                 PROVIDERS_DIR_PATH / _get_provider_only_path(provider_id) / "docs" / "integration-logos"
             )
@@ -378,6 +381,7 @@ def move_provider_yaml(provider_id: str) -> tuple[list[str], list[str], list[str
                 remove_empty_parent_dir=True,
             )
             line = f"    logo: /docs/integration-logos/{logo_name}"
+            copied_logo.add(logo_path)
         if line == "dependencies:" and not in_dependencies:
             in_dependencies = True
             continue

docs/.gitignore

@@ -18,6 +18,7 @@ apache-airflow-providers-cohere
 apache-airflow-providers-common-compat
 apache-airflow-providers-common-io
 apache-airflow-providers-common-sql
+apache-airflow-providers-databricks
 apache-airflow-providers-datadog
 apache-airflow-providers-discord
 apache-airflow-providers-docker

docs/apache-airflow-providers-databricks/changelog.rst

@@ -0,0 +1,87 @@
+
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+ .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+ .. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+    `PROVIDER_README_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+
+
+Package ``apache-airflow-providers-databricks``
+
+Release: ``7.0.0``
+
+
+`Databricks <https://databricks.com/>`__
+
+
+Provider package
+----------------
+
+This is a provider package for ``databricks`` provider. All classes for this provider package
+are in ``airflow.providers.databricks`` python package.
+
+You can find package information and changelog for the provider
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-databricks/7.0.0/>`_.
+
+Installation
+------------
+
+You can install this package on top of an existing Airflow 2 installation (see ``Requirements`` below
+for the minimum Airflow version supported) via
+``pip install apache-airflow-providers-databricks``
+
+The package supports the following python versions: 3.9,3.10,3.11,3.12
+
+Requirements
+------------
+
+=======================================  ==================
+PIP package                              Version required
+=======================================  ==================
+``apache-airflow``                       ``>=2.9.0``
+``apache-airflow-providers-common-sql``  ``>=1.20.0``
+``requests``                             ``>=2.27.0,<3``
+``databricks-sql-connector``             ``>=3.0.0``
+``aiohttp``                              ``>=3.9.2,<4``
+``mergedeep``                            ``>=1.3.4``
+``pandas``                               ``>=2.1.2,<2.2``
+``pyarrow``                              ``>=14.0.1``
+=======================================  ==================
+
+Cross provider package dependencies
+-----------------------------------
+
+Those are dependencies that might be needed in order to use all the features of the package.
+You need to install the specified provider packages in order to use them.
+
+You can install such cross-provider dependencies when installing from PyPI. For example:
+
+.. code-block:: bash
+
+    pip install apache-airflow-providers-databricks[common.sql]
+
+
+============================================================================================================  ==============
+Dependent package                                                                                             Extra
+============================================================================================================  ==============
+`apache-airflow-providers-common-sql <https://airflow.apache.org/docs/apache-airflow-providers-common-sql>`_  ``common.sql``
+============================================================================================================  ==============
+
+The changelog for the provider package can be found in the
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-databricks/7.0.0/changelog.html>`_.

providers/databricks/README.rst

@@ -139,7 +139,7 @@ Features
 Misc
 ~~~~
 
-* ``Removed deprecated method referance airflow.www.auth.has_access when min airflow version >= 2.8.0 (#41747)``
+* ``Removed deprecated method reference airflow.www.auth.has_access when min airflow version >= 2.8.0 (#41747)``
 * ``remove deprecated soft_fail from providers (#41710)``
 
 6.9.0
@@ -451,7 +451,7 @@ Misc
 Features
 ~~~~~~~~
 
-* ``Add "QUEUED" to RUN_LIFE_CYCLE_STATES following deployement of … (#33886)``
+* ``Add "QUEUED" to RUN_LIFE_CYCLE_STATES following deployment of … (#33886)``
 * ``allow DatabricksSubmitRunOperator to accept a pipeline name for a pipeline_task (#32903)``
 
 Misc