Problem with DolphinDBOperator pre-commit check

[peeyee]

I'm trying to develop an provider for dolphindb database. When setting up pre-commit checks, comes with the Problem:

1. pre-commit

Found 1 errors in providers Error: The airflow.providers.dolphindb.hooks.dolphindb object in hooks list in airflow/providers/dolphindb/provider.yaml does not exist or is not a module: No module named 'pydolphindb'

2. provider.yaml :

---
package-name: apache-airflow-providers-dolphindb
name: DolphinDB
description: |
    `DolphinDB <https://www.dolphindb.com/>`__

dependencies:
  - apache-airflow-providers-common-sql>=1.3.1
  - pydolphindb>=1.0.0

suspended: false
versions:
  - 1.0.0


integrations:
  - integration-name: DolphinDB
    external-doc-url: https://www.dolphindb.com/help/index.html
    how-to-guide:
      - /docs/apache-airflow-providers-dolphindb/operators.rst
    logo: /integration-logos/dolphindb/DolphinDB.png
    tags: [software]

operators:
  - integration-name: DolphinDB
    python-modules:
      - airflow.providers.dolphindb.operators.dolphindb

hooks:
  - integration-name: DolphinDB
    python-modules:
      - airflow.providers.dolphindb.hooks.dolphindb

connection-types:
  - hook-class-name: airflow.providers.dolphindb.hooks.dolphindb.DolphinDBHook
    connection-type: dolphindb

[potiuk]

You need to run breeze ci-image build --upgrade-to-newer-dependencies if you want pre-commits to use new dedpendencies.

However - I have a word of warning - your job by trying to build provider might by in vein. We are extremely cautious with accepting new providers to be "community" managed and by default our approach is that if the service has a team that manages their services and if the provider is going to be connecting to the services, then far better approach is that the service releases and maintains their own provider following https://airflow.apache.org/docs/apache-airflow-providers/howto/create-custom-providers.html#how-to-create-a-provider

There are very clear guidelines on that https://github.com/apache/airflow/blob/main/PROVIDERS.rst#accepting-new-community-providers and it requires reaching consensus and passing vote or at least lazy consensus at the devlist in order to accept new provider to be managed by us in the community.,

You might see similar discussions in the past:

• Proposal to add Huawei provider https://lists.apache.org/thread/f5tk9c734wlyv616vyy8r34ymth3dqbc
• Vote (lost) on adding Cloudera provider https://lists.apache.org/thread/8b1jvld3npgzz2z0o3gv14lvtornbdrm
• Discussion on Pandera Provider https://lists.apache.org/thread/qk2co6trd7gm57744shprw2fhgmjr637
• Discussion on Kafka provider: https://lists.apache.org/thread/xf4wn1hw08cs725fp962252y7mmptk3b and following lazy consensus: https://lists.apache.org/thread/y85s9ql68xyszt1ph0ljdz312nr2pvtw
• Lazy consensus on Apprise https://lists.apache.org/thread/2gh845jb8k0nlvzpqdm4pk1t44djhs92

So you migh be doing all the community integration in vein. You need to be aware of that - that we often see contribution of new code to the community as a liabiity - not an asset and we are currently working on having less code, complexity and dependencies in Airlfow - not more. Providers managed by 3rd-parties and not in community - especially when the integration is not a highly requested feature by a lot of our users, is one of the ways we are taking care about it.

Read the docs and see if you want to spend your effort before raising the discussion.

And please, do not raise your arguments here. This is not a place and I am not the only person that you should discuss it with. The devlist is the right place.

If you want to argue why you think DolphinDB provider should be managed by community and not by you, personally or dolphindb - you need to start a devlist discussion. Details on how to subscribe and link to archives can be found in https://airflow.apache.org/community/

[peeyee]

Information about such 3rd-party providers are usually published at the Ecosystem: plugins and providers page of the Airflow website and we encourage the service providers to publish their providers there.

How could I pulish the provider there?

[potiuk]

'Suggest a change on this Page' bottom right will open a PR