Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add Databricks db support #62

Merged
merged 5 commits into from
Sep 1, 2022
Merged

add Databricks db support #62

merged 5 commits into from
Sep 1, 2022

Conversation

leo-schick
Copy link
Member

@leo-schick leo-schick commented Aug 22, 2022

This PR adds support for Databricks including

Open tasks

  • add rtd documentation
  • update Readme.md
  • testing the implementation

@leo-schick
Copy link
Member Author

leo-schick commented Aug 22, 2022

Hi @alexeyegorov,
in case you still use Mara somewhere, are you interested to do a review of this PR? This PR adds Databricks support for Mara. This might be helpful to you when you still have some legacy mara pipelines you want to migrate to Databricks.

Note that reading data from STDIN via mara_db.shell.copy_from_stdin_command is not supported yet. This means the commands ReadFile, ReadSQLite and Copy from mara_pipelines are not supported. But you can do a woraround by writing the data to a data lake and then use a unmanaged table or COPY INTO in databricks.

@leo-schick leo-schick marked this pull request as ready for review August 22, 2022 16:22
@leo-schick leo-schick added this to the Version 4.8.0 milestone Aug 22, 2022
@alexeyegorov
Copy link

Hey Leo!
I am so sorry. I have also missed your other request. Let me get back on that tomorrow or better on Monday!
We are currently migrating to Databricks completely and had a lot of work to do. Mara will be used for some more months still and maybe this MR is really an interesting one. ;)

Copy link

@alexeyegorov alexeyegorov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So basically it looks good. I will forward this again to our team and discuss if we could also use this feature as we are switching to Databricks completely, but maybe it can help us on the transition.

docs/databases-overview.md Outdated Show resolved Hide resolved
@leo-schick leo-schick merged commit f71695a into main Sep 1, 2022
@leo-schick leo-schick deleted the databricks branch September 26, 2022 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants