Automating Tools Version Updates in Databricks Quick Start Notebooks

[parthosa]

Currently, we have to manually update the Tools version used in the Databricks Qualification/Profiling Quick Start Notebooks. This process should be automated using the following steps:

1. On every new release of Tools, run validation tests across both cloud platforms:
   • AWS: Create a single-node cluster
     1. Run Databricks Qualification Quick Start Notebook
     2. Run Databricks Profiling Quick Start Notebook
   • Azure: Create a single-node cluster
     1. Run Databricks Qualification Quick Start Notebook
     2. Run Databricks Profiling Quick Start Notebook
2. Upon successful completion of all tests
   • Automatically create a PR updating the Tools version in the notebooks