diff --git a/docsite/docs/core-concepts/semantic-intelligence/semantic-model.md b/docsite/docs/core-concepts/semantic-intelligence/semantic-model.md index 0b2741a..ad4b0e2 100644 --- a/docsite/docs/core-concepts/semantic-intelligence/semantic-model.md +++ b/docsite/docs/core-concepts/semantic-intelligence/semantic-model.md @@ -31,6 +31,10 @@ sm = SemanticModel(datasets, domain="Healthcare") While these examples use local CSV files, `intugle` can connect to various data sources. See our **[Connectors documentation](../../connectors/snowflake)** for details on specific integrations like Snowflake. ::: +:::warning Working with Remote URLs +While `intugle` supports providing remote URLs (e.g., a raw GitHub file link) in the `path`, it is **not recommended for large datasets**. For best performance and stability, we recommend downloading files to your local filesystem first and providing a local path. +::: + ### Method 2: From a List of DataSet Objects For more advanced scenarios, you can initialize the `SemanticModel` with a list of pre-configured `DataSet` objects. This is useful if you have already instantiated `DataSet` objects for other purposes. diff --git a/notebooks/quickstart_fmcg.ipynb b/notebooks/quickstart_fmcg.ipynb index ac3f39c..27af7cb 100644 --- a/notebooks/quickstart_fmcg.ipynb +++ b/notebooks/quickstart_fmcg.ipynb @@ -80,6 +80,14 @@ "print(\"All CSV files downloaded successfully.\")" ] }, + { + "cell_type": "markdown", + "id": "5c3c36bf", + "metadata": {}, + "source": [ + "> **Working with Remote URLs** : While `intugle` supports providing remote URLs (e.g., a raw GitHub file link) in the `path`, it is **not recommended for large datasets**. For best performance and stability, we recommend downloading files to your local filesystem first and providing a local path." + ] + }, { "cell_type": "markdown", "id": "27de922f", diff --git a/notebooks/quickstart_healthcare.ipynb b/notebooks/quickstart_healthcare.ipynb index d09aac5..0b6539b 100644 --- a/notebooks/quickstart_healthcare.ipynb +++ b/notebooks/quickstart_healthcare.ipynb @@ -80,6 +80,14 @@ "print(\"All CSV files downloaded successfully.\")" ] }, + { + "cell_type": "markdown", + "id": "eba33fbc", + "metadata": {}, + "source": [ + "> **Working with Remote URLs** : While `intugle` supports providing remote URLs (e.g., a raw GitHub file link) in the `path`, it is **not recommended for large datasets**. For best performance and stability, we recommend downloading files to your local filesystem first and providing a local path." + ] + }, { "cell_type": "markdown", "id": "4887528e", diff --git a/notebooks/quickstart_healthcare_databricks.ipynb b/notebooks/quickstart_healthcare_databricks.ipynb index 8156358..1e564a6 100644 --- a/notebooks/quickstart_healthcare_databricks.ipynb +++ b/notebooks/quickstart_healthcare_databricks.ipynb @@ -192,7 +192,7 @@ "4. **Glossary Generation:** Generates a business glossary for each column using an LLM.\n", "5. **Link Prediction:** Predicts the relationships (foreign keys) between tables.\n", "\n", - "Let's start by defining the datasets we want to use. The path shown below can be a local file path or a remote URL." + "Let's start by defining the datasets we want to use. " ] }, { diff --git a/notebooks/quickstart_native_databricks.ipynb b/notebooks/quickstart_native_databricks.ipynb index cb46adb..e6dc45e 100644 --- a/notebooks/quickstart_native_databricks.ipynb +++ b/notebooks/quickstart_native_databricks.ipynb @@ -168,7 +168,7 @@ "4. **Glossary Generation:** Generates a business glossary for each column using an LLM.\n", "5. **Link Prediction:** Predicts the relationships (foreign keys) between tables.\n", "\n", - "Let's start by defining the datasets we want to use. The path shown below can be a local file path or a remote URL." + "Let's start by defining the datasets we want to use. " ] }, { @@ -293,7 +293,46 @@ "name": "stdout", "output_type": "stream", "text": [ - "Found existing YAML for 'campaigns'. Checking for staleness.\ncampaigns loaded\nFound existing YAML for 'campaign_survey'. Checking for staleness.\ncampaign_survey loaded\nFound existing YAML for 'customer_hierarchy'. Checking for staleness.\ncustomer_hierarchy loaded\nFound existing YAML for 'customers'. Checking for staleness.\ncustomers loaded\nFound existing YAML for 'delivery_survey'. Checking for staleness.\ndelivery_survey loaded\nFound existing YAML for 'expense'. Checking for staleness.\nexpense loaded\nFound existing YAML for 'install_base'. Checking for staleness.\ninstall_base loaded\nFound existing YAML for 'inventory'. Checking for staleness.\ninventory loaded\nFound existing YAML for 'logistics'. Checking for staleness.\nlogistics loaded\nFound existing YAML for 'nps_survey'. Checking for staleness.\nnps_survey loaded\nFound existing YAML for 'opportunity'. Checking for staleness.\nopportunity loaded\nFound existing YAML for 'orders'. Checking for staleness.\norders loaded\nFound existing YAML for 'prob_statement_issue'. Checking for staleness.\nprob_statement_issue loaded\nFound existing YAML for 'product_feature'. Checking for staleness.\nproduct_feature loaded\nFound existing YAML for 'product_hierarchy'. Checking for staleness.\nproduct_hierarchy loaded\nFound existing YAML for 'products'. Checking for staleness.\nproducts loaded\nFound existing YAML for 'renewals'. Checking for staleness.\nrenewals loaded\nFound existing YAML for 'returns'. Checking for staleness.\nreturns loaded\nFound existing YAML for 'service_requests'. Checking for staleness.\nservice_requests loaded\nFound existing YAML for 'website'. Checking for staleness.\nwebsite loaded\n" + "Found existing YAML for 'campaigns'. Checking for staleness.\n", + "campaigns loaded\n", + "Found existing YAML for 'campaign_survey'. Checking for staleness.\n", + "campaign_survey loaded\n", + "Found existing YAML for 'customer_hierarchy'. Checking for staleness.\n", + "customer_hierarchy loaded\n", + "Found existing YAML for 'customers'. Checking for staleness.\n", + "customers loaded\n", + "Found existing YAML for 'delivery_survey'. Checking for staleness.\n", + "delivery_survey loaded\n", + "Found existing YAML for 'expense'. Checking for staleness.\n", + "expense loaded\n", + "Found existing YAML for 'install_base'. Checking for staleness.\n", + "install_base loaded\n", + "Found existing YAML for 'inventory'. Checking for staleness.\n", + "inventory loaded\n", + "Found existing YAML for 'logistics'. Checking for staleness.\n", + "logistics loaded\n", + "Found existing YAML for 'nps_survey'. Checking for staleness.\n", + "nps_survey loaded\n", + "Found existing YAML for 'opportunity'. Checking for staleness.\n", + "opportunity loaded\n", + "Found existing YAML for 'orders'. Checking for staleness.\n", + "orders loaded\n", + "Found existing YAML for 'prob_statement_issue'. Checking for staleness.\n", + "prob_statement_issue loaded\n", + "Found existing YAML for 'product_feature'. Checking for staleness.\n", + "product_feature loaded\n", + "Found existing YAML for 'product_hierarchy'. Checking for staleness.\n", + "product_hierarchy loaded\n", + "Found existing YAML for 'products'. Checking for staleness.\n", + "products loaded\n", + "Found existing YAML for 'renewals'. Checking for staleness.\n", + "renewals loaded\n", + "Found existing YAML for 'returns'. Checking for staleness.\n", + "returns loaded\n", + "Found existing YAML for 'service_requests'. Checking for staleness.\n", + "service_requests loaded\n", + "Found existing YAML for 'website'. Checking for staleness.\n", + "website loaded\n" ] }, { @@ -313,7 +352,26 @@ "name": "stdout", "output_type": "stream", "text": [ - "Dataset 'campaigns' already profiled. Skipping.\nDataset 'campaign_survey' already profiled. Skipping.\nDataset 'customer_hierarchy' already profiled. Skipping.\nDataset 'customers' already profiled. Skipping.\nDataset 'delivery_survey' already profiled. Skipping.\nDataset 'expense' already profiled. Skipping.\nDataset 'install_base' already profiled. Skipping.\nDataset 'inventory' already profiled. Skipping.\nDataset 'logistics' already profiled. Skipping.\nDataset 'nps_survey' already profiled. Skipping.\nDataset 'opportunity' already profiled. Skipping.\nDataset 'orders' already profiled. Skipping.\nDataset 'prob_statement_issue' already profiled. Skipping.\nDataset 'product_feature' already profiled. Skipping.\nDataset 'product_hierarchy' already profiled. Skipping.\nDataset 'products' already profiled. Skipping.\nDataset 'renewals' already profiled. Skipping.\nDataset 'returns' already profiled. Skipping.\nDataset 'service_requests' already profiled. Skipping.\nDataset 'website' already profiled. Skipping.\n" + "Dataset 'campaigns' already profiled. Skipping.\n", + "Dataset 'campaign_survey' already profiled. Skipping.\n", + "Dataset 'customer_hierarchy' already profiled. Skipping.\n", + "Dataset 'customers' already profiled. Skipping.\n", + "Dataset 'delivery_survey' already profiled. Skipping.\n", + "Dataset 'expense' already profiled. Skipping.\n", + "Dataset 'install_base' already profiled. Skipping.\n", + "Dataset 'inventory' already profiled. Skipping.\n", + "Dataset 'logistics' already profiled. Skipping.\n", + "Dataset 'nps_survey' already profiled. Skipping.\n", + "Dataset 'opportunity' already profiled. Skipping.\n", + "Dataset 'orders' already profiled. Skipping.\n", + "Dataset 'prob_statement_issue' already profiled. Skipping.\n", + "Dataset 'product_feature' already profiled. Skipping.\n", + "Dataset 'product_hierarchy' already profiled. Skipping.\n", + "Dataset 'products' already profiled. Skipping.\n", + "Dataset 'renewals' already profiled. Skipping.\n", + "Dataset 'returns' already profiled. Skipping.\n", + "Dataset 'service_requests' already profiled. Skipping.\n", + "Dataset 'website' already profiled. Skipping.\n" ] }, { @@ -346,7 +404,27 @@ "name": "stdout", "output_type": "stream", "text": [ - "Dataset 'campaigns' already processed. Skipping analysis.\nDataset 'campaign_survey' already processed. Skipping analysis.\nDataset 'customer_hierarchy' already processed. Skipping analysis.\nDataset 'customers' already processed. Skipping analysis.\nDataset 'delivery_survey' already processed. Skipping analysis.\nDataset 'expense' already processed. Skipping analysis.\nDataset 'install_base' already processed. Skipping analysis.\nDataset 'inventory' already processed. Skipping analysis.\nDataset 'logistics' already processed. Skipping analysis.\nDataset 'nps_survey' already processed. Skipping analysis.\nDataset 'opportunity' already processed. Skipping analysis.\nDataset 'orders' already processed. Skipping analysis.\nDataset 'prob_statement_issue' already processed. Skipping analysis.\nDataset 'product_feature' already processed. Skipping analysis.\nDataset 'product_hierarchy' already processed. Skipping analysis.\nDataset 'products' already processed. Skipping analysis.\nDataset 'renewals' already processed. Skipping analysis.\nDataset 'returns' already processed. Skipping analysis.\nDataset 'service_requests' already processed. Skipping analysis.\nDataset 'website' already processed. Skipping analysis.\nLinkPredictor initialized with datasets: ['campaigns', 'campaign_survey', 'customer_hierarchy', 'customers', 'delivery_survey', 'expense', 'install_base', 'inventory', 'logistics', 'nps_survey', 'opportunity', 'orders', 'prob_statement_issue', 'product_feature', 'product_hierarchy', 'products', 'renewals', 'returns', 'service_requests', 'website']\n" + "Dataset 'campaigns' already processed. Skipping analysis.\n", + "Dataset 'campaign_survey' already processed. Skipping analysis.\n", + "Dataset 'customer_hierarchy' already processed. Skipping analysis.\n", + "Dataset 'customers' already processed. Skipping analysis.\n", + "Dataset 'delivery_survey' already processed. Skipping analysis.\n", + "Dataset 'expense' already processed. Skipping analysis.\n", + "Dataset 'install_base' already processed. Skipping analysis.\n", + "Dataset 'inventory' already processed. Skipping analysis.\n", + "Dataset 'logistics' already processed. Skipping analysis.\n", + "Dataset 'nps_survey' already processed. Skipping analysis.\n", + "Dataset 'opportunity' already processed. Skipping analysis.\n", + "Dataset 'orders' already processed. Skipping analysis.\n", + "Dataset 'prob_statement_issue' already processed. Skipping analysis.\n", + "Dataset 'product_feature' already processed. Skipping analysis.\n", + "Dataset 'product_hierarchy' already processed. Skipping analysis.\n", + "Dataset 'products' already processed. Skipping analysis.\n", + "Dataset 'renewals' already processed. Skipping analysis.\n", + "Dataset 'returns' already processed. Skipping analysis.\n", + "Dataset 'service_requests' already processed. Skipping analysis.\n", + "Dataset 'website' already processed. Skipping analysis.\n", + "LinkPredictor initialized with datasets: ['campaigns', 'campaign_survey', 'customer_hierarchy', 'customers', 'delivery_survey', 'expense', 'install_base', 'inventory', 'logistics', 'nps_survey', 'opportunity', 'orders', 'prob_statement_issue', 'product_feature', 'product_hierarchy', 'products', 'renewals', 'returns', 'service_requests', 'website']\n" ] }, { @@ -672,7 +750,8 @@ "name": "stderr", "output_type": "stream", "text": [ - "Could not initialize semantic search: Error code: 401 - {'error': {'message': 'Incorrect API key provided: your-ope*******-key. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}\nSemantic search initialization failed during build: Error code: 401 - {'error': {'message': 'Incorrect API key provided: your-ope*******-key. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}\n" + "Could not initialize semantic search: Error code: 401 - {'error': {'message': 'Incorrect API key provided: your-ope*******-key. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}\n", + "Semantic search initialization failed during build: Error code: 401 - {'error': {'message': 'Incorrect API key provided: your-ope*******-key. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}\n" ] }, { @@ -1237,7 +1316,180 @@ "name": "stdout", "output_type": "stream", "text": [ - "Setting primary key constraints...\nSet primary key on `intugle`.`tech_manufacturing`.`product_hierarchy` (`SKU ID`)\nSkipping primary key for table 'product_feature' due to missing or invalid key.\nSet primary key on `intugle`.`tech_manufacturing`.`website` (`Customer Session ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`prob_statement_issue` (`Issue ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`customer_hierarchy` (`Account Manager`)\nSet primary key on `intugle`.`tech_manufacturing`.`campaigns` (`Campaign ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`products` (`Product ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`service_requests` (`SR ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`logistics` (`Logistics ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`customers` (`C_ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`expense` (`Expense ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`install_base` (`Order ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`campaign_survey` (`Participant Name`)\nSet primary key on `intugle`.`tech_manufacturing`.`nps_survey` (`NPS_ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`orders` (`Order ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`opportunity` (`Opportunity ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`renewals` (`Renw ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`returns` (`Return_ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`delivery_survey` (`Odr ID`)\nSet primary key on `intugle`.`tech_manufacturing`.`inventory` (`Inventory ID`)\nPrimary key setting complete.\nSetting foreign key constraints...\nCould not set foreign key for relationship install_base_orders: Failed to create foreign key constraint `fk_install_base_orders`: table `intugle.tech_manufacturing.orders` already has a foreign key constraint: `fk_delivery_survey_orders`, that has the same set of child columns.\n\nJVM stacktrace:\norg.apache.spark.sql.catalyst.analysis.ConstraintAlreadyExistsException\n\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.$anonfun$createTableConstraint$1(ManagedCatalogClientImpl.scala:4008)\n\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.$anonfun$recordAndWrapExceptionBase$2(ManagedCatalogClientImpl.scala:7505)\n\tat com.databricks.spark.util.FrameProfiler$.$anonfun$record$1(FrameProfiler.scala:114)\n\tat com.databricks.spark.util.FrameProfilerExporter$.maybeExportFrameProfiler(FrameProfilerExporter.scala:200)\n\tat com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:105)\n\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.$anonfun$recordAndWrapExceptionBase$1(ManagedCatalogClientImpl.scala:7504)\n\tat com.databricks.managedcatalog.ErrorDetailsHandler.wrapServiceException(ErrorDetailsHandler.scala:74)\n\tat com.databricks.managedcatalog.ErrorDetailsHandler.wrapServiceException$(ErrorDetailsHandler.scala:66)\n\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.wrapServiceException(ManagedCatalogClientImpl.scala:268)\n\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.recordAndWrapExceptionBase(ManagedCatalogClientImpl.scala:7485)\n\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.recordAndWrapException(ManagedCatalogClientImpl.scala:7471)\n\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.createTableConstraint(ManagedCatalogClientImpl.scala:3913)\n\tat com.databricks.sql.managedcatalog.ManagedCatalogCommon.$anonfun$addTableConstraint$1(ManagedCatalogCommon.scala:2084)\n\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)\n\tat com.databricks.sql.managedcatalog.ManagedCatalogCommon.withTableCacheInvalidated(ManagedCatalogCommon.scala:2336)\n\tat com.databricks.sql.managedcatalog.ManagedCatalogCommon.addTableConstraint(ManagedCatalogCommon.scala:2076)\n\tat com.databricks.sql.managedcatalog.ProfiledManagedCatalog.$anonfun$addTableConstraint$1(ProfiledManagedCatalog.scala:356)\n\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)\n\tat org.apache.spark.sql.catalyst.MetricKeyUtils$.measure(MetricKey.scala:1892)\n\tat com.databricks.sql.managedcatalog.ProfiledManagedCatalog.$anonfun$profile$1(ProfiledManagedCatalog.scala:64)\n\tat com.databricks.spark.util.FrameProfiler$.$anonfun$record$1(FrameProfiler.scala:114)\n\tat com.databricks.spark.util.FrameProfilerExporter$.maybeExportFrameProfiler(FrameProfilerExporter.scala:200)\n\tat com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:105)\n\tat com.databricks.sql.managedcatalog.ProfiledManagedCatalog.profile(ProfiledManagedCatalog.scala:63)\n\tat com.databricks.sql.managedcatalog.ProfiledManagedCatalog.addTableConstraint(ProfiledManagedCatalog.scala:356)\n\tat com.databricks.sql.managedcatalog.ManagedCatalogSessionCatalog.addTableConstraint(ManagedCatalogSessionCatalog.scala:1551)\n\tat com.databricks.sql.transaction.tahoe.commands.AlterTableAddTableConstraintDeltaCommand.$anonfun$run$66(alterDeltaTableCommands.scala:1924)\n\tat com.databricks.sql.transaction.tahoe.commands.AlterTableAddTableConstraintDeltaCommand.$anonfun$run$66$adapted(alterDeltaTableCommands.scala:1923)\n\tat scala.Option.foreach(Option.scala:437)\n\tat com.databricks.sql.transaction.tahoe.commands.AlterTableAddTableConstraintDeltaCommand.run(alterDeltaTableCommands.scala:1923)\n\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.$anonfun$alterTable$44(DeltaCatalog.scala:2083)\n\tat scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)\n\tat scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)\n\tat scala.collection.AbstractIterable.foreach(Iterable.scala:935)\n\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.$anonfun$alterTable$22(DeltaCatalog.scala:2079)\n\tat scala.collection.immutable.HashMap.foreach(HashMap.scala:1115)\n\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.$anonfun$alterTable$8(DeltaCatalog.scala:1853)\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)\n\tat com.databricks.sql.transaction.tahoe.redirect.RedirectFeature$.withUpdateTableRedirectDDL(TableRedirect.scala:796)\n\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.$anonfun$alterTable$1(DeltaCatalog.scala:1762)\n\tat com.databricks.spark.util.FrameProfiler$.$anonfun$record$1(FrameProfiler.scala:114)\n\tat com.databricks.spark.util.FrameProfilerExporter$.maybeExportFrameProfiler(FrameProfilerExporter.scala:200)\n\tat com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:105)\n\tat com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:420)\n\tat com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:418)\n\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.recordFrameProfile(DeltaCatalog.scala:144)\n\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.alterTable(DeltaCatalog.scala:1741)\n\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.alterTable(DeltaCatalog.scala:144)\n\tat com.databricks.sql.managedcatalog.UnityCatalogV2Proxy.alterTable(UnityCatalogV2Proxy.scala:261)\n\tat com.databricks.sql.managedcatalog.UnityCatalogV2Proxy.alterTable(UnityCatalogV2Proxy.scala:57)\n\tat org.apache.spark.sql.execution.datasources.v2.AlterTableExec.run(AlterTableExec.scala:38)\n\tat org.apache.spark.sql.execution.datasources.v2.V2CommandExec.$anonfun$result$2(V2CommandExec.scala:48)\n\tat org.apache.spark.sql.execution.SparkPlan.runCommandInAetherOrSpark(SparkPlan.scala:195)\n\tat org.apache.spark.sql.execution.datasources.v2.V2CommandExec.$anonfun$result$1(V2CommandExec.scala:48)\n\tat com.databricks.spark.util.FrameProfiler$.$anonfun$record$1(FrameProfiler.scala:114)\n\tat com.databricks.spark.util.FrameProfilerExporter$.maybeExportFrameProfiler(FrameProfilerExporter.scala:200)\n\tat com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:105)\n\tat org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:47)\n\tat org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:45)\n\tat org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:56)\n\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$5(QueryExecution.scala:507)\n\tat com.databricks.util.LexicalThreadLocal$Handle.runWith(LexicalThreadLocal.scala:63)\n\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$4(QueryExecution.scala:507)\n\tat org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:267)\n\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$3(QueryExecution.scala:506)\n\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$14(SQLExecution.scala:561)\n\tat com.databricks.sql.util.MemoryTrackerHelper.withMemoryTracking(MemoryTrackerHelper.scala:111)\n\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$13(SQLExecution.scala:475)\n\tat org.apache.spark.sql.execution.SQLExecution$.withSessionTagsApplied(SQLExecution.scala:859)\n\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$12(SQLExecution.scala:403)\n\tat org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:97)\n\tat org.apache.spark.sql.artifact.ArtifactManager.$anonfun$withResources$1(ArtifactManager.scala:121)\n\tat org.apache.spark.sql.artifact.ArtifactManager.withClassLoaderIfNeeded(ArtifactManager.scala:115)\n\tat org.apache.spark.sql.artifact.ArtifactManager.withResources(ArtifactManager.scala:120)\n\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$11(SQLExecution.scala:403)\n\tat org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:888)\n\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$1(SQLExecution.scala:402)\n\tat org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:860)\n\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId0(SQLExecution.scala:238)\n\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:812)\n\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$2(QueryExecution.scala:502)\n\tat org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:1449)\n\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$1(QueryExecution.scala:498)\n\tat org.apache.spark.sql.execution.QueryExecution.withMVTagsIfNecessary(QueryExecution.scala:418)\n\tat org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$eagerlyExecute$1(QueryExecution.scala:496)\n\tat org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$8$1.applyOrElse(QueryExecution.scala:578)\n\tat org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$8$1.applyOrElse(QueryExecution.scala:570)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:529)\n\tat org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:121)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:529)\n\tat org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:42)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:361)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:357)\n\tat org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:42)\n\tat org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:42)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:505)\n\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$8(QueryExecution.scala:570)\n\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:418)\n\tat org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:570)\n\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$lazyCommandExecuted$1(QueryExecution.scala:374)\n\tat scala.util.Try$.apply(Try.scala:217)\n\tat org.apache.spark.util.Utils$.doTryWithCallerStacktrace(Utils.scala:1686)\n\tat org.apache.spark.util.Utils$.getTryWithCallerStacktrace(Utils.scala:1747)\n\tat org.apache.spark.util.LazyTry.get(LazyTry.scala:75)\n\tat org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:379)\n\tat org.apache.spark.sql.classic.Dataset.(Dataset.scala:432)\n\tat org.apache.spark.sql.classic.Dataset$.$anonfun$ofRows$3(Dataset.scala:155)\n\tat org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:860)\n\tat org.apache.spark.sql.classic.SparkSession.$anonfun$withActiveAndFrameProfiler$1(SparkSession.scala:1072)\n\tat com.databricks.spark.util.FrameProfiler$.$anonfun$record$1(FrameProfiler.scala:114)\n\tat com.databricks.spark.util.FrameProfilerExporter$.maybeExportFrameProfiler(FrameProfilerExporter.scala:200)\n\tat com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:105)\n\tat org.apache.spark.sql.classic.SparkSession.withActiveAndFrameProfiler(SparkSession.scala:1072)\n\tat org.apache.spark.sql.classic.Dataset$.ofRows(Dataset.scala:146)\n\tat org.apache.spark.sql.classic.SparkSession.$anonfun$sql$4(SparkSession.scala:851)\n\tat org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:860)\n\tat org.apache.spark.sql.classic.SparkSession.sql(SparkSession.scala:814)\n\tat org.apache.spark.sql.connect.planner.SparkConnectPlanner.executeSQL(SparkConnectPlanner.scala:3531)\n\tat org.apache.spark.sql.connect.planner.SparkConnectPlanner.handleSqlCommand(SparkConnectPlanner.scala:3361)\n\tat org.apache.spark.sql.connect.planner.SparkConnectPlanner.process(SparkConnectPlanner.scala:3238)\n\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.handleCommand(ExecuteThreadRunner.scala:385)\n\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1(ExecuteThreadRunner.scala:282)\n\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1$adapted(ExecuteThreadRunner.scala:238)\n\tat org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$2(SessionHolder.scala:466)\n\tat org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:860)\n\tat org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$1(SessionHolder.scala:466)\n\tat org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:97)\n\tat org.apache.spark.sql.artifact.ArtifactManager.$anonfun$withResources$1(ArtifactManager.scala:121)\n\tat org.apache.spark.sql.artifact.ArtifactManager.withClassLoaderIfNeeded(ArtifactManager.scala:115)\n\tat org.apache.spark.sql.artifact.ArtifactManager.withResources(ArtifactManager.scala:120)\n\tat org.apache.spark.sql.connect.service.SessionHolder.withSession(SessionHolder.scala:465)\n\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.executeInternal(ExecuteThreadRunner.scala:238)\n\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$execute$1(ExecuteThreadRunner.scala:141)\n\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)\n\tat com.databricks.spark.connect.service.UtilizationMetrics.recordActiveQueries(UtilizationMetrics.scala:43)\n\tat com.databricks.spark.connect.service.UtilizationMetrics.recordActiveQueries$(UtilizationMetrics.scala:40)\n\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.recordActiveQueries(ExecuteThreadRunner.scala:53)\n\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.org$apache$spark$sql$connect$execution$ExecuteThreadRunner$$execute(ExecuteThreadRunner.scala:139)\n\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner$ExecutionThread.$anonfun$run$2(ExecuteThreadRunner.scala:586)\n\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)\n\tat com.databricks.unity.UCSEphemeralState$Handle.runWith(UCSEphemeralState.scala:51)\n\tat com.databricks.unity.HandleImpl.runWith(UCSHandle.scala:104)\n\tat com.databricks.unity.HandleImpl.$anonfun$runWithAndClose$1(UCSHandle.scala:109)\n\tat scala.util.Using$.resource(Using.scala:296)\n\tat com.databricks.unity.HandleImpl.runWithAndClose(UCSHandle.scala:108)\n\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner$ExecutionThread.run(ExecuteThreadRunner.scala:586)\nForeign key setting complete.\n" + "Setting primary key constraints...\n", + "Set primary key on `intugle`.`tech_manufacturing`.`product_hierarchy` (`SKU ID`)\n", + "Skipping primary key for table 'product_feature' due to missing or invalid key.\n", + "Set primary key on `intugle`.`tech_manufacturing`.`website` (`Customer Session ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`prob_statement_issue` (`Issue ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`customer_hierarchy` (`Account Manager`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`campaigns` (`Campaign ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`products` (`Product ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`service_requests` (`SR ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`logistics` (`Logistics ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`customers` (`C_ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`expense` (`Expense ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`install_base` (`Order ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`campaign_survey` (`Participant Name`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`nps_survey` (`NPS_ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`orders` (`Order ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`opportunity` (`Opportunity ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`renewals` (`Renw ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`returns` (`Return_ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`delivery_survey` (`Odr ID`)\n", + "Set primary key on `intugle`.`tech_manufacturing`.`inventory` (`Inventory ID`)\n", + "Primary key setting complete.\n", + "Setting foreign key constraints...\n", + "Could not set foreign key for relationship install_base_orders: Failed to create foreign key constraint `fk_install_base_orders`: table `intugle.tech_manufacturing.orders` already has a foreign key constraint: `fk_delivery_survey_orders`, that has the same set of child columns.\n", + "\n", + "JVM stacktrace:\n", + "org.apache.spark.sql.catalyst.analysis.ConstraintAlreadyExistsException\n", + "\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.$anonfun$createTableConstraint$1(ManagedCatalogClientImpl.scala:4008)\n", + "\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.$anonfun$recordAndWrapExceptionBase$2(ManagedCatalogClientImpl.scala:7505)\n", + "\tat com.databricks.spark.util.FrameProfiler$.$anonfun$record$1(FrameProfiler.scala:114)\n", + "\tat com.databricks.spark.util.FrameProfilerExporter$.maybeExportFrameProfiler(FrameProfilerExporter.scala:200)\n", + "\tat com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:105)\n", + "\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.$anonfun$recordAndWrapExceptionBase$1(ManagedCatalogClientImpl.scala:7504)\n", + "\tat com.databricks.managedcatalog.ErrorDetailsHandler.wrapServiceException(ErrorDetailsHandler.scala:74)\n", + "\tat com.databricks.managedcatalog.ErrorDetailsHandler.wrapServiceException$(ErrorDetailsHandler.scala:66)\n", + "\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.wrapServiceException(ManagedCatalogClientImpl.scala:268)\n", + "\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.recordAndWrapExceptionBase(ManagedCatalogClientImpl.scala:7485)\n", + "\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.recordAndWrapException(ManagedCatalogClientImpl.scala:7471)\n", + "\tat com.databricks.managedcatalog.ManagedCatalogClientImpl.createTableConstraint(ManagedCatalogClientImpl.scala:3913)\n", + "\tat com.databricks.sql.managedcatalog.ManagedCatalogCommon.$anonfun$addTableConstraint$1(ManagedCatalogCommon.scala:2084)\n", + "\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)\n", + "\tat com.databricks.sql.managedcatalog.ManagedCatalogCommon.withTableCacheInvalidated(ManagedCatalogCommon.scala:2336)\n", + "\tat com.databricks.sql.managedcatalog.ManagedCatalogCommon.addTableConstraint(ManagedCatalogCommon.scala:2076)\n", + "\tat com.databricks.sql.managedcatalog.ProfiledManagedCatalog.$anonfun$addTableConstraint$1(ProfiledManagedCatalog.scala:356)\n", + "\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)\n", + "\tat org.apache.spark.sql.catalyst.MetricKeyUtils$.measure(MetricKey.scala:1892)\n", + "\tat com.databricks.sql.managedcatalog.ProfiledManagedCatalog.$anonfun$profile$1(ProfiledManagedCatalog.scala:64)\n", + "\tat com.databricks.spark.util.FrameProfiler$.$anonfun$record$1(FrameProfiler.scala:114)\n", + "\tat com.databricks.spark.util.FrameProfilerExporter$.maybeExportFrameProfiler(FrameProfilerExporter.scala:200)\n", + "\tat com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:105)\n", + "\tat com.databricks.sql.managedcatalog.ProfiledManagedCatalog.profile(ProfiledManagedCatalog.scala:63)\n", + "\tat com.databricks.sql.managedcatalog.ProfiledManagedCatalog.addTableConstraint(ProfiledManagedCatalog.scala:356)\n", + "\tat com.databricks.sql.managedcatalog.ManagedCatalogSessionCatalog.addTableConstraint(ManagedCatalogSessionCatalog.scala:1551)\n", + "\tat com.databricks.sql.transaction.tahoe.commands.AlterTableAddTableConstraintDeltaCommand.$anonfun$run$66(alterDeltaTableCommands.scala:1924)\n", + "\tat com.databricks.sql.transaction.tahoe.commands.AlterTableAddTableConstraintDeltaCommand.$anonfun$run$66$adapted(alterDeltaTableCommands.scala:1923)\n", + "\tat scala.Option.foreach(Option.scala:437)\n", + "\tat com.databricks.sql.transaction.tahoe.commands.AlterTableAddTableConstraintDeltaCommand.run(alterDeltaTableCommands.scala:1923)\n", + "\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.$anonfun$alterTable$44(DeltaCatalog.scala:2083)\n", + "\tat scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)\n", + "\tat scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)\n", + "\tat scala.collection.AbstractIterable.foreach(Iterable.scala:935)\n", + "\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.$anonfun$alterTable$22(DeltaCatalog.scala:2079)\n", + "\tat scala.collection.immutable.HashMap.foreach(HashMap.scala:1115)\n", + "\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.$anonfun$alterTable$8(DeltaCatalog.scala:1853)\n", + "\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)\n", + "\tat com.databricks.sql.transaction.tahoe.redirect.RedirectFeature$.withUpdateTableRedirectDDL(TableRedirect.scala:796)\n", + "\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.$anonfun$alterTable$1(DeltaCatalog.scala:1762)\n", + "\tat com.databricks.spark.util.FrameProfiler$.$anonfun$record$1(FrameProfiler.scala:114)\n", + "\tat com.databricks.spark.util.FrameProfilerExporter$.maybeExportFrameProfiler(FrameProfilerExporter.scala:200)\n", + "\tat com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:105)\n", + "\tat com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:420)\n", + "\tat com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:418)\n", + "\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.recordFrameProfile(DeltaCatalog.scala:144)\n", + "\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.alterTable(DeltaCatalog.scala:1741)\n", + "\tat com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.alterTable(DeltaCatalog.scala:144)\n", + "\tat com.databricks.sql.managedcatalog.UnityCatalogV2Proxy.alterTable(UnityCatalogV2Proxy.scala:261)\n", + "\tat com.databricks.sql.managedcatalog.UnityCatalogV2Proxy.alterTable(UnityCatalogV2Proxy.scala:57)\n", + "\tat org.apache.spark.sql.execution.datasources.v2.AlterTableExec.run(AlterTableExec.scala:38)\n", + "\tat org.apache.spark.sql.execution.datasources.v2.V2CommandExec.$anonfun$result$2(V2CommandExec.scala:48)\n", + "\tat org.apache.spark.sql.execution.SparkPlan.runCommandInAetherOrSpark(SparkPlan.scala:195)\n", + "\tat org.apache.spark.sql.execution.datasources.v2.V2CommandExec.$anonfun$result$1(V2CommandExec.scala:48)\n", + "\tat com.databricks.spark.util.FrameProfiler$.$anonfun$record$1(FrameProfiler.scala:114)\n", + "\tat com.databricks.spark.util.FrameProfilerExporter$.maybeExportFrameProfiler(FrameProfilerExporter.scala:200)\n", + "\tat com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:105)\n", + "\tat org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:47)\n", + "\tat org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:45)\n", + "\tat org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:56)\n", + "\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$5(QueryExecution.scala:507)\n", + "\tat com.databricks.util.LexicalThreadLocal$Handle.runWith(LexicalThreadLocal.scala:63)\n", + "\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$4(QueryExecution.scala:507)\n", + "\tat org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:267)\n", + "\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$3(QueryExecution.scala:506)\n", + "\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$14(SQLExecution.scala:561)\n", + "\tat com.databricks.sql.util.MemoryTrackerHelper.withMemoryTracking(MemoryTrackerHelper.scala:111)\n", + "\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$13(SQLExecution.scala:475)\n", + "\tat org.apache.spark.sql.execution.SQLExecution$.withSessionTagsApplied(SQLExecution.scala:859)\n", + "\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$12(SQLExecution.scala:403)\n", + "\tat org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:97)\n", + "\tat org.apache.spark.sql.artifact.ArtifactManager.$anonfun$withResources$1(ArtifactManager.scala:121)\n", + "\tat org.apache.spark.sql.artifact.ArtifactManager.withClassLoaderIfNeeded(ArtifactManager.scala:115)\n", + "\tat org.apache.spark.sql.artifact.ArtifactManager.withResources(ArtifactManager.scala:120)\n", + "\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$11(SQLExecution.scala:403)\n", + "\tat org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:888)\n", + "\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$1(SQLExecution.scala:402)\n", + "\tat org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:860)\n", + "\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId0(SQLExecution.scala:238)\n", + "\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:812)\n", + "\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$2(QueryExecution.scala:502)\n", + "\tat org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:1449)\n", + "\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$1(QueryExecution.scala:498)\n", + "\tat org.apache.spark.sql.execution.QueryExecution.withMVTagsIfNecessary(QueryExecution.scala:418)\n", + "\tat org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$eagerlyExecute$1(QueryExecution.scala:496)\n", + "\tat org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$8$1.applyOrElse(QueryExecution.scala:578)\n", + "\tat org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$8$1.applyOrElse(QueryExecution.scala:570)\n", + "\tat org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:529)\n", + "\tat org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:121)\n", + "\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:529)\n", + "\tat org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:42)\n", + "\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:361)\n", + "\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:357)\n", + "\tat org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:42)\n", + "\tat org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:42)\n", + "\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:505)\n", + "\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$8(QueryExecution.scala:570)\n", + "\tat org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:418)\n", + "\tat org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:570)\n", + "\tat org.apache.spark.sql.execution.QueryExecution.$anonfun$lazyCommandExecuted$1(QueryExecution.scala:374)\n", + "\tat scala.util.Try$.apply(Try.scala:217)\n", + "\tat org.apache.spark.util.Utils$.doTryWithCallerStacktrace(Utils.scala:1686)\n", + "\tat org.apache.spark.util.Utils$.getTryWithCallerStacktrace(Utils.scala:1747)\n", + "\tat org.apache.spark.util.LazyTry.get(LazyTry.scala:75)\n", + "\tat org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:379)\n", + "\tat org.apache.spark.sql.classic.Dataset.(Dataset.scala:432)\n", + "\tat org.apache.spark.sql.classic.Dataset$.$anonfun$ofRows$3(Dataset.scala:155)\n", + "\tat org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:860)\n", + "\tat org.apache.spark.sql.classic.SparkSession.$anonfun$withActiveAndFrameProfiler$1(SparkSession.scala:1072)\n", + "\tat com.databricks.spark.util.FrameProfiler$.$anonfun$record$1(FrameProfiler.scala:114)\n", + "\tat com.databricks.spark.util.FrameProfilerExporter$.maybeExportFrameProfiler(FrameProfilerExporter.scala:200)\n", + "\tat com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:105)\n", + "\tat org.apache.spark.sql.classic.SparkSession.withActiveAndFrameProfiler(SparkSession.scala:1072)\n", + "\tat org.apache.spark.sql.classic.Dataset$.ofRows(Dataset.scala:146)\n", + "\tat org.apache.spark.sql.classic.SparkSession.$anonfun$sql$4(SparkSession.scala:851)\n", + "\tat org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:860)\n", + "\tat org.apache.spark.sql.classic.SparkSession.sql(SparkSession.scala:814)\n", + "\tat org.apache.spark.sql.connect.planner.SparkConnectPlanner.executeSQL(SparkConnectPlanner.scala:3531)\n", + "\tat org.apache.spark.sql.connect.planner.SparkConnectPlanner.handleSqlCommand(SparkConnectPlanner.scala:3361)\n", + "\tat org.apache.spark.sql.connect.planner.SparkConnectPlanner.process(SparkConnectPlanner.scala:3238)\n", + "\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.handleCommand(ExecuteThreadRunner.scala:385)\n", + "\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1(ExecuteThreadRunner.scala:282)\n", + "\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1$adapted(ExecuteThreadRunner.scala:238)\n", + "\tat org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$2(SessionHolder.scala:466)\n", + "\tat org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:860)\n", + "\tat org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$1(SessionHolder.scala:466)\n", + "\tat org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:97)\n", + "\tat org.apache.spark.sql.artifact.ArtifactManager.$anonfun$withResources$1(ArtifactManager.scala:121)\n", + "\tat org.apache.spark.sql.artifact.ArtifactManager.withClassLoaderIfNeeded(ArtifactManager.scala:115)\n", + "\tat org.apache.spark.sql.artifact.ArtifactManager.withResources(ArtifactManager.scala:120)\n", + "\tat org.apache.spark.sql.connect.service.SessionHolder.withSession(SessionHolder.scala:465)\n", + "\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.executeInternal(ExecuteThreadRunner.scala:238)\n", + "\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$execute$1(ExecuteThreadRunner.scala:141)\n", + "\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)\n", + "\tat com.databricks.spark.connect.service.UtilizationMetrics.recordActiveQueries(UtilizationMetrics.scala:43)\n", + "\tat com.databricks.spark.connect.service.UtilizationMetrics.recordActiveQueries$(UtilizationMetrics.scala:40)\n", + "\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.recordActiveQueries(ExecuteThreadRunner.scala:53)\n", + "\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner.org$apache$spark$sql$connect$execution$ExecuteThreadRunner$$execute(ExecuteThreadRunner.scala:139)\n", + "\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner$ExecutionThread.$anonfun$run$2(ExecuteThreadRunner.scala:586)\n", + "\tat scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)\n", + "\tat com.databricks.unity.UCSEphemeralState$Handle.runWith(UCSEphemeralState.scala:51)\n", + "\tat com.databricks.unity.HandleImpl.runWith(UCSHandle.scala:104)\n", + "\tat com.databricks.unity.HandleImpl.$anonfun$runWithAndClose$1(UCSHandle.scala:109)\n", + "\tat scala.util.Using$.resource(Using.scala:296)\n", + "\tat com.databricks.unity.HandleImpl.runWithAndClose(UCSHandle.scala:108)\n", + "\tat org.apache.spark.sql.connect.execution.ExecuteThreadRunner$ExecutionThread.run(ExecuteThreadRunner.scala:586)\n", + "Foreign key setting complete.\n" ] }, { @@ -1440,7 +1692,47 @@ "name": "stdout", "output_type": "stream", "text": [ - "Found existing YAML for 'product_hierarchy'. Checking for staleness.\nproduct_hierarchy loaded\nFound existing YAML for 'product_feature'. Checking for staleness.\nproduct_feature loaded\nFound existing YAML for 'website'. Checking for staleness.\nwebsite loaded\nFound existing YAML for 'prob_statement_issue'. Checking for staleness.\nprob_statement_issue loaded\nFound existing YAML for 'customer_hierarchy'. Checking for staleness.\ncustomer_hierarchy loaded\nFound existing YAML for 'campaigns'. Checking for staleness.\ncampaigns loaded\nFound existing YAML for 'products'. Checking for staleness.\nproducts loaded\nFound existing YAML for 'service_requests'. Checking for staleness.\nservice_requests loaded\nFound existing YAML for 'logistics'. Checking for staleness.\nlogistics loaded\nFound existing YAML for 'customers'. Checking for staleness.\ncustomers loaded\nFound existing YAML for 'expense'. Checking for staleness.\nexpense loaded\nFound existing YAML for 'install_base'. Checking for staleness.\ninstall_base loaded\nFound existing YAML for 'campaign_survey'. Checking for staleness.\ncampaign_survey loaded\nFound existing YAML for 'nps_survey'. Checking for staleness.\nnps_survey loaded\nFound existing YAML for 'orders'. Checking for staleness.\norders loaded\nFound existing YAML for 'opportunity'. Checking for staleness.\nopportunity loaded\nFound existing YAML for 'renewals'. Checking for staleness.\nrenewals loaded\nFound existing YAML for 'returns'. Checking for staleness.\nreturns loaded\nFound existing YAML for 'delivery_survey'. Checking for staleness.\ndelivery_survey loaded\nFound existing YAML for 'inventory'. Checking for staleness.\ninventory loaded\ncustomer_kpi_summary loaded\n" + "Found existing YAML for 'product_hierarchy'. Checking for staleness.\n", + "product_hierarchy loaded\n", + "Found existing YAML for 'product_feature'. Checking for staleness.\n", + "product_feature loaded\n", + "Found existing YAML for 'website'. Checking for staleness.\n", + "website loaded\n", + "Found existing YAML for 'prob_statement_issue'. Checking for staleness.\n", + "prob_statement_issue loaded\n", + "Found existing YAML for 'customer_hierarchy'. Checking for staleness.\n", + "customer_hierarchy loaded\n", + "Found existing YAML for 'campaigns'. Checking for staleness.\n", + "campaigns loaded\n", + "Found existing YAML for 'products'. Checking for staleness.\n", + "products loaded\n", + "Found existing YAML for 'service_requests'. Checking for staleness.\n", + "service_requests loaded\n", + "Found existing YAML for 'logistics'. Checking for staleness.\n", + "logistics loaded\n", + "Found existing YAML for 'customers'. Checking for staleness.\n", + "customers loaded\n", + "Found existing YAML for 'expense'. Checking for staleness.\n", + "expense loaded\n", + "Found existing YAML for 'install_base'. Checking for staleness.\n", + "install_base loaded\n", + "Found existing YAML for 'campaign_survey'. Checking for staleness.\n", + "campaign_survey loaded\n", + "Found existing YAML for 'nps_survey'. Checking for staleness.\n", + "nps_survey loaded\n", + "Found existing YAML for 'orders'. Checking for staleness.\n", + "orders loaded\n", + "Found existing YAML for 'opportunity'. Checking for staleness.\n", + "opportunity loaded\n", + "Found existing YAML for 'renewals'. Checking for staleness.\n", + "renewals loaded\n", + "Found existing YAML for 'returns'. Checking for staleness.\n", + "returns loaded\n", + "Found existing YAML for 'delivery_survey'. Checking for staleness.\n", + "delivery_survey loaded\n", + "Found existing YAML for 'inventory'. Checking for staleness.\n", + "inventory loaded\n", + "customer_kpi_summary loaded\n" ] }, { @@ -1796,7 +2088,12 @@ "name": "stderr", "output_type": "stream", "text": [ - "I0000 00:00:1759658943.597447 27837 fork_posix.cc:75] Other threads are currently calling into gRPC, skipping fork() handlers\nI0000 00:00:1759658943.622177 27837 fork_posix.cc:75] Other threads are currently calling into gRPC, skipping fork() handlers\n/databricks/python/lib/python3.12/site-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.\n return _methods._mean(a, axis=axis, dtype=dtype,\n/databricks/python/lib/python3.12/site-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide\n ret = ret.dtype.type(ret / rcount)\n" + "I0000 00:00:1759658943.597447 27837 fork_posix.cc:75] Other threads are currently calling into gRPC, skipping fork() handlers\n", + "I0000 00:00:1759658943.622177 27837 fork_posix.cc:75] Other threads are currently calling into gRPC, skipping fork() handlers\n", + "/databricks/python/lib/python3.12/site-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.\n", + " return _methods._mean(a, axis=axis, dtype=dtype,\n", + "/databricks/python/lib/python3.12/site-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide\n", + " ret = ret.dtype.type(ret / rcount)\n" ] }, { @@ -2163,7 +2460,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "Syncing metadata to Databricks tables...\nMetadata sync complete.\n" + "Syncing metadata to Databricks tables...\n", + "Metadata sync complete.\n" ] }, { @@ -2279,4 +2577,4 @@ }, "nbformat": 4, "nbformat_minor": 0 -} \ No newline at end of file +} diff --git a/notebooks/quickstart_native_snowflake.ipynb b/notebooks/quickstart_native_snowflake.ipynb index b3a68eb..fc0662b 100644 --- a/notebooks/quickstart_native_snowflake.ipynb +++ b/notebooks/quickstart_native_snowflake.ipynb @@ -187,7 +187,7 @@ "4. **Glossary Generation:** Generates a business glossary for each column using an LLM.\n", "5. **Link Prediction:** Predicts the relationships (foreign keys) between tables.\n", "\n", - "Let's start by defining the datasets we want to use. The path shown below can be a local file path or a remote URL." + "Let's start by defining the datasets we want to use. " ] }, { diff --git a/notebooks/quickstart_sports_media.ipynb b/notebooks/quickstart_sports_media.ipynb index f2e89e9..34c41f1 100644 --- a/notebooks/quickstart_sports_media.ipynb +++ b/notebooks/quickstart_sports_media.ipynb @@ -80,6 +80,14 @@ "print(\"All CSV files downloaded successfully.\")" ] }, + { + "cell_type": "markdown", + "id": "1d7b58c2", + "metadata": {}, + "source": [ + "> **Working with Remote URLs** : While `intugle` supports providing remote URLs (e.g., a raw GitHub file link) in the `path`, it is **not recommended for large datasets**. For best performance and stability, we recommend downloading files to your local filesystem first and providing a local path." + ] + }, { "cell_type": "markdown", "id": "941ae2c6", diff --git a/notebooks/quickstart_tech_manufacturing.ipynb b/notebooks/quickstart_tech_manufacturing.ipynb index 238330c..339f8ca 100644 --- a/notebooks/quickstart_tech_manufacturing.ipynb +++ b/notebooks/quickstart_tech_manufacturing.ipynb @@ -80,6 +80,14 @@ "print(\"All CSV files downloaded successfully.\")" ] }, + { + "cell_type": "markdown", + "id": "c0fcd0de", + "metadata": {}, + "source": [ + "> **Working with Remote URLs** : While `intugle` supports providing remote URLs (e.g., a raw GitHub file link) in the `path`, it is **not recommended for large datasets**. For best performance and stability, we recommend downloading files to your local filesystem first and providing a local path." + ] + }, { "cell_type": "markdown", "id": "9a63283cbaf04dbcab1f6479b197f3a8",