Refactor/dataset structure

.gitignore

@@ -209,6 +209,7 @@ notes.txt
 
 testing_base
 /models
+/notebooks/models/
 models_bak
 
 settings.json

docsite/docs/core-concepts/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "Core Concepts",
+  "position": 3,
+  "link": {
+    "type": "generated-index",
+    "description": "Learn about the core concepts and architecture of the Intugle Data Tools."
+  }
+}

docsite/docs/core-concepts/dataset.md

@@ -0,0 +1,133 @@
+---
+sidebar_position: 3
+title: DataSet
+---
+
+# DataSet
+
+The `DataSet` class is the heart of the analysis pipeline. It acts as a powerful, in-memory container for a single data source, holding not just the raw data but also all the rich metadata that is generated about it during the `KnowledgeBuilder`'s workflow.
+
+## Overview
+
+Think of a `DataSet` as a "unit of work" that gets progressively enriched as it moves through the analysis stages. Its key responsibilities are:
+
+1.  **Data Abstraction**: It provides a consistent way to interact with data, regardless of its source.
+2.  **Metadata Storage**: It serves as the central object for storing all analysis results.
+3.  **Intelligent Caching**: It automatically persists and loads its own state, making the entire process efficient and resilient.
+
+## Key Features
+
+### Data Abstraction
+
+The `DataSet` uses a system of **Adapters** under the hood to connect to different data backends. When you initialize a `DataSet` with a file-based source, it uses the appropriate adapter (e.g., `DuckdbAdapter` for CSVs, Parquet, etc.) to handle the specific implementation details for profiling and querying the data. This makes the system easily extensible to support new data sources in the future.
+
+### Centralized Metadata
+
+All analysis results for a data source are stored within the `dataset.source_table_model` attribute. This attribute is a structured Pydantic model that makes accessing metadata predictable and easy.
+
+#### Metadata Structure
+
+The metadata is organized using the following Pydantic models:
+
+-   `SourceTables`: The root object, containing table-level information.
+    -   `name: str`
+    -   `description: str`
+    -   `key: Optional[str]`
+    -   `columns: List[Column]`
+    -   `profiling_metrics: Optional[ModelProfilingMetrics]`
+-   `Column`: Contains metadata for a single column.
+    -   `name: str`
+    -   `description: Optional[str]`
+    -   `type: Optional[str]` (e.g., 'integer', 'date')
+    -   `category: Literal["dimension", "measure"]`
+    -   `tags: Optional[List[str]]`
+    -   `profiling_metrics: Optional[ColumnProfilingMetrics]`
+-   `ColumnProfilingMetrics`: Detailed statistics for a column.
+    -   `count: Optional[int]`
+    -   `null_count: Optional[int]`
+    -   `distinct_count: Optional[int]`
+
+#### Accessing Metadata
+
+You can access this rich metadata directly from the `DataSet` object.
+
+```python
+# Assuming 'kb' is a built KnowledgeBuilder instance
+customers_dataset = kb.datasets['customers']
+
+# Access table-level metadata
+print(f"Table Name: {customers_dataset.source_table_model.name}")
+print(f"Table Description: {customers_dataset.source_table_model.description}")
+print(f"Primary Key: {customers_dataset.source_table_model.key}")
+
+# Access column-level metadata for the first column
+first_column = customers_dataset.source_table_model.columns[0]
+print(f"Column Name: {first_column.name}")
+print(f"Column Description: {first_column.description}")
+print(f"Column Type: {first_column.type}")
+
+# Access profiling metrics for that column
+metrics = first_column.profiling_metrics
+if metrics:
+    print(f"Distinct Count: {metrics.distinct_count}")
+```
+
+### Automatic Caching
+
+The `DataSet` object is designed to be efficient by avoiding redundant work. When you initialize a `DataSet`, it automatically checks for a corresponding `.yml` file. If it finds one, it validates that the source data file hasn't changed since the last run. If the data is fresh, it loads the saved metadata, saving significant processing time.
+
+### Analysis Stage Functions
+
+You can run the analysis pipeline step-by-step for more granular control. Each of these methods includes a `save=True` option to persist the results of that specific stage.
+
+```python
+from intugle import DataSet
+
+# Initialize the dataset
+data_source = {"path": "path/to/my_data.csv", "type": "csv"}
+dataset = DataSet(data_source, name="my_data")
+
+# Run each stage individually and save progress
+print("Step 1: Profiling...")
+dataset.profile(save=True)
+
+print("Step 2: Identifying Datatypes...")
+dataset.identify_datatypes(save=True)
+
+print("Step 3: Identifying Keys...")
+dataset.identify_keys(save=True)
+
+print("Step 4: Generating Glossary...")
+dataset.generate_glossary(domain="my_domain", save=True)
+```
+
+### Other Useful Methods and Properties
+
+#### `save_yaml()`
+
+Manually trigger a save of the dataset's current state to its `.yml` file.
+
+```python
+dataset.save_yaml()
+```
+
+#### `reload_from_yaml()`
+
+Force a reload from the YAML file, bypassing the staleness check. This can be useful if you manually edit the YAML and want to load your changes.
+
+```python
+file_path = "path/to/my_data.yml"
+dataset.reload_from_yaml(file_path)
+```
+
+#### `profiling_df`
+
+Access a Pandas DataFrame containing the complete profiling results for all columns in the dataset. This is very useful for exploration and validation in a notebook environment.
+
+```python
+# Get a comprehensive DataFrame of all column profiles
+profiles = dataset.profiling_df
+
+# Display the first 5 rows
+print(profiles.head())
+```

docsite/docs/core-concepts/index.md

@@ -0,0 +1,17 @@
+---
+sidebar_position: 1
+---
+
+# Core Concepts
+
+Welcome to the core concepts section. Here, we'll dive deep into the fundamental components and architectural ideas that power the Intugle Data Tools library.
+
+Understanding these concepts is key to using the library effectively and leveraging its full potential for automating data intelligence.
+
+### Key Components:
+
+*   [**Knowledge Builder**](./knowledge-builder.md): The main orchestrator that manages the end-to-end analysis pipeline.
+*   [**DataSet**](./dataset.md): The central in-memory container for a data source and all its rich metadata.
+*   [**Link Prediction**](./link-prediction.md): The process of automatically discovering relationships between your datasets.
+*   **Data Product Builder**: The tool used to generate unified data products from the enriched semantic layer.
+*   **Semantic Search**: The feature that enables natural language queries against your connected data assets.

docsite/docs/core-concepts/knowledge-builder.md

@@ -0,0 +1,135 @@
+---
+sidebar_position: 2
+title: The Knowledge Builder
+---
+
+# The Knowledge Builder
+
+The `KnowledgeBuilder` is the primary orchestrator of the data intelligence pipeline. It is the main user-facing class designed to manage multiple data sources and run the end-to-end process of transforming them from raw, disconnected tables into a fully enriched and interconnected semantic layer.
+
+## Overview
+
+At a high level, the `KnowledgeBuilder` is responsible for:
+
+1.  **Initializing and Managing Datasets**: It takes your raw data sources (e.g., file paths) and wraps each one in a `DataSet` object.
+2.  **Executing the Analysis Pipeline**: It runs a series of analysis stages in a specific, logical order to build up a rich understanding of your data.
+3.  **Ensuring Resilience**: The pipeline is designed to be modular and resilient. It automatically saves its progress after each major stage, allowing you to resume an interrupted run without losing completed work.
+
+## Initialization
+
+You can initialize the `KnowledgeBuilder` in two ways:
+
+1.  **With a Dictionary of File-Based Sources**: This is the most common method. You provide a dictionary where keys are the desired names for your datasets and values are dictionary configs pointing to your data.
+
+    ```python
+    from intugle import KnowledgeBuilder
+
+    data_sources = {
+        "customers": {"path": "path/to/customers.csv", "type": "csv"},
+        "orders": {"path": "path/to/orders.csv", "type": "csv"},
+    }
+
+    kb = KnowledgeBuilder(data_input=data_sources, domain="e-commerce")
+    ```
+
+2.  **With a List of `DataSet` Objects**: If you have already created `DataSet` objects, you can pass a list of them directly.
+
+    ```python
+    from intugle import KnowledgeBuilder, DataSet
+
+    # Create DataSet objects from file-based sources
+    customers_data = {"path": "path/to/customers.csv", "type": "csv"}
+    orders_data = {"path": "path/to/orders.csv", "type": "csv"}
+
+    dataset_one = DataSet(customers_data, name="customers")
+    dataset_two = DataSet(orders_data, name="orders")
+
+    datasets = [dataset_one, dataset_two]
+
+    kb = KnowledgeBuilder(data_input=datasets, domain="e-commerce")
+    ```
+
+The `domain` parameter is an optional but highly recommended string that provides context to the underlying AI models, helping them generate more accurate and relevant business glossary terms.
+
+## The Analysis Pipeline
+
+The `KnowledgeBuilder` executes its workflow in distinct, modular stages. This allows for greater control and makes the process resilient to interruptions.
+
+### 1. `profile()`
+
+This is the first and most foundational stage. It performs a deep analysis of each dataset to understand its structure and content, covering profiling, datatype identification, and key identification.
+
+```python
+# Run only the profiling and key identification stage
+kb.profile()
+```
+
+Progress from this stage is automatically saved to a `.yml` file for each dataset.
+
+### 2. `predict_links()`
+
+Once the datasets are profiled, this stage uses the `LinkPredictor` to analyze the metadata from all datasets and discover potential relationships between them. You can learn more about the [Link Prediction](./link-prediction.md) process in its dedicated section.
+
+```python
+# Run the link prediction stage
+# This assumes profile() has already been run
+kb.predict_links()
+
+# Access the links via the `links` attribute, which is a shortcut
+discovered_links = kb.links
+print(discovered_links)
+
+# You can also access the full LinkPredictor instance for more options
+# See the section below for more details.
+```
+
+The discovered relationships are saved to a central `__relationships__.yml` file.
+
+### 3. `generate_glossary()`
+
+In the final stage, the `KnowledgeBuilder` uses a Large Language Model (LLM) to generate business-friendly context for your data.
+
+```python
+# Run the glossary generation stage
+# This assumes profile() has already been run
+kb.generate_glossary()
+```
+
+This information is saved back into each dataset's `.yml` file.
+
+### The `build()` Method
+
+For convenience, the `build()` method runs all three stages (`profile`, `predict_links`, `generate_glossary`) in the correct sequence.
+
+```python
+# Run the full pipeline from start to finish
+kb.build()
+
+# You can also force it to re-run everything, ignoring any cached results
+kb.build(force_recreate=True)
+```
+
+This modular design means that if your process is interrupted during the `generate_glossary` stage, you can simply re-run `kb.build()`, and it will quickly skip the already-completed stages, picking up right where it left off.
+
+## Accessing Processed Datasets and Predictor
+
+After running any stage of the pipeline, you can easily access the enriched `DataSet` objects and the `LinkPredictor` instance to explore the results programmatically.
+
+```python
+# Run the full build
+kb.build()
+
+# Access the 'customers' dataset
+customers_dataset = kb.datasets['customers']
+
+# Access the LinkPredictor instance
+link_predictor = kb.link_predictor
+
+# Now you can explore rich metadata or results
+print(f"Primary Key for customers: {customers_dataset.source_table_model.description}")
+print("Discovered Links:")
+print(link_predictor.get_links_df())
+
+```
+Learn more about what you can do with these objects. See the [DataSet](./dataset.md) and [Link Prediction](./link-prediction.md) documentation.
+