Merge remote-tracking branch 'origin/main' into tests/text_splitter

Author: dayesouza

.semversioner/1.1.0.json

@@ -0,0 +1,58 @@
+{
+  "changes": [
+    {
+      "description": "Make gleanings independent of encoding",
+      "type": "minor"
+    },
+    {
+      "description": "Remove DataShaper (first steps).",
+      "type": "minor"
+    },
+    {
+      "description": "Remove old pipeline runner.",
+      "type": "minor"
+    },
+    {
+      "description": "new search implemented as a new option for the api",
+      "type": "minor"
+    },
+    {
+      "description": "Fix gleanings loop check",
+      "type": "patch"
+    },
+    {
+      "description": "Implement cosmosdb storage option for cache and output",
+      "type": "patch"
+    },
+    {
+      "description": "Move extractor code to co-locate with operations.",
+      "type": "patch"
+    },
+    {
+      "description": "Remove config input models.",
+      "type": "patch"
+    },
+    {
+      "description": "Ruff update",
+      "type": "patch"
+    },
+    {
+      "description": "Simplify and streamline internal config.",
+      "type": "patch"
+    },
+    {
+      "description": "Simplify callbacks model.",
+      "type": "patch"
+    },
+    {
+      "description": "Streamline flows.",
+      "type": "patch"
+    },
+    {
+      "description": "fix instantiation of storage classes.",
+      "type": "patch"
+    }
+  ],
+  "created_at": "2025-01-07T20:25:57+00:00",
+  "version": "1.1.0"
+}

.semversioner/1.1.1.json

@@ -0,0 +1,14 @@
+{
+  "changes": [
+    {
+      "description": "Fix a bug on creating community hierarchy for dynamic search",
+      "type": "patch"
+    },
+    {
+      "description": "Increase LOCAL_SEARCH_COMMUNITY_PROP to 15%",
+      "type": "patch"
+    }
+  ],
+  "created_at": "2025-01-08T21:53:16+00:00",
+  "version": "1.1.1"
+}

.semversioner/1.1.2.json

@@ -0,0 +1,10 @@
+{
+  "changes": [
+    {
+      "description": "Basic Rag minor fix",
+      "type": "patch"
+    }
+  ],
+  "created_at": "2025-01-09T22:29:23+00:00",
+  "version": "1.2.0"
+}

.semversioner/next-release/patch-20241121202210026640.json

@@ -0,0 +1,4 @@
+{
+  "type": "patch",
+  "description": "Set default rate limits."
+}

.semversioner/next-release/patch-20241212190223784600.json

@@ -1,6 +1,31 @@
 # Changelog
 Note: version releases in the 0.x.y range may introduce breaking changes.
 
+## 1.1.2
+
+- patch: Basic Rag minor fix
+
+## 1.1.1
+
+- patch: Fix a bug on creating community hierarchy for dynamic search
+- patch: Increase LOCAL_SEARCH_COMMUNITY_PROP to 15%
+
+## 1.1.0
+
+- minor: Make gleanings independent of encoding
+- minor: Remove DataShaper (first steps).
+- minor: Remove old pipeline runner.
+- minor: new search implemented as a new option for the api
+- patch: Fix gleanings loop check
+- patch: Implement cosmosdb storage option for cache and output
+- patch: Move extractor code to co-locate with operations.
+- patch: Remove config input models.
+- patch: Ruff update
+- patch: Simplify and streamline internal config.
+- patch: Simplify callbacks model.
+- patch: Streamline flows.
+- patch: fix instantiation of storage classes.
+
 ## 1.0.1
 
 - patch: Fix encoding model config parsing

.semversioner/next-release/patch-20241213181544864279.json

@@ -148,10 +148,6 @@ codebases
 # Microsoft
 MSRC
 
-# Broken Upstream
-# TODO FIX IN DATASHAPER
-Arrary
-
 # Prompt Inputs
 ABILA
 Abila

.semversioner/next-release/patch-20241224192900934104.json

@@ -51,6 +51,7 @@ Available scripts are:
 - `poetry run poe test_unit` - This will execute unit tests.
 - `poetry run poe test_integration` - This will execute integration tests.
 - `poetry run poe test_smoke` - This will execute smoke tests.
+- `poetry run poe test_verbs` - This will execute tests of the basic workflows.
 - `poetry run poe check` - This will perform a suite of static checks across the package, including:
   - formatting
   - documentation formatting

.semversioner/next-release/patch-20241227225850465466.json

@@ -206,9 +206,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from datashaper import NoopVerbCallbacks\n",
-    "\n",
     "from graphrag.cache.factory import create_cache\n",
+    "from graphrag.callbacks.noop_workflow_callbacks import NoopWorkflowCallbacks\n",
     "from graphrag.index.flows.generate_text_embeddings import generate_text_embeddings\n",
     "\n",
     "# We only need to re-run the embeddings workflow, to ensure that embeddings for all required search fields are in place\n",
@@ -220,7 +219,7 @@
     "config = workflow.config\n",
     "text_embed = config.get(\"text_embed\", {})\n",
     "embedded_fields = config.get(\"embedded_fields\", {})\n",
-    "callbacks = NoopVerbCallbacks()\n",
+    "callbacks = NoopWorkflowCallbacks()\n",
     "cache = create_cache(pipeline_config.cache, PROJECT_DIRECTORY)\n",
     "\n",
     "await generate_text_embeddings(\n",

.semversioner/next-release/patch-20250110173747052255.json

@@ -8,33 +8,9 @@ In order to support the GraphRAG system, the outputs of the indexing engine (in
 This model is designed to be an abstraction over the underlying data storage technology, and to provide a common interface for the GraphRAG system to interact with.
 In normal use-cases the outputs of the GraphRAG Indexer would be loaded into a database system, and the GraphRAG's Query Engine would interact with the database using the knowledge model data-store types.
 
-### DataShaper Workflows
-
-GraphRAG's Indexing Pipeline is built on top of our open-source library, [DataShaper](https://github.com/microsoft/datashaper).
-DataShaper is a data processing library that allows users to declaratively express data pipelines, schemas, and related assets using well-defined schemas.
-DataShaper has implementations in JavaScript and Python, and is designed to be extensible to other languages.
-
-One of the core resource types within DataShaper is a [Workflow](https://github.com/microsoft/datashaper/blob/main/javascript/schema/src/workflow/WorkflowSchema.ts).
-Workflows are expressed as sequences of steps, which we call [verbs](https://github.com/microsoft/datashaper/blob/main/javascript/schema/src/workflow/verbs.ts).
-Each step has a verb name and a configuration object.
-In DataShaper, these verbs model relational concepts such as SELECT, DROP, JOIN, etc.. Each verb transforms an input data table, and that table is passed down the pipeline.
-
-```mermaid
----
-title: Sample Workflow
----
-flowchart LR
-    input[Input Table] --> select[SELECT] --> join[JOIN] --> binarize[BINARIZE] --> output[Output Table]
-```
-
-### LLM-based Workflow Steps
-
-GraphRAG's Indexing Pipeline implements a handful of custom verbs on top of the standard, relational verbs that our DataShaper library provides. These verbs give us the ability to augment text documents with rich, structured data using the power of LLMs such as GPT-4. We utilize these verbs in our standard workflow to extract entities, relationships, claims, community structures, and community reports and summaries. This behavior is customizable and can be extended to support many kinds of AI-based data enrichment and extraction tasks.
-
-### Workflow Graphs
+### Workflows
 
 Because of the complexity of our data indexing tasks, we needed to be able to express our data pipeline as series of multiple, interdependent workflows.
-In the GraphRAG Indexing Pipeline, each workflow may define dependencies on other workflows, effectively forming a directed acyclic graph (DAG) of workflows, which is then used to schedule processing.
 
 ```mermaid
 ---
@@ -55,7 +31,7 @@ stateDiagram-v2
 The primary unit of communication between workflows, and between workflow steps is an instance of `pandas.DataFrame`.
 Although side-effects are possible, our goal is to be _data-centric_ and _table-centric_ in our approach to data processing.
 This allows us to easily reason about our data, and to leverage the power of dataframe-based ecosystems.
-Our underlying dataframe technology may change over time, but our primary goal is to support the DataShaper workflow schema while retaining single-machine ease of use and developer ergonomics.
+Our underlying dataframe technology may change over time, but our primary goal is to support the workflow schema while retaining single-machine ease of use and developer ergonomics.
 
 ### LLM Caching