-
-
-### Connecting to your cluster
-
-Depending on your external IP address, you may need to use different strategies to connect to your cluster.
-
-If the external IP for both services is `127.0.0.1`, connect to Weaviate with:
-
-- Another way to confirm access to the gRPC service -
- -If you have netcat installed, you can also try: - -```bash -nc -zv 127.0.0.1 50051 -``` - -Which will show: - -```bash -Connection to 127.0.0.1 port 50051 [tcp/*] succeeded! -``` - -Note that not all systems have `nc` installed by default. It's okay if you don't have it - the `kubectl get svc` command output is sufficient to confirm access to the gRPC service. - -
-
-## Install Node.js
-
-### Is Node.js installed?
-
-Open a terminal window (e.g. bash, zsh, Windows PowerShell, Windows Terminal), and run:
-
-```shell
-node --version # or node -v
-```
-
-If you have Node.js installed, you should see a response like `v22.3.0`. The minimum version of Node.js supported by the Weaviate Typescript/Javascript library is `v18`.
-
-
-### Install Node.js
-
-To install, follow the instructions for your system on [nodejs.org](https://nodejs.org/en/download/package-manager).
-
-Once you have Node.js installed, check the version again to confirm that you have a recommended version installed.
-
-:::tip Advanced option: `nvm`
-Another good way to install Python is to install `nvm`. This will allow you to manage multiple versions of Node.js on your system. You can find instructions on how to install `nvm` [here](https://github.com/nvm-sh/nvm?tab=readme-ov-file#installing-and-updating).
-:::
-
-
-### (Optional) Set up Typescript
-
-To install, follow the instructions for your system on [typescriptlang.org](https://www.typescriptlang.org/download/). Once installed, you can find instruction on how to configure Typescript to work with the Weaviate client documented [here](../../weaviate/client-libraries/typescript/index.mdx).
-
-Now Typescript is ready to use with the Weaviate client.
-
-## Install the Weaviate client
-
-Now, you can install the [Weaviate client library](../../weaviate/client-libraries/index.mdx), which will make it much easier to interact with Weaviate using Typescript.
-
-In a new folder for your project, install the Weaviate client with:
-
-```shell
-npm install weaviate-client
-```
-
-### Confirm the installation
-
-To confirm that the Weaviate client is installed, run the following in your terminal:
-
-```shell
-npm view weaviate-client
-```
-
-You should see an output like:
-
-```text
-3.1.4.
-```
-
-Congratulations, you are now set up to use Weaviate with the Weaviate TypeScript/JavaScript client library!
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-
-
-This section aims to guide you through how to do that as you build applications with the Weaviate Typescript v3 client: [weaviate-client](https://www.npmjs.com/package/weaviate-client).
-
-
-
-### Using the weaviate-client in a client-server application
-
-The v3 client uses gRPC to connect to your Weaviate instance. The client supports Node.js, server-based development. It does not support browser-based web client development.
-
-Install the client by following [these instructions](../../../../weaviate/client-libraries/typescript/index.mdx#installation). A big benefit of using the new v3 client is the introduction of the gRPC protocol. A faster, more reliable platform to handle interactions with the database at scale. Unfortunately, gRPC does not support browser-based client development.
-
-Besides the requirements of running the [weaviate-client](../../../../weaviate/client-libraries/typescript/index.mdx), the client-server architecture is reliably more secure than interactions directly from the client.
-
-Having a client-server approach means you can optimize your use of Weaviate by implementing load balancing, user and session management, middleware and various other optimizations.
-
-In the next sections we'll look at how to build client-server applications with..
-- [Using backend web frameworks](./20_building-client-server.mdx)
-- [Using fullstack web frameworks](./30_fullstack.mdx)
-
diff --git a/docs/academy/js/standalone/client-server/20_building-client-server.mdx b/docs/academy/js/standalone/client-server/20_building-client-server.mdx
deleted file mode 100644
index 3e6917ed1..000000000
--- a/docs/academy/js/standalone/client-server/20_building-client-server.mdx
+++ /dev/null
@@ -1,133 +0,0 @@
----
-title: Using Backend Web frameworks
----
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-import FilteredTextBlock from '@site/src/components/Documentation/FilteredTextBlock';
-import TSCode from '!!raw-loader!./_snippets/20_backend.js';
-import ClientCode from '!!raw-loader!./_snippets/index.html';
-import WeaviateTypescriptImgUrl from '/docs/academy/js/standalone/client-server/_img/backend.jpg';
-
-
-
-This approach involves having two separate tools. One to build your server application; ideally a backend framework and another to build your client application. For this example, we will be using [Express.js](https://expressjs.com/en/starter/hello-world.html) to build a backend server, and [Thunder Client](https://www.thunderclient.com/) to act as a client and make API calls to our backend server.
-
-
-
-
-## Building a server
-
-The server will have a single route that accepts a `searchTerm` as a query parameter.
-
-### 1. Initialize a Node.js application
-
-We will use Express to build our server, in a new directory, run the following command to initialize a new project with Node.js
-
-```bash
-npm init
-```
-### 2. Install project dependencies
-
-With our project initialized, install `dotenv` to manage environment variables, `express` to build our server and the `weaviate-client` to manage communication with our Weaviate database.
-
-```bash
-npm install express dotenv weaviate-client
-```
-
-
-### 3. Setup your Weaviate database
-
-We'll start by creating a free sandbox account on [Weaviate Cloud](https://console.weaviate.cloud/). Follow [this guide](/cloud/manage-clusters/connect) if you have trouble setting up a sandbox project.
-
-
-You will need your Weaviate cluster URL and API key. If you don't already have one, create a new Cohere [API key](https://dashboard.cohere.com/api-keys), we use Cohere as our [embedding model](../using-ml-models/10_embedding.mdx). When done, add all three to your `.env` file.
-
-
-
-
-## Building with Next.js
-
-### 1. Create a Next.js application
-
-To create a new application with Next.js, run the following command in your terminal.
-
-```bash
-create-next-app - { data } -
- - - ) - -} - -``` -#### 6. Run your Fullstack App - -In your terminal, run the following command to start your application. - - -```bash -npm run dev -``` - -Your application should be running on `localhost:3000`. - -## Other frameworks - - -Although only detailing Next.js in guide, you can build with Weaviate using a number of fullstack frameworks including but not limited to [Nuxt](https://nuxt.com/), [Solid](https://www.solidjs.com/) and [Angular](https://angular.dev/) - -We have a list of [starter](https://github.com/topics/weaviate-starter) applications you can play around with as well. - - - - diff --git a/docs/academy/js/standalone/client-server/_img/architecture.jpg b/docs/academy/js/standalone/client-server/_img/architecture.jpg deleted file mode 100644 index 9a8ddb7cf..000000000 Binary files a/docs/academy/js/standalone/client-server/_img/architecture.jpg and /dev/null differ diff --git a/docs/academy/js/standalone/client-server/_img/backend.jpg b/docs/academy/js/standalone/client-server/_img/backend.jpg deleted file mode 100644 index c5fe962ce..000000000 Binary files a/docs/academy/js/standalone/client-server/_img/backend.jpg and /dev/null differ diff --git a/docs/academy/js/standalone/client-server/_img/fullstack.jpg b/docs/academy/js/standalone/client-server/_img/fullstack.jpg deleted file mode 100644 index 417f0d518..000000000 Binary files a/docs/academy/js/standalone/client-server/_img/fullstack.jpg and /dev/null differ diff --git a/docs/academy/js/standalone/client-server/_snippets/20_backend.js b/docs/academy/js/standalone/client-server/_snippets/20_backend.js deleted file mode 100644 index fdac27ebb..000000000 --- a/docs/academy/js/standalone/client-server/_snippets/20_backend.js +++ /dev/null @@ -1,62 +0,0 @@ -// START weaviate.js -import weaviate from 'weaviate-client' -import 'dotenv/config'; - -export const connectToDB = async () => { - try { - const client = await weaviate.connectToWeaviateCloud(process.env.WEAVIATE_URL,{ - authCredentials: new weaviate.ApiKey(process.env.WEAVIATE_API_KEY), - headers: { - 'X-Cohere-Api-Key': process.env.COHERE_API_KEY || '', - } - } - ) - console.log(`We are connected! ${await client.isReady()}`); - return client - } catch (error) { - console.error(`Error: ${error.message}`); - process.exit(1); - } -}; - -// END weaviate.js -const dotEnv = ``` -// START .env -COHERE_API_KEY= -WEAVIATE_URL= -WEAVIATE_API_KEY= -// END .env -``` - -// START app.js -import express from 'express'; -import { connectToDB } from './config/weaviate.js'; - -const app = express(); -const port = 3005 - -const client = await connectToDB(); - -app.get('/', async function(req, res, next) { - var searchTerm = req.query.searchTerm; - - const wikipedia = client.collections.use("Wikipedia") - - try { - const response = await wikipedia.query.nearText(searchTerm, { - limit: 3 - }) - - res.send(response.objects) - } catch (error) { - console.error(`Error: ${error.message}`); - } - }) - -app.listen(port, () => { - console.log(`App listening on port ${port}`) -}) - - - -// END app.js diff --git a/docs/academy/js/standalone/client-server/_snippets/30_fullstack.js b/docs/academy/js/standalone/client-server/_snippets/30_fullstack.js deleted file mode 100644 index 2458d472a..000000000 --- a/docs/academy/js/standalone/client-server/_snippets/30_fullstack.js +++ /dev/null @@ -1,67 +0,0 @@ -// START weaviate.js - -import weaviate from 'weaviate-client' -import 'dotenv/config'; - -export const connectToDB = async () => { - try { - const client = await weaviate.connectToWeaviateCloud(process.env.WEAVIATE_URL,{ - authCredentials: new weaviate.ApiKey(process.env.WEAVIATE_API_KEY), - headers: { - 'X-Cohere-Api-Key': process.env.COHERE_API_KEY || '', - } - } - ) - console.log(`We are connected! ${await client.isReady()}`); - return client - } catch (error) { - console.error(`Error: ${error.message}`); - process.exit(1); - } -}; - -// END weaviate.js - -// .env -` -COHERE_API_KEY= -WEAVIATE_URL= -WEAVIATE_API_KEY= -` -// END .env - - -// START app.js -import express from 'express'; -import { connectToDB } from './config/weaviate.js'; - -const app = express(); -const port = 3005 - -const client = await connectToDB(); - - - -app.get('/', async function(req, res, next) { - var searchTerm = req.query.searchTerm; - - const wikipedia = client.collections.use("Wikipedia") - - try { - const response = await wikipedia.query.nearText(searchTerm, { - limit: 5 - }) - - res.send(response.objects) - } catch (error) { - console.error(`Error: ${error.message}`); - } - }) - -app.listen(port, () => { - console.log(`App listening on port ${port}`) -}) - - - -// END app.js diff --git a/docs/academy/js/standalone/client-server/_snippets/index.html b/docs/academy/js/standalone/client-server/_snippets/index.html deleted file mode 100644 index 0819a2dfe..000000000 --- a/docs/academy/js/standalone/client-server/_snippets/index.html +++ /dev/null @@ -1,83 +0,0 @@ - - - - -
-
-
-
-
-
-
\ No newline at end of file
diff --git a/docs/academy/js/standalone/client-server/index.md b/docs/academy/js/standalone/client-server/index.md
deleted file mode 100644
index 55bc75cc1..000000000
--- a/docs/academy/js/standalone/client-server/index.md
+++ /dev/null
@@ -1,44 +0,0 @@
----
-title: Building client-server Applications
----
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-import FilteredTextBlock from '@site/src/components/Documentation/FilteredTextBlock';
-
-
-## Overview
-
-When building web applications in JavaScript with Weaviate using the [weaviate-client](https://www.npmjs.com/package/weaviate-client), it is recommended that you employ the client-server architecture.
-
-This may vary depending what tools you are using to build your web application.
-
-Fullstack frameworks like Next.js have support for server side development and API creation to communicate with Weaviate. This would happen via REST calls or for Next.js specifically, Server functions. This approach means coupling your client and server applications.
-
-Backend web frameworks like Express let you create an API to communicate with Weaviate. This API can be consumed via REST calls from your client application. This approach means completely decoupling your client and server applications.
-
-
-### Prerequisites
-
-- A Node.js environment with `weaviate-client` installed.
-- Familiarity with Weaviate's search capabilities.
-- Some experience building Modern Web Applications with JavaScript.
-- Intermediate coding proficiency (e.g. JavaScript).
-
-## Learning objectives
-
-import LearningGoalsExp from '/src/components/Academy/learningGoalsExp.mdx';
-
-Semantic Search Results
- -Loading...
-
-
-
-
-Generative Models encompass so many types of models, we will specifically focus on large language models (LLMs).
-
-## When to use Generative Models
-
-Generative models are stars in the limelight of retrieval augmented generation (RAG) and agentic workflows. They are great for...
-
-- **Translation:** Models can perform zero-shot translate text from one language to another with extremely high accuracy.
-- **Code Generation:** Models can take high-level instructions and turn them into functional custom code.
-- **Image Generation:** Models can consistently generate high quality images from text instructions in a prompt.
-
-
-## Applications of Generative Models
-
-
-Large Language Models (LLMs), like [Claude](https://www.anthropic.com/claude) family by Anthropic or [Gemini](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference) by Google are specialized types of generative models focused on text data. These models, like most machine learning models are typically limited to one or more modalities.
-
-We use modality to describe the type of input or output that a machine learning model can process or interact with to run. Typically, generative modals fall into two buckets, uni-modal or multimodal.
-
-
-- **Uni-modal Generation:** In the context on LLMs, uni-modal generation defines a models ability to generate content and receive instructions in a single modality, this modality is usually text.
-
-
- Example
-
-
-
-### Close the connection
-
-After you have finished using the Weaviate client, you should close the connection. This frees up resources and ensures that the connection is properly closed.
-
-We suggest using a `try`-`finally` block as a best practice. For brevity, we will not include the `try`-`finally` blocks in the remaining code snippets.
-
-Example getMeta() output
-
-
-
-
-Next, you will create a corresponding object collection and import the data.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-See sample text data
- -| | backdrop_path | genre_ids | id | original_language | original_title | overview | popularity | poster_path | release_date | title | video | vote_average | vote_count | -|---:|:---------------------------------|:----------------|-----:|:--------------------|:----------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------:|:---------------------------------|:---------------|:----------------------------|:--------|---------------:|-------------:| -| 0 | /3Nn5BOM1EVw1IYrv6MsbOS6N1Ol.jpg | [14, 18, 10749] | 162 | en | Edward Scissorhands | A small suburban town receives a visit from a castaway unfinished science experiment named Edward. | 45.694 | /1RFIbuW9Z3eN9Oxw2KaQG5DfLmD.jpg | 1990-12-07 | Edward Scissorhands | False | 7.7 | 12305 | -| 1 | /sw7mordbZxgITU877yTpZCud90M.jpg | [18, 80] | 769 | en | GoodFellas | The true story of Henry Hill, a half-Irish, half-Sicilian Brooklyn kid who is adopted by neighbourhood gangsters at an early age and climbs the ranks of a Mafia family under the guidance of Jimmy Conway. | 57.228 | /aKuFiU82s5ISJpGZp7YkIr3kCUd.jpg | 1990-09-12 | GoodFellas | False | 8.5 | 12106 | -| 2 | /6uLhSLXzB1ooJ3522ydrBZ2Hh0W.jpg | [35, 10751] | 771 | en | Home Alone | Eight-year-old Kevin McCallister makes the most of the situation after his family unwittingly leaves him behind when they go on Christmas vacation. But when a pair of bungling burglars set their sights on Kevin's house, the plucky kid stands ready to defend his territory. By planting booby traps galore, adorably mischievous Kevin stands his ground as his frantic mother attempts to race home before Christmas Day. | 3.538 | /onTSipZ8R3bliBdKfPtsDuHTdlL.jpg | 1990-11-16 | Home Alone | False | 7.4 | 10599 | -| 3 | /vKp3NvqBkcjHkCHSGi6EbcP7g4J.jpg | [12, 35, 878] | 196 | en | Back to the Future Part III | The final installment of the Back to the Future trilogy finds Marty digging the trusty DeLorean out of a mineshaft and looking for Doc in the Wild West of 1885. But when their time machine breaks down, the travelers are stranded in a land of spurs. More problems arise when Doc falls for pretty schoolteacher Clara Clayton, and Marty tangles with Buford Tannen. | 28.896 | /crzoVQnMzIrRfHtQw0tLBirNfVg.jpg | 1990-05-25 | Back to the Future Part III | False | 7.5 | 9918 | -| 4 | /3tuWpnCTe14zZZPt6sI1W9ByOXx.jpg | [35, 10749] | 114 | en | Pretty Woman | When a millionaire wheeler-dealer enters a business contract with a Hollywood hooker Vivian Ward, he loses his heart in the bargain. | 97.953 | /hVHUfT801LQATGd26VPzhorIYza.jpg | 1990-03-23 | Pretty Woman | False | 7.5 | 7671 | - -
-
-
-Query image
- - - -
- 
-
-
-
-
-
-Weaviate output:
-
-```text
-Interstellar 2014 157336
-Distance to query: 0.354
-
-Gravity 2013 49047
-Distance to query: 0.384
-
-Arrival 2016 329865
-Distance to query: 0.386
-
-Armageddon 1998 95
-Distance to query: 0.400
-
-Godzilla 1998 929
-Distance to query: 0.441
-```
-
-
-
-### Response object
-
-The returned object is an instance of a custom class. Its `objects` attribute is a list of search results, each object being an instance of another custom class.
-
-Each returned object will:
-- Include all properties and its UUID by default except those with blob data types.
- - Since the `poster` property is a blob, it is not included by default.
- - To include the `poster` property, you must specify it and the other properties to fetch in the `returnProperties` parameter.
-- Not include any other information (e.g. references, metadata, vectors.) by default.
-
-
-## Text search
-
-### Code
-
-This example finds entries in "Movie" based on their similarity to the query "red", and prints out the title and release year of the top 5 matches.
-
-Example results
- -Posters for the top 5 matches: -
-
-
-
-
-
-Weaviate output:
-
-```text
-Interstellar 2014 157336
-Distance to query: 0.354
-
-Gravity 2013 49047
-Distance to query: 0.384
-
-Arrival 2016 329865
-Distance to query: 0.386
-
-Armageddon 1998 95
-Distance to query: 0.400
-
-Godzilla 1998 929
-Distance to query: 0.441
-```
-
-
- 
-
-
-
-
-
-Weaviate output:
-
-```text
-Deadpool 2 2018 383498
-Distance to query: 0.670
-
-Bloodshot 2020 338762
-Distance to query: 0.677
-
-Deadpool 2016 293660
-Distance to query: 0.678
-
-300 2007 1271
-Distance to query: 0.682
-
-The Hunt for Red October 1990 1669
-Distance to query: 0.683
-```
-
-
-
-### Response object
-
-The returned object is in the same format as in the previous example.
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -Posters for the top 5 matches: -
-
-
-
-
-
-Weaviate output:
-
-```text
-Deadpool 2 2018 383498
-Distance to query: 0.670
-
-Bloodshot 2020 338762
-Distance to query: 0.677
-
-Deadpool 2016 293660
-Distance to query: 0.678
-
-300 2007 1271
-Distance to query: 0.682
-
-The Hunt for Red October 1990 1669
-Distance to query: 0.683
-```
-
-
-
-
-
-## Hybrid search
-
-### Code
-
-This example finds entries in "Movie" with the highest hybrid search scores for the term "history", and prints out the title and release year of the top 5 matches.
-
-Example results
- -```text -American History X 1998 -BM25 score: 2.707 - -A Beautiful Mind 2001 -BM25 score: 1.896 - -Legends of the Fall 1994 -BM25 score: 1.663 - -Hacksaw Ridge 2016 -BM25 score: 1.554 - -Night at the Museum 2006 -BM25 score: 1.529 -``` - -
-
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Legends of the Fall 1994 -Hybrid score: 0.016 - -Hacksaw Ridge 2016 -Hybrid score: 0.016 - -A Beautiful Mind 2001 -Hybrid score: 0.015 - -The Butterfly Effect 2004 -Hybrid score: 0.015 - -Night at the Museum 2006 -Hybrid score: 0.012 -``` - -
-
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Dune 2021 -Distance to query: 0.199 - -Tenet 2020 -Distance to query: 0.200 - -Mission: Impossible - Dead Reckoning Part One 2023 -Distance to query: 0.207 - -Onward 2020 -Distance to query: 0.214 - -Jurassic World Dominion 2022 -Distance to query: 0.216 -``` - -
-
-
-### Response object
-
-Each response object is similar to that from a regular search query, with an additional `generated` attribute. This attribute will contain the generated output for each object.
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Interstellar -Interstellaire -Gravity -Gravité -Arrival -Arrivée -Armageddon -Armageddon -Godzilla -Godzilla -``` - -
-
-
-### Optional parameters
-
-You can also pass on a list of properties to be used, as the `groupedProperties` parameter. This can be useful to reduce the amount of data passed on to the large language model and omit irrelevant properties.
-
-### Response object
-
-A RAG query with the `groupedTask` parameter will return a response with an additional `generated` attribute. This attribute will contain the generated output for the set of objects.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Interstellar -Gravity -Arrival -Armageddon -Godzilla -These movies all involve space exploration, extraterrestrial beings, or catastrophic events threatening Earth. They all deal with themes of survival, human ingenuity, and the unknown mysteries of the universe. -``` - -
- Example
-
-
-
-### Close the connection
-
-After you have finished using the Weaviate client, you should close the connection. This frees up resources and ensures that the connection is properly closed.
-
-We suggest using a `try`-`finally` block as a best practice. For brevity, we will not include the `try`-`finally` blocks in the remaining code snippets.
-
-Example getMeta() output
-
-
-
-
-Next, you will create a corresponding object collection and import the data.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-See sample data
- -| | backdrop_path | genre_ids | id | original_language | original_title | overview | popularity | poster_path | release_date | title | video | vote_average | vote_count | -|---:|:---------------------------------|:----------------|-----:|:--------------------|:----------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------:|:---------------------------------|:---------------|:----------------------------|:--------|---------------:|-------------:| -| 0 | /3Nn5BOM1EVw1IYrv6MsbOS6N1Ol.jpg | [14, 18, 10749] | 162 | en | Edward Scissorhands | A small suburban town receives a visit from a castaway unfinished science experiment named Edward. | 45.694 | /1RFIbuW9Z3eN9Oxw2KaQG5DfLmD.jpg | 1990-12-07 | Edward Scissorhands | False | 7.7 | 12305 | -| 1 | /sw7mordbZxgITU877yTpZCud90M.jpg | [18, 80] | 769 | en | GoodFellas | The true story of Henry Hill, a half-Irish, half-Sicilian Brooklyn kid who is adopted by neighbourhood gangsters at an early age and climbs the ranks of a Mafia family under the guidance of Jimmy Conway. | 57.228 | /aKuFiU82s5ISJpGZp7YkIr3kCUd.jpg | 1990-09-12 | GoodFellas | False | 8.5 | 12106 | -| 2 | /6uLhSLXzB1ooJ3522ydrBZ2Hh0W.jpg | [35, 10751] | 771 | en | Home Alone | Eight-year-old Kevin McCallister makes the most of the situation after his family unwittingly leaves him behind when they go on Christmas vacation. But when a pair of bungling burglars set their sights on Kevin's house, the plucky kid stands ready to defend his territory. By planting booby traps galore, adorably mischievous Kevin stands his ground as his frantic mother attempts to race home before Christmas Day. | 3.538 | /onTSipZ8R3bliBdKfPtsDuHTdlL.jpg | 1990-11-16 | Home Alone | False | 7.4 | 10599 | -| 3 | /vKp3NvqBkcjHkCHSGi6EbcP7g4J.jpg | [12, 35, 878] | 196 | en | Back to the Future Part III | The final installment of the Back to the Future trilogy finds Marty digging the trusty DeLorean out of a mineshaft and looking for Doc in the Wild West of 1885. But when their time machine breaks down, the travelers are stranded in a land of spurs. More problems arise when Doc falls for pretty schoolteacher Clara Clayton, and Marty tangles with Buford Tannen. | 28.896 | /crzoVQnMzIrRfHtQw0tLBirNfVg.jpg | 1990-05-25 | Back to the Future Part III | False | 7.5 | 9918 | -| 4 | /3tuWpnCTe14zZZPt6sI1W9ByOXx.jpg | [35, 10749] | 114 | en | Pretty Woman | When a millionaire wheeler-dealer enters a business contract with a Hollywood hooker Vivian Ward, he loses his heart in the bargain. | 97.953 | /hVHUfT801LQATGd26VPzhorIYza.jpg | 1990-03-23 | Pretty Woman | False | 7.5 | 7671 | - -
-
-
-### Response object
-
-The returned object is an instance of a custom class. Its `objects` attribute is a list of search results, each object being an instance of another custom class.
-
-Each returned object will:
-- Include all properties and its UUID by default except those with blob data types.
-- Not include any other information (e.g. references, metadata, vectors.) by default.
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -In Time 2011 -Distance to query: 0.179 - -Gattaca 1997 -Distance to query: 0.180 - -I, Robot 2004 -Distance to query: 0.182 - -Mad Max: Fury Road 2015 -Distance to query: 0.190 - -The Maze Runner 2014 -Distance to query: 0.193 -``` - -
-
-
-
-## Hybrid search
-
-### Code
-
-This example finds entries in "Movie" with the highest hybrid search scores for the term "history", and prints out the title and release year of the top 5 matches.
-
-Example results
- -```text -American History X 1998 -BM25 score: 2.707 - -A Beautiful Mind 2001 -BM25 score: 1.896 - -Legends of the Fall 1994 -BM25 score: 1.663 - -Hacksaw Ridge 2016 -BM25 score: 1.554 - -Night at the Museum 2006 -BM25 score: 1.529 -``` - -
-
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Legends of the Fall 1994 -Hybrid score: 0.016 - -Hacksaw Ridge 2016 -Hybrid score: 0.016 - -A Beautiful Mind 2001 -Hybrid score: 0.015 - -The Butterfly Effect 2004 -Hybrid score: 0.015 - -Night at the Museum 2006 -Hybrid score: 0.012 -``` - -
-
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Dune 2021 -Distance to query: 0.199 - -Tenet 2020 -Distance to query: 0.200 - -Mission: Impossible - Dead Reckoning Part One 2023 -Distance to query: 0.207 - -Onward 2020 -Distance to query: 0.214 - -Jurassic World Dominion 2022 -Distance to query: 0.216 -``` - -
-
-
-### Response object
-
-Each response object is similar to that from a regular search query, with an additional `generated` attribute. This attribute will contain the generated output for each object.
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -In Time -À temps -Looper -Boucleur -I, Robot -Je, Robot -The Matrix -La Matrice -Children of Men -Les enfants des hommes -``` - -
-
-
-### Optional parameters
-
-You can also pass on a list of properties to be used, as the `groupedProperties` parameter. This can be useful to reduce the amount of data passed on to the large language model and omit irrelevant properties.
-
-### Response object
-
-A RAG query with the `groupedTask` parameter will return a response with an additional `generated` attribute. This attribute will contain the generated output for the set of objects.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -In Time -Looper -I, Robot -The Matrix -Children of Men -These movies all involve futuristic settings and explore themes related to the manipulation of time, technology, and the potential consequences of advancements in society. They also touch on issues such as inequality, control, and the impact of human actions on the future of humanity. -``` - -
-
-## Install Python
-
-### Is Python installed?
-
-Open a terminal window (e.g. bash, zsh, Windows PowerShell, Windows Terminal), and run:
-
-```shell
-python --version
-```
-
-If that did not work, you may need to use `python3` instead of `python`:
-
-```shell
-python3 --version
-```
-
-If you have Python installed, you should see a response like `Python 3.11.8`. If you have Python 3.8 or higher installed, you can skip the remainder of this section.
-
-### Install Python
-
-To install, follow the instructions for your system on [Python.org](https://www.python.org/downloads/).
-
-Once you have Python installed, check the version again to confirm that you have a recommended version installed.
-
-:::tip Advanced option: `pyenv`
-Another good way to install Python is to install `pyenv`. This will allow you to manage multiple versions of Python on your system. You can find instructions on how to install `pyenv` [here](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation).
-:::
-
-## Set up a virtual environment
-
-A virtual environment allows you to isolate various Python projects from each other. This is useful because it allows you to install dependencies for each project without affecting the others.
-
-### Create a virtual environment
-
-We recommend using `venv` to create a virtual environment. Navigate to your project directory (e.g. `cd PATH/TO/PROJECT`), and run:
-
-```shell
-python -m venv .venv
-```
-
-Or, if `python3` is your Python command:
-
-```shell
-python3 -m venv .venv
-```
-
-This will create a virtual environment in a directory called `.venv` in your project directory.
-
-### Activate the virtual environment
-
-Each virtual environment can be 'activated' and 'deactivated'. When activated, the Python commands you run will use the Python version and libraries installed in the virtual environment.
-
-To activate the virtual environment, go to your project directory and run:
-
-```shell
-source .venv/bin/activate
-```
-
-Or, if you are using Windows:
-
-- -Additionally, there are many other environment management tools available, such as `conda`, `pipenv`, and `poetry`. If you are already using one of these tools, you can use them instead of `venv`. -::: - -## Install the Weaviate client - -Now, you can install the [Weaviate client library](../../weaviate/client-libraries/index.mdx), which will make it much easier to interact with Weaviate using Python. - -[Activate your virtual environment](#-activate-the-virtual-environment), then install the Weaviate client with: - -```shell -pip install weaviate-client -``` - -### Confirm the installation - -To confirm that the Weaviate client is installed, run the following Python code: - -
- -If you are using an earlier version of Weaviate, or have asynchronous indexing disabled, you will need to use a different configuration. Please refer to the [PQ configuration documentation](/weaviate/configuration/compression/pq-compression.md#manually-configure-pq) for more information. -::: - -## Customize PQ - -Many PQ parameters are configurable. While the default settings are suitable for many use cases, you may want to customize the PQ configuration to suit your specific requirements. - -The example below shows how to configure PQ with custom settings, such as with a lower training set size, and a different number of centroids. - -
-
-
-## Why use multi-tenancy?
-
-A typical multi-tenancy use-case is in a software-as-a-service (SaaS) application. In many SaaS applications, each end user or account will have private data that should be not be accessible to anyone else.
-
-### Example case study
-
-In this course, we'll learn about multi-tenancy by putting ourselves in the shoes of a developer building an application called `MyPrivateJournal`.
-
-`MyPrivateJournal` is a SaaS (software-as-a-service) application where users like *Steve*, *Alice* and so on can write and store their journal entries. Each user's entries should be private and not accessible to anyone else.
-
-Using single-tenant collections, you might implement this with:
-
-1. **A monolithic collection**: To store the entire dataset, with an end user identifier property
-1. **Per end-user collections**: Where each end user's data would be in a separate collection
-
-While these may work to some extent, both of these options have significant limitations.
-
-- Using a monolithic collection:
- - A developer mistake could easily expose Steve's entries to Alice, which would be a significant privacy breach.
- - As `MyPrivateJournal` grows, Steve's query would become slower as it must look through the entire collection.
- - When Steve asks `MyPrivateJournal` to delete his data, the process would be complex and error-prone.
-- Using end-user-specific collections:
- - `MyPrivteJournal` may need to spend more on hardware to support the high number of collections.
- - Changes to configurations (e.g. adding a new property) would need to be run separately for each collection.
-
-Multi-tenancy in Weaviate solves these problems by providing a way to isolate each user's data while sharing the same configuration.
-
-### Benefits of multi-tenancy
-
-In multi-tenant collection, each "tenant" is isolated from each other, while sharing the same set of configurations. This arrangement helps make multi-tenancy far more resource-efficient than using many individual collections.
-
-A Weaviate node can host more tenants than single-tenant collections.
-
-It also makes developers' job easier, as there is only one set of collection configurations. The data isolation between tenants eliminates risks of accidental data leakage and makes it easier to manage individual tenants and tenant data.
-
-#### `MyPrivateJournal` and multi-tenancy
-
-So, the `MyPrivateJournal` app can use multi-tenancy and store each user's journal entries in a separate tenant. This way, Steve's entries are isolated from Alice's, and vice versa. This isolation makes it easier to manage each user's data and reduces the risk of data leakage.
-
-As you will see later, `MyPrivateJournal` can also offload inactive users' data to cold storage, reducing the hot (memory) and warm (disk) resource usage of the Weaviate node.
-
-## Tenants vs collections
-
-Each multi-tenant collection can have any number of tenants.
-
-A tenant is very similar to a single-tenant collection. For example:
-
-| Aspect | Tenant | Single-tenant collection |
-| ----- | ----- | ----- |
-| Objects | Belong to a tenant | Belong to a collection |
-| Vector indexes | Belong to a tenant | Belong to a collection |
-| Inverted indexes | Belong to a tenant | Belong to a collection |
-| Deletion | Deleting a tenant deletes all tenant data | Deleting a collection deletes all collection data |
-| Query | Can search one tenant at a time | Can search one collection at a time |
-
-But as you will have guessed, there are also differences. We'll cover these in the next sections, as we follow `MyPrivateJournal` implementing multi-tenancy in Weaviate.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-"Multi-tenancy" in other contexts
- -In general, the term "multi-tenancy" refers to a software architecture where a single instance of the software serves multiple "tenants". In that context, each tenant may be a group of users who share common access. - -This is similar to the concept of multi-tenancy in Weaviate, where each tenant is a group of data that is isolated from other tenants. - -
-
- -But we can easily extend this to a multi-node setup by adding additional services. This will allow you to scale your Weaviate instance horizontally, and provide fault tolerance with replication. -
- -For example, here is a multi-node setup with three nodes. - -```yaml ---- -services: - weaviate-node-1: # Founding member service name - command: - - --host - - 0.0.0.0 - - --port - - '8080' - - --scheme - - http - image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| - restart: on-failure:0 - ports: - - "8180:8080" - - 50151:50051 - environment: - AUTOSCHEMA_ENABLED: 'false' - QUERY_DEFAULTS_LIMIT: 25 - QUERY_MAXIMUM_RESULTS: 10000 - AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' - PERSISTENCE_DATA_PATH: '/var/lib/weaviate' - ASYNC_INDEXING: 'true' - ENABLE_MODULES: 'text2vec-ollama,generative-ollama,backup-filesystem,offload-s3' - ENABLE_API_BASED_MODULES: 'true' - AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY:-} - AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_KEY:-} - OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true' - BACKUP_FILESYSTEM_PATH: '/var/lib/weaviate/backups' - CLUSTER_HOSTNAME: 'node1' - CLUSTER_GOSSIP_BIND_PORT: '7100' - CLUSTER_DATA_BIND_PORT: '7101' - weaviate-node-2: # Founding member service name - command: - - --host - - 0.0.0.0 - - --port - - '8080' - - --scheme - - http - image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| - restart: on-failure:0 - ports: - - "8181:8080" - - 50152:50051 - environment: - AUTOSCHEMA_ENABLED: 'false' - QUERY_DEFAULTS_LIMIT: 25 - QUERY_MAXIMUM_RESULTS: 10000 - AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' - PERSISTENCE_DATA_PATH: '/var/lib/weaviate' - ASYNC_INDEXING: 'true' - ENABLE_MODULES: 'text2vec-ollama,generative-ollama,backup-filesystem,offload-s3' - ENABLE_API_BASED_MODULES: 'true' - AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY:-} - AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_KEY:-} - OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true' - BACKUP_FILESYSTEM_PATH: '/var/lib/weaviate/backups' - CLUSTER_HOSTNAME: 'node2' - CLUSTER_GOSSIP_BIND_PORT: '7102' - CLUSTER_DATA_BIND_PORT: '7103' - CLUSTER_JOIN: 'weaviate-node-1:7100' - weaviate-node-3: # Founding member service name - command: - - --host - - 0.0.0.0 - - --port - - '8080' - - --scheme - - http - image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| - restart: on-failure:0 - ports: - - "8182:8080" - - 50153:50051 - environment: - AUTOSCHEMA_ENABLED: 'false' - QUERY_DEFAULTS_LIMIT: 25 - QUERY_MAXIMUM_RESULTS: 10000 - AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' - PERSISTENCE_DATA_PATH: '/var/lib/weaviate' - ASYNC_INDEXING: 'true' - image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| - ENABLE_API_BASED_MODULES: 'true' - AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY:-} - AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_KEY:-} - OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true' - BACKUP_FILESYSTEM_PATH: '/var/lib/weaviate/backups' - CLUSTER_HOSTNAME: 'node3' - CLUSTER_GOSSIP_BIND_PORT: '7104' - CLUSTER_DATA_BIND_PORT: '7105' - CLUSTER_JOIN: 'weaviate-node-1:7100' -... -``` - -
-
-### Configuration highlights
-
-You may have seen Docker configurations elsewhere ([e.g. Docs](/deploy/installation-guides/docker-installation.md), [Academy](../starter_text_data/101_setup_weaviate/20_create_instance/20_create_docker.mdx)). But these highlighted configurations may be new to you:
-
-- `ASYNC_INDEXING`: This will enable asynchronous indexing. This is useful for high-volume data insertion, and enables us to use the `dynamic` index type, which you will learn about later on.
-- `ENABLE_MODULES`: We enable `offload-s3` to demonstrate tenant offloading later on. Offloading helps us to manage inactive users' data efficiently.
-- `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`: These are the AWS credentials that Weaviate will use to access the S3 bucket.
-- `OFFLOAD_S3_BUCKET_AUTO_CREATE`: This will automatically create the S3 bucket if it does not exist.
-
-Save the file to `docker-compose.yaml`, and run the following command to start Weaviate:
-
-```bash
-docker compose up
-```
-
-import OffloadingLimitation from '/_includes/offloading-limitation.mdx';
-
-What about a multi-node setup?
- -Great question! As you probably noticed, we are using a single-node setup here for simplicity. -- -But we can easily extend this to a multi-node setup by adding additional services. This will allow you to scale your Weaviate instance horizontally, and provide fault tolerance with replication. -
- -For example, here is a multi-node setup with three nodes. - -```yaml ---- -services: - weaviate-node-1: # Founding member service name - command: - - --host - - 0.0.0.0 - - --port - - '8080' - - --scheme - - http - image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| - restart: on-failure:0 - ports: - - "8180:8080" - - 50151:50051 - environment: - AUTOSCHEMA_ENABLED: 'false' - QUERY_DEFAULTS_LIMIT: 25 - QUERY_MAXIMUM_RESULTS: 10000 - AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' - PERSISTENCE_DATA_PATH: '/var/lib/weaviate' - ASYNC_INDEXING: 'true' - ENABLE_MODULES: 'text2vec-ollama,generative-ollama,backup-filesystem,offload-s3' - ENABLE_API_BASED_MODULES: 'true' - AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY:-} - AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_KEY:-} - OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true' - BACKUP_FILESYSTEM_PATH: '/var/lib/weaviate/backups' - CLUSTER_HOSTNAME: 'node1' - CLUSTER_GOSSIP_BIND_PORT: '7100' - CLUSTER_DATA_BIND_PORT: '7101' - weaviate-node-2: # Founding member service name - command: - - --host - - 0.0.0.0 - - --port - - '8080' - - --scheme - - http - image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| - restart: on-failure:0 - ports: - - "8181:8080" - - 50152:50051 - environment: - AUTOSCHEMA_ENABLED: 'false' - QUERY_DEFAULTS_LIMIT: 25 - QUERY_MAXIMUM_RESULTS: 10000 - AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' - PERSISTENCE_DATA_PATH: '/var/lib/weaviate' - ASYNC_INDEXING: 'true' - ENABLE_MODULES: 'text2vec-ollama,generative-ollama,backup-filesystem,offload-s3' - ENABLE_API_BASED_MODULES: 'true' - AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY:-} - AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_KEY:-} - OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true' - BACKUP_FILESYSTEM_PATH: '/var/lib/weaviate/backups' - CLUSTER_HOSTNAME: 'node2' - CLUSTER_GOSSIP_BIND_PORT: '7102' - CLUSTER_DATA_BIND_PORT: '7103' - CLUSTER_JOIN: 'weaviate-node-1:7100' - weaviate-node-3: # Founding member service name - command: - - --host - - 0.0.0.0 - - --port - - '8080' - - --scheme - - http - image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| - restart: on-failure:0 - ports: - - "8182:8080" - - 50153:50051 - environment: - AUTOSCHEMA_ENABLED: 'false' - QUERY_DEFAULTS_LIMIT: 25 - QUERY_MAXIMUM_RESULTS: 10000 - AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' - PERSISTENCE_DATA_PATH: '/var/lib/weaviate' - ASYNC_INDEXING: 'true' - image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| - ENABLE_API_BASED_MODULES: 'true' - AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY:-} - AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_KEY:-} - OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true' - BACKUP_FILESYSTEM_PATH: '/var/lib/weaviate/backups' - CLUSTER_HOSTNAME: 'node3' - CLUSTER_GOSSIP_BIND_PORT: '7104' - CLUSTER_DATA_BIND_PORT: '7105' - CLUSTER_JOIN: 'weaviate-node-1:7100' -... -``` - -
- More about
-
-:::info Added in `v1.25`
-The auto tenant creation feature is available from `v1.25.0` for batch imports, and from `v1.25.2` for single object insertions.
-:::
-
-Enabling `auto_tenant_creation` will cause Weaviate to automatically create the tenant when an object is inserted against a non-existent tenant.
-
- -This option is particularly useful for bulk data ingestion, as it removes the need to create the tenant prior to object insertion. Instead, `auto_tenant_creation` will allow the object insertion process to continue without interruption. -
- -A risk of using `auto_tenant_creation` is that an error in the source data will not be caught during import. For example, a source object with erroneously spelt `"TenntOn"` instead of `"TenantOne"` will create a new tenant for `"TenntOne"` instead of raising an error. -
- -The server-side default for `auto_tenant_creation` is `false`. -
-
-More about auto_tenant_creation
-
-:::info Added in `v1.25`
-The auto tenant creation feature is available from `v1.25.0` for batch imports, and from `v1.25.2` for single object insertions.
-:::
-
-Enabling `auto_tenant_creation` will cause Weaviate to automatically create the tenant when an object is inserted against a non-existent tenant.
-- -This option is particularly useful for bulk data ingestion, as it removes the need to create the tenant prior to object insertion. Instead, `auto_tenant_creation` will allow the object insertion process to continue without interruption. -
- -A risk of using `auto_tenant_creation` is that an error in the source data will not be caught during import. For example, a source object with erroneously spelt `"TenntOn"` instead of `"TenantOne"` will create a new tenant for `"TenntOne"` instead of raising an error. -
- -The server-side default for `auto_tenant_creation` is `false`. -
- More about
-
-:::info Added in `v1.25.2`
-The auto tenant activation feature is available from `v1.25.2`.
-:::
-
-If `auto_tenant_activation` is enabled, Weaviate will automatically activate any deactivated (`INACTIVE` or `OFFLOADED`) tenants when they are accessed.
-
- -This option is particularly useful for scenarios where you have a large number of tenants, but only a subset of them are active at any given time. An example is a SaaS app where some tenants may be unlikely due to their local time zone, or their recent activity level. -
- -By enabling `auto_tenant_activation`, you can safely set those less active users to be inactive, knowing that they will be loaded onto memory once requested. -
- -This can help to reduce the memory footprint of your Weaviate instance, as only the active tenants are loaded into memory. -
- -The server-side default for `auto_tenant_activation` is `false`. - -
-
-### Configure vector index
-
-From what we know about other journal use cases, a majority of users will only have a small number of entries. But, a few of those users may have a large number of entries.
-
-This is a tricky situation to balance. If we use a `hnsw` index, it will be fast for users with many entries, but it will require a lot of memory. If we use a `flat` index, it will require less memory, but potentially slower for users with many entries.
-
-What we can do here is to choose a `dynamic` index. A `dynamic` index will automatically switch from `flat` to `hnsw` once it passes a threshold count. This way, we can balance the memory usage and speed for our users.
-
-Here is an example code snippet, configuring a "note" named vector with a `dynamic` index.
-
-More about auto_tenant_activation
-
-:::info Added in `v1.25.2`
-The auto tenant activation feature is available from `v1.25.2`.
-:::
-
-If `auto_tenant_activation` is enabled, Weaviate will automatically activate any deactivated (`INACTIVE` or `OFFLOADED`) tenants when they are accessed.
-- -This option is particularly useful for scenarios where you have a large number of tenants, but only a subset of them are active at any given time. An example is a SaaS app where some tenants may be unlikely due to their local time zone, or their recent activity level. -
- -By enabling `auto_tenant_activation`, you can safely set those less active users to be inactive, knowing that they will be loaded onto memory once requested. -
- -This can help to reduce the memory footprint of your Weaviate instance, as only the active tenants are loaded into memory. -
- -The server-side default for `auto_tenant_activation` is `false`. - -
-
-
-Data being inserted
- -The objects to be added can be a list of dictionaries, as shown here. Note the use of `datetime` objects with a timezone for `DATE` type properties. - -
-
-
-### Search for entries
-
-Additionally, Steve might want to search for entries. For example - he might want to search for entries relating to some food experience that he had.
-
-`MyPrivateJournal` can leverage Weaviate's `hybrid` search to help Steve find the most relevant entries.
-
-Example response
- -Such a query should return a response like: - -
-
-
-You can see that the search syntax is essentially identical to that of a single-tenant collection. So, any search method available for a single-tenant collection can be applied to a multi-tenant collection.
-
-### Summary
-
-In this section, we learned how to work with tenants and their data in a multi-tenant collection. We saw how to create tenants, add data objects, and query tenant data.
-
-In the next section, we will learn how `MyPrivateJournal` can keep their application running smoothly and efficiently by managing tenants.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example response
- -
-
- -To use tenant offloading in Weaviate, you need enable a relevant offloading [module](../../../weaviate/configuration/modules.md). Depending on whether your deployment is on Docker or Kubernetes, you can enable the `offload-s3` module as shown below. - -
-
-
-```yaml
-services:
- weaviate:
- environment:
- # highlight-start
- ENABLE_MODULES: 'offload-s3' # plus other modules you may need
- OFFLOAD_S3_BUCKET: 'weaviate-offload' # the name of the S3 bucket
- OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true' # create the bucket if it does not exist
- # highlight-end
-```
-
-
-
-
-```yaml
-offload:
- s3:
- enabled: true # Set this value to true to enable the offload-s3 module
- envconfig:
- OFFLOAD_S3_BUCKET: weaviate-offload # the name of the S3 bucket
- OFFLOAD_S3_BUCKET_AUTO_CREATE: true # create the bucket if it does not exist
-```
-
-
-
-
-If the target S3 bucket does not exist, the `OFFLOAD_S3_BUCKET_AUTO_CREATE` variable must be set to `true` so that Weaviate can create the bucket automatically.
-
- -#### AWS permissions - -You must provide Weaviate with AWS authentication details. You can choose between access-key or ARN-based authentication. -
- -:::tip Requirements -The Weaviate instance must have the [necessary permissions to access the S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). -- The provided AWS identity must be able to write to the bucket. -- If `OFFLOAD_S3_BUCKET_AUTO_CREATE` is set to `true`, the AWS identity must have permission to create the bucket. -::: - -**Option 1: With IAM and ARN roles** -
- -The backup module will first try to authenticate itself using AWS IAM. If the authentication fails then it will try to authenticate with `Option 2`. -
- -**Option 2: With access key and secret access key** -
- -| Environment variable | Description | -| --- | --- | -| `AWS_ACCESS_KEY_ID` | The id of the AWS access key for the desired account. | -| `AWS_SECRET_ACCESS_KEY` | The secret AWS access key for the desired account. | -| `AWS_REGION` | (Optional) The AWS Region. If not provided, the module will try to parse `AWS_DEFAULT_REGION`. | - -Once the `offload-s3` module is enabled, you can offload tenants to the S3 bucket by [setting their activity status](#offload-tenants) to `OFFLOADED`, or load them back to local storage by setting their status to `ACTIVE` or `INACTIVE`. - -
-
-#### Activate users
-
-And then, `MyPrivateJournal` can activate tenants as required. For example, they could activate a tenant when the user logs in, or based on their local time in an inverse pattern to deactivation:
-
-How to set up offloading
- -import OffloadingLimitation from '/_includes/offloading-limitation.mdx'; - -- -To use tenant offloading in Weaviate, you need enable a relevant offloading [module](../../../weaviate/configuration/modules.md). Depending on whether your deployment is on Docker or Kubernetes, you can enable the `offload-s3` module as shown below. - -
- -#### AWS permissions - -You must provide Weaviate with AWS authentication details. You can choose between access-key or ARN-based authentication. -
- -:::tip Requirements -The Weaviate instance must have the [necessary permissions to access the S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). -- The provided AWS identity must be able to write to the bucket. -- If `OFFLOAD_S3_BUCKET_AUTO_CREATE` is set to `true`, the AWS identity must have permission to create the bucket. -::: - -**Option 1: With IAM and ARN roles** -
- -The backup module will first try to authenticate itself using AWS IAM. If the authentication fails then it will try to authenticate with `Option 2`. -
- -**Option 2: With access key and secret access key** -
- -| Environment variable | Description | -| --- | --- | -| `AWS_ACCESS_KEY_ID` | The id of the AWS access key for the desired account. | -| `AWS_SECRET_ACCESS_KEY` | The secret AWS access key for the desired account. | -| `AWS_REGION` | (Optional) The AWS Region. If not provided, the module will try to parse `AWS_DEFAULT_REGION`. | - -Once the `offload-s3` module is enabled, you can offload tenants to the S3 bucket by [setting their activity status](#offload-tenants) to `OFFLOADED`, or load them back to local storage by setting their status to `ACTIVE` or `INACTIVE`. - -
- Example
-
-Note that this output is a little longer due to the additional details from the CLIP models.
-
-
-
-### Close the connection
-
-After you have finished using the Weaviate client, you should close the connection. This frees up resources and ensures that the connection is properly closed.
-
-We suggest using a `try`-`finally` block as a best practice. For brevity, we will not include the `try`-`finally` blocks in the remaining code snippets.
-
-Example get_meta output
-
-Note that this output is a little longer due to the additional details from the CLIP models.
-
-
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-See sample text data
- -| | backdrop_path | genre_ids | id | original_language | original_title | overview | popularity | poster_path | release_date | title | video | vote_average | vote_count | -|---:|:---------------------------------|:----------------|-----:|:--------------------|:----------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------:|:---------------------------------|:---------------|:----------------------------|:--------|---------------:|-------------:| -| 0 | /3Nn5BOM1EVw1IYrv6MsbOS6N1Ol.jpg | [14, 18, 10749] | 162 | en | Edward Scissorhands | A small suburban town receives a visit from a castaway unfinished science experiment named Edward. | 45.694 | /1RFIbuW9Z3eN9Oxw2KaQG5DfLmD.jpg | 1990-12-07 | Edward Scissorhands | False | 7.7 | 12305 | -| 1 | /sw7mordbZxgITU877yTpZCud90M.jpg | [18, 80] | 769 | en | GoodFellas | The true story of Henry Hill, a half-Irish, half-Sicilian Brooklyn kid who is adopted by neighbourhood gangsters at an early age and climbs the ranks of a Mafia family under the guidance of Jimmy Conway. | 57.228 | /aKuFiU82s5ISJpGZp7YkIr3kCUd.jpg | 1990-09-12 | GoodFellas | False | 8.5 | 12106 | -| 2 | /6uLhSLXzB1ooJ3522ydrBZ2Hh0W.jpg | [35, 10751] | 771 | en | Home Alone | Eight-year-old Kevin McCallister makes the most of the situation after his family unwittingly leaves him behind when they go on Christmas vacation. But when a pair of bungling burglars set their sights on Kevin's house, the plucky kid stands ready to defend his territory. By planting booby traps galore, adorably mischievous Kevin stands his ground as his frantic mother attempts to race home before Christmas Day. | 3.538 | /onTSipZ8R3bliBdKfPtsDuHTdlL.jpg | 1990-11-16 | Home Alone | False | 7.4 | 10599 | -| 3 | /vKp3NvqBkcjHkCHSGi6EbcP7g4J.jpg | [12, 35, 878] | 196 | en | Back to the Future Part III | The final installment of the Back to the Future trilogy finds Marty digging the trusty DeLorean out of a mineshaft and looking for Doc in the Wild West of 1885. But when their time machine breaks down, the travelers are stranded in a land of spurs. More problems arise when Doc falls for pretty schoolteacher Clara Clayton, and Marty tangles with Buford Tannen. | 28.896 | /crzoVQnMzIrRfHtQw0tLBirNfVg.jpg | 1990-05-25 | Back to the Future Part III | False | 7.5 | 9918 | -| 4 | /3tuWpnCTe14zZZPt6sI1W9ByOXx.jpg | [35, 10749] | 114 | en | Pretty Woman | When a millionaire wheeler-dealer enters a business contract with a Hollywood hooker Vivian Ward, he loses his heart in the bargain. | 97.953 | /hVHUfT801LQATGd26VPzhorIYza.jpg | 1990-03-23 | Pretty Woman | False | 7.5 | 7671 | - -
-
-
-
-
-
-```text
-Life Is Beautiful 1997 637
-Distance to query: 0.621
-
-Groundhog Day 1993 137
-Distance to query: 0.623
-
-Jingle All the Way 1996 9279
-Distance to query: 0.625
-
-Training Day 2001 2034
-Distance to query: 0.627
-
-Misery 1990 1700
-Distance to query: 0.632
-```
-
- 
-
-Luckily for them, the `MovieNVDemo` collection has `poster_title` named vectors which is primarily based on the poster design. So Aesthetico's designers can search against the `poster_title` named vector and find movies that are similar to their poster design. And, they can then perform RAG to summarize the movies that are found.
-
-### Code
-
-This query will find similar movies to the input image, and then provide insights using RAG.
-
-
-
-Inception
-Mission: Impossible
-The Dark Knight
-Lost in Translation
-Independence Day
-Godzilla vs. Kong
-Fargo
-The Amazing Spider-Man
-Godzilla
- -
-
-## Film writers: evaluating ideas
-
-Now, in another project, a set of writers at *ScriptManiacs* are working on a movie script for a science fiction film. They are working a few ideas for the movie title, and they want to see what kinds of imagery and themes are associated with each title.
-
-They could also use the same collection to do what they want to do. In fact, they could run multiple queries against the same collection, each with a different `target_vector` parameter.
-
-The ScriptManiacs writers can:
-- Search against the `title` named vector to find movies with *similar titles*;
-- Search against the `overview` named vector to find movies whose *plots are similar* to their title idea; and
-
-Let's see how they could do it for a title - "Chrono Tides: The Anomaly Rift".
-
-### Code
-
-This example finds entries in "MovieNVDemo" based on their similarity to "Chrono Tides: The Anomaly Rift", then instructs the large language model to find commonalities between them.
-
-Note the `for tgt_vector` loop, which allows the writers to run the same query against different named vectors.
-
-Search results
- -Predator 2-Inception
-Mission: Impossible
-The Dark Knight
-Lost in Translation
-Independence Day
-Godzilla vs. Kong
-Fargo
-The Amazing Spider-Man
-Godzilla
- -
-
-Lara Croft: Tomb Raider
-The Croods: A New Age
-The Twilight Saga: Breaking Dawn - Part 1
-Meg 2: The Trench
- -
-
-#### Similar overviews
-
-While movies with overviews that are most similar to the search "Chrono Tides: The Anomaly Rift" have the following commonalities:
-
-> These movies can be categorized as **science fiction and adventure films**. They are aimed at audiences who enjoy stories about space exploration, ancient civilizations, dinosaurs, natural disasters, and mythical adventures. The target audience may include fans of action-packed and visually stunning movies with elements of fantasy and suspense.
-
-Search results
- -Pirates of the Caribbean: On Stranger Tides-Lara Croft: Tomb Raider
-The Croods: A New Age
-The Twilight Saga: Breaking Dawn - Part 1
-Meg 2: The Trench
- -
-
-Interstellar
-Jurassic Park III
-2012
-Moana
- -
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Search results
- -Stargate-Interstellar
-Jurassic Park III
-2012
-Moana
- -
-
-
-The schema specifies for each class the metadata such as its name and description as well as the same for properties of each class. Additionally, it defines any cross-references between classes and settings for indexes including the object index and the vector index.
-
-It's important to note that the schema holds information about the vectorization process, including which vectorization model to use, which properties to include or exclude, and whether to include property headings or not.
-
-:::info Auto-schema
-Any missing information required for schema definition will be automatically inferred by Weaviate based on default values and the imported data.
-:::
-
-#### Data types
-
-Each `class` definition will include one or more properties, which must have a data type. Currently, Weaviate data type support includes the following types:
-
-import DataTypes from '/_includes/datatypes.mdx';
-
-Example schema
- TODO: Add actual example schema -
-
-
-Note that most data types can include one such instance, or a list of instances, such as `string` or `string[]`.
-
-We will look at schema definition including data types in more detail in another unit.
-
-Available data types in Weaviate
-
-
-
-### Chunking
-
-Now, let's apply the following chunking methods on each section:
-
-- Fixed-length chunks (and 20% overlap)
- - With 25 words per chunk, and
- - With 100 words per chunk
-- Variable-length chunks, using paragraph markers, and
-- Mixed-strategy chunks, using paragraph markers and a minimum chunk length of 25 words.
-
-We'll also add metadata to the chunk, such as the filename and the chapter name, as well as the chunk number.
-
-One implementation is shown below:
-
-Take a look at some sample sections
- -``` -{ - "body": "=== About Version Control\n\n(((version control)))\nWhat is \"`version control`\", and why should you care?\nVersion control is a system that records changes to a file or set of files over time so that you can recall specific versions later.\nFor the examples in this book, you will use software source code as the files being version controlled, though in reality you can do this with nearly any type of file on a computer.\n\nIf you are a graphic or web designer and want to keep every version of an image or layout (which you would most certainly want to), a Version Control System (VCS) is a very wise thing to use.\nIt allows you to revert selected files back to a previous state, revert the entire project back to a previous state, compare changes over time, see who last modified something that might be causing a problem, who introduced an issue and when, and more.\nUsing a VCS also generally means that if you screw things up or lose files, you can easily recover.\nIn addition, you get all this for very little overhead.\n\n==== Local Version Control Systems\n\n(((version control,local)))\nMany people's version-control method of choice is to copy files into another directory (perhaps a time-stamped directory, if they're clever).\nThis approach is very common because it is so simple, but it is also incredibly error prone.\nIt is easy to forget which directory you're in and accidentally write to the wrong file or copy over files you don't mean to.\n\nTo deal with this issue, programmers long ago developed local VCSs that had a simple database that kept all the changes to files under revision control.\n\n.Local version control diagram\nimage::images/local.png[Local version control diagram]\n\nOne of the most popular VCS tools was a system called RCS, which is still distributed with many computers today.\nhttps://www.gnu.org/software/rcs/[RCS^] works by keeping patch sets (that is, the differences between files) in a special format on disk; it can then re-create what any file looked like at any point in time by adding up all the patches.\n\n==== Centralized Version Control Systems\n\n(((version control,centralized)))\nThe next major issue that people encounter is that they need to collaborate with developers on other systems.\nTo deal with this problem, Centralized Version Control Systems (CVCSs) were developed.\nThese systems (such as CVS, Subversion, and Perforce) have a single server that contains all the versioned files, and a number of clients that check out files from that central place.(((CVS)))(((Subversion)))(((Perforce)))\nFor many years, this has been the standard for version control.\n\n.Centralized version control diagram\nimage::images/centralized.png[Centralized version control diagram]\n\nThis setup offers many advantages, especially over local VCSs.\nFor example, everyone knows to a certain degree what everyone else on the project is doing.\nAdministrators have fine-grained control over who can do what, and it's far easier to administer a CVCS than it is to deal with local databases on every client.\n\nHowever, this setup also has some serious downsides.\nThe most obvious is the single point of failure that the centralized server represents.\nIf that server goes down for an hour, then during that hour nobody can collaborate at all or save versioned changes to anything they're working on.\nIf the hard disk the central database is on becomes corrupted, and proper backups haven't been kept, you lose absolutely everything -- the entire history of the project except whatever single snapshots people happen to have on their local machines.\nLocal VCSs suffer from this same problem -- whenever you have the entire history of the project in a single place, you risk losing everything.\n\n==== Distributed Version Control Systems\n\n(((version control,distributed)))\nThis is where Distributed Version Control Systems (DVCSs) step in.\nIn a DVCS (such as Git, Mercurial, Bazaar or Darcs), clients don't just check out the latest snapshot of the files; rather, they fully mirror the repository, including its full history.\nThus, if any server dies, and these systems were collaborating via that server, any of the client repositories can be copied back up to the server to restore it.\nEvery clone is really a full backup of all the data.\n\n.Distributed version control diagram\nimage::images/distributed.png[Distributed version control diagram]\n\nFurthermore, many of these systems deal pretty well with having several remote repositories they can work with, so you can collaborate with different groups of people in different ways simultaneously within the same project.\nThis allows you to set up several types of workflows that aren't possible in centralized systems, such as hierarchical models.\n", - "chapter_title": "01-introduction", - "filename": "about-version-control.asc" -} -``` - -
-
-
-
-
-
-
-
-Helper functions
- -- - - - - -Chunking is an important concept in the world of vector databases and language models. Although we've looked at relatively small pieces of text in previous units, real-world text data can be much longer. - -Think about lengths of articles, transcripts, or even books. Instead of a few words, these texts can be thousands, or tens of thousands of words long if not longer. *The Lord of the Rings*, for example, is over 500,000 words long! - -Chunking splits texts like these into smaller pieces of texts, i.e. "chunks", before storing them in a vector database, or passing them to a language model. - -This can seem relatively innocuous at first, like deciding where to split a sentence or a paragraph into two. But chunking decisions can significantly impact the search performance and behavior of vector databases as well as the output from a language model. This unit covers this seemingly simple, nuanced topic from the perspective of a user. - -We will begin by covering what chunking is, and why it is used. Then, we will move on to cover various chunking methods before discussing key considerations when deciding on a chunking strategy, as well as some suggested starting points. - -By the end of this unit, you will have a good understanding of chunking in general, and be able to implement some solid chunking strategies based on your actual needs. - -### Prerequisites - -- (**Required**) A Python (3) environment with `weaviate-client` installed. -- (**Required**) Complete [101A Weaviate Academy Preparation](../../zero_to_mvp/setup.mdx) -- (*Recommended*) Complete [Hello, Weaviate](../../zero_to_mvp/101_hello_weaviate/index.mdx) -- (*Recommended*) Complete [Queries 1](../../zero_to_mvp/102_queries_1/index.mdx) -- (*Recommended*) Complete [Schema and Imports](../../zero_to_mvp/103_schema_and_imports/index.mdx) - -## Learning objectives - -import LearningGoalsExp from '/src/components/Academy/learningGoalsExp.mdx'; - -
- Example
-
-
-
-### Close the connection
-
-After you have finished using the Weaviate client, you should close the connection. This frees up resources and ensures that the connection is properly closed.
-
-We suggest using a `try`-`finally` block as a best practice. For brevity, we will not include the `try`-`finally` blocks in the remaining code snippets.
-
-Example get_meta output
-
-
-
-
-Next, you will create a corresponding object collection and import the data.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-See sample data
- -| | backdrop_path | genre_ids | id | original_language | original_title | overview | popularity | poster_path | release_date | title | video | vote_average | vote_count | -|---:|:---------------------------------|:----------------|-----:|:--------------------|:----------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------:|:---------------------------------|:---------------|:----------------------------|:--------|---------------:|-------------:| -| 0 | /3Nn5BOM1EVw1IYrv6MsbOS6N1Ol.jpg | [14, 18, 10749] | 162 | en | Edward Scissorhands | A small suburban town receives a visit from a castaway unfinished science experiment named Edward. | 45.694 | /1RFIbuW9Z3eN9Oxw2KaQG5DfLmD.jpg | 1990-12-07 | Edward Scissorhands | False | 7.7 | 12305 | -| 1 | /sw7mordbZxgITU877yTpZCud90M.jpg | [18, 80] | 769 | en | GoodFellas | The true story of Henry Hill, a half-Irish, half-Sicilian Brooklyn kid who is adopted by neighbourhood gangsters at an early age and climbs the ranks of a Mafia family under the guidance of Jimmy Conway. | 57.228 | /aKuFiU82s5ISJpGZp7YkIr3kCUd.jpg | 1990-09-12 | GoodFellas | False | 8.5 | 12106 | -| 2 | /6uLhSLXzB1ooJ3522ydrBZ2Hh0W.jpg | [35, 10751] | 771 | en | Home Alone | Eight-year-old Kevin McCallister makes the most of the situation after his family unwittingly leaves him behind when they go on Christmas vacation. But when a pair of bungling burglars set their sights on Kevin's house, the plucky kid stands ready to defend his territory. By planting booby traps galore, adorably mischievous Kevin stands his ground as his frantic mother attempts to race home before Christmas Day. | 3.538 | /onTSipZ8R3bliBdKfPtsDuHTdlL.jpg | 1990-11-16 | Home Alone | False | 7.4 | 10599 | -| 3 | /vKp3NvqBkcjHkCHSGi6EbcP7g4J.jpg | [12, 35, 878] | 196 | en | Back to the Future Part III | The final installment of the Back to the Future trilogy finds Marty digging the trusty DeLorean out of a mineshaft and looking for Doc in the Wild West of 1885. But when their time machine breaks down, the travelers are stranded in a land of spurs. More problems arise when Doc falls for pretty schoolteacher Clara Clayton, and Marty tangles with Buford Tannen. | 28.896 | /crzoVQnMzIrRfHtQw0tLBirNfVg.jpg | 1990-05-25 | Back to the Future Part III | False | 7.5 | 9918 | -| 4 | /3tuWpnCTe14zZZPt6sI1W9ByOXx.jpg | [35, 10749] | 114 | en | Pretty Woman | When a millionaire wheeler-dealer enters a business contract with a Hollywood hooker Vivian Ward, he loses his heart in the bargain. | 97.953 | /hVHUfT801LQATGd26VPzhorIYza.jpg | 1990-03-23 | Pretty Woman | False | 7.5 | 7671 | - -
-
-
-### Response object
-
-The returned object is an instance of a custom class. Its `objects` attribute is a list of search results, each object being an instance of another custom class.
-
-Each returned object will:
-- Include all properties and its UUID by default except those with blob data types.
-- Not include any other information (e.g. references, metadata, vectors.) by default.
-
-### Where did the query vector come from?
-
-The query vector in this example is obtained similarly to how it was in the [data ingestion](../102_object_collections/25_generate_vectors.mdx). The only difference is that the vector is not stored in the database, but is used directly in the query.
-
-Example results
- -```text -In Time 2011 -Distance to query: 0.179 - -Gattaca 1997 -Distance to query: 0.180 - -I, Robot 2004 -Distance to query: 0.182 - -Mad Max: Fury Road 2015 -Distance to query: 0.190 - -The Maze Runner 2014 -Distance to query: 0.193 -``` - -
-
-
-
-## Hybrid search
-
-### Code
-
-This example finds entries in "Movie" with the highest hybrid search scores for the term "history", and prints out the title and release year of the top 5 matches.
-
-Example results
- -```text -American History X 1998 -BM25 score: 2.707 - -A Beautiful Mind 2001 -BM25 score: 1.896 - -Legends of the Fall 1994 -BM25 score: 1.663 - -Hacksaw Ridge 2016 -BM25 score: 1.554 - -Night at the Museum 2006 -BM25 score: 1.529 -``` - -
-
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Night at the Museum 2006 -Hybrid score: 0.016 - -The Butterfly Effect 2004 -Hybrid score: 0.014 - -Legends of the Fall 1994 -Hybrid score: 0.014 - -Hidden Figures 2016 -Hybrid score: 0.012 - -A Beautiful Mind 2001 -Hybrid score: 0.012 -``` - -
-
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Oppenheimer 2023 -Distance to query: 0.754 - -Everything Everywhere All at Once 2022 -Distance to query: 0.778 - -Meg 2: The Trench 2023 -Distance to query: 0.779 - -Eternals 2021 -Distance to query: 0.787 - -John Wick: Chapter 4 2023 -Distance to query: 0.790 -``` - -
-
-
-### Response object
-
-Each response object is similar to that from a regular search query, with an additional `generated` attribute. This attribute will contain the generated output for each object.
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -In Time -À temps -Gattaca -Gattaca -I, Robot -Je, Robot -Mad Max: Fury Road -Mad Max: Fury Road -The Maze Runner -Le Labyrinthe -``` - -
-
-
-### Optional parameters
-
-You can also pass on a list of properties to be used, as the `grouped_properties` parameter. This can be useful to reduce the amount of data passed on to the large language model and omit irrelevant properties.
-
-### Response object
-
-A RAG query with the `grouped_task` parameter will return a response with an additional `generated` attribute. This attribute will contain the generated output for the set of objects.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -In Time -Gattaca -I, Robot -Mad Max: Fury Road -The Maze Runner -These movies all take place in a futuristic or dystopian society where the characters must navigate complex systems and face challenges related to technology, society, and survival. They all explore themes of control, power, and the consequences of scientific advancements on humanity. -``` - -
- Example
-
-Note that this output is a little longer due to the additional details from the CLIP models.
-
-
-
-### Close the connection
-
-After you have finished using the Weaviate client, you should close the connection. This frees up resources and ensures that the connection is properly closed.
-
-We suggest using a `try`-`finally` block as a best practice. For brevity, we will not include the `try`-`finally` blocks in the remaining code snippets.
-
-Example get_meta output
-
-Note that this output is a little longer due to the additional details from the CLIP models.
-
-
-
-
-Next, you will create a corresponding object collection and import the data.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-See sample text data
- -| | backdrop_path | genre_ids | id | original_language | original_title | overview | popularity | poster_path | release_date | title | video | vote_average | vote_count | -|---:|:---------------------------------|:----------------|-----:|:--------------------|:----------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------:|:---------------------------------|:---------------|:----------------------------|:--------|---------------:|-------------:| -| 0 | /3Nn5BOM1EVw1IYrv6MsbOS6N1Ol.jpg | [14, 18, 10749] | 162 | en | Edward Scissorhands | A small suburban town receives a visit from a castaway unfinished science experiment named Edward. | 45.694 | /1RFIbuW9Z3eN9Oxw2KaQG5DfLmD.jpg | 1990-12-07 | Edward Scissorhands | False | 7.7 | 12305 | -| 1 | /sw7mordbZxgITU877yTpZCud90M.jpg | [18, 80] | 769 | en | GoodFellas | The true story of Henry Hill, a half-Irish, half-Sicilian Brooklyn kid who is adopted by neighbourhood gangsters at an early age and climbs the ranks of a Mafia family under the guidance of Jimmy Conway. | 57.228 | /aKuFiU82s5ISJpGZp7YkIr3kCUd.jpg | 1990-09-12 | GoodFellas | False | 8.5 | 12106 | -| 2 | /6uLhSLXzB1ooJ3522ydrBZ2Hh0W.jpg | [35, 10751] | 771 | en | Home Alone | Eight-year-old Kevin McCallister makes the most of the situation after his family unwittingly leaves him behind when they go on Christmas vacation. But when a pair of bungling burglars set their sights on Kevin's house, the plucky kid stands ready to defend his territory. By planting booby traps galore, adorably mischievous Kevin stands his ground as his frantic mother attempts to race home before Christmas Day. | 3.538 | /onTSipZ8R3bliBdKfPtsDuHTdlL.jpg | 1990-11-16 | Home Alone | False | 7.4 | 10599 | -| 3 | /vKp3NvqBkcjHkCHSGi6EbcP7g4J.jpg | [12, 35, 878] | 196 | en | Back to the Future Part III | The final installment of the Back to the Future trilogy finds Marty digging the trusty DeLorean out of a mineshaft and looking for Doc in the Wild West of 1885. But when their time machine breaks down, the travelers are stranded in a land of spurs. More problems arise when Doc falls for pretty schoolteacher Clara Clayton, and Marty tangles with Buford Tannen. | 28.896 | /crzoVQnMzIrRfHtQw0tLBirNfVg.jpg | 1990-05-25 | Back to the Future Part III | False | 7.5 | 9918 | -| 4 | /3tuWpnCTe14zZZPt6sI1W9ByOXx.jpg | [35, 10749] | 114 | en | Pretty Woman | When a millionaire wheeler-dealer enters a business contract with a Hollywood hooker Vivian Ward, he loses his heart in the bargain. | 97.953 | /hVHUfT801LQATGd26VPzhorIYza.jpg | 1990-03-23 | Pretty Woman | False | 7.5 | 7671 | - -
-
-
-Query image
- - - -
-
-
-
-
-
-
-Weaviate output:
-
-```text
-Interstellar 2014 157336
-Distance to query: 0.354
-
-Gravity 2013 49047
-Distance to query: 0.384
-
-Arrival 2016 329865
-Distance to query: 0.386
-
-Armageddon 1998 95
-Distance to query: 0.400
-
-Godzilla 1998 929
-Distance to query: 0.441
-```
-
-
-
-### Response object
-
-The returned object is an instance of a custom class. Its `objects` attribute is a list of search results, each object being an instance of another custom class.
-
-Each returned object will:
-- Include all properties and its UUID by default except those with blob data types.
- - Since the `poster` property is a blob, it is not included by default.
- - To include the `poster` property, you must specify it and the other properties to fetch in the `return_properties` parameter.
-- Not include any other information (e.g. references, metadata, vectors.) by default.
-
-
-## Text search
-
-### Code
-
-This example finds entries in "MovieMM" based on their similarity to the query "red", and prints out the title and release year of the top 5 matches.
-
-Example results
- -Posters for the top 5 matches: -
-
-
-
-
-
-Weaviate output:
-
-```text
-Interstellar 2014 157336
-Distance to query: 0.354
-
-Gravity 2013 49047
-Distance to query: 0.384
-
-Arrival 2016 329865
-Distance to query: 0.386
-
-Armageddon 1998 95
-Distance to query: 0.400
-
-Godzilla 1998 929
-Distance to query: 0.441
-```
-
-
-
-
-
-
-
-
-Weaviate output:
-
-```text
-Deadpool 2 2018 383498
-Distance to query: 0.670
-
-Bloodshot 2020 338762
-Distance to query: 0.677
-
-Deadpool 2016 293660
-Distance to query: 0.678
-
-300 2007 1271
-Distance to query: 0.682
-
-The Hunt for Red October 1990 1669
-Distance to query: 0.683
-```
-
-
-
-### Response object
-
-The returned object is in the same format as in the previous example.
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -Posters for the top 5 matches: -
-
-
-
-
-
-Weaviate output:
-
-```text
-Deadpool 2 2018 383498
-Distance to query: 0.670
-
-Bloodshot 2020 338762
-Distance to query: 0.677
-
-Deadpool 2016 293660
-Distance to query: 0.678
-
-300 2007 1271
-Distance to query: 0.682
-
-The Hunt for Red October 1990 1669
-Distance to query: 0.683
-```
-
-
-
-
-
-## Hybrid search
-
-### Code
-
-This example finds entries in "MovieMM" with the highest hybrid search scores for the term "history", and prints out the title and release year of the top 5 matches.
-
-Example results
- -```text -American History X 1998 -BM25 score: 2.707 - -A Beautiful Mind 2001 -BM25 score: 1.896 - -Legends of the Fall 1994 -BM25 score: 1.663 - -Hacksaw Ridge 2016 -BM25 score: 1.554 - -Night at the Museum 2006 -BM25 score: 1.529 -``` - -
-
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Legends of the Fall 1994 -Hybrid score: 0.016 - -Hacksaw Ridge 2016 -Hybrid score: 0.016 - -A Beautiful Mind 2001 -Hybrid score: 0.015 - -The Butterfly Effect 2004 -Hybrid score: 0.015 - -Night at the Museum 2006 -Hybrid score: 0.012 -``` - -
-
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Dune 2021 -Distance to query: 0.199 - -Tenet 2020 -Distance to query: 0.200 - -Mission: Impossible - Dead Reckoning Part One 2023 -Distance to query: 0.207 - -Onward 2020 -Distance to query: 0.214 - -Jurassic World Dominion 2022 -Distance to query: 0.216 -``` - -
-
-
-### Response object
-
-Each response object is similar to that from a regular search query, with an additional `generated` attribute. This attribute will contain the generated output for each object.
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Interstellar -Interstellaire -Gravity -Gravité -Arrival -Arrivée -Armageddon -Armageddon -Godzilla -Godzilla -``` - -
-
-
-### Optional parameters
-
-You can also pass on a list of properties to be used, as the `grouped_properties` parameter. This can be useful to reduce the amount of data passed on to the large language model and omit irrelevant properties.
-
-### Response object
-
-A RAG query with the `grouped_task` parameter will return a response with an additional `generated` attribute. This attribute will contain the generated output for the set of objects.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Interstellar -Gravity -Arrival -Armageddon -Godzilla -These movies all involve space exploration, extraterrestrial beings, or catastrophic events threatening Earth. They all deal with themes of survival, human ingenuity, and the unknown mysteries of the universe. -``` - -
- Example
-
-
-
-### Close the connection
-
-After you have finished using the Weaviate client, you should close the connection. This frees up resources and ensures that the connection is properly closed.
-
-We suggest using a `try`-`finally` block as a best practice. For brevity, we will not include the `try`-`finally` blocks in the remaining code snippets.
-
-Example get_meta output
-
-
-
-
-Next, you will create a corresponding object collection and import the data.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-See sample data
- -| | backdrop_path | genre_ids | id | original_language | original_title | overview | popularity | poster_path | release_date | title | video | vote_average | vote_count | -|---:|:---------------------------------|:----------------|-----:|:--------------------|:----------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------:|:---------------------------------|:---------------|:----------------------------|:--------|---------------:|-------------:| -| 0 | /3Nn5BOM1EVw1IYrv6MsbOS6N1Ol.jpg | [14, 18, 10749] | 162 | en | Edward Scissorhands | A small suburban town receives a visit from a castaway unfinished science experiment named Edward. | 45.694 | /1RFIbuW9Z3eN9Oxw2KaQG5DfLmD.jpg | 1990-12-07 | Edward Scissorhands | False | 7.7 | 12305 | -| 1 | /sw7mordbZxgITU877yTpZCud90M.jpg | [18, 80] | 769 | en | GoodFellas | The true story of Henry Hill, a half-Irish, half-Sicilian Brooklyn kid who is adopted by neighbourhood gangsters at an early age and climbs the ranks of a Mafia family under the guidance of Jimmy Conway. | 57.228 | /aKuFiU82s5ISJpGZp7YkIr3kCUd.jpg | 1990-09-12 | GoodFellas | False | 8.5 | 12106 | -| 2 | /6uLhSLXzB1ooJ3522ydrBZ2Hh0W.jpg | [35, 10751] | 771 | en | Home Alone | Eight-year-old Kevin McCallister makes the most of the situation after his family unwittingly leaves him behind when they go on Christmas vacation. But when a pair of bungling burglars set their sights on Kevin's house, the plucky kid stands ready to defend his territory. By planting booby traps galore, adorably mischievous Kevin stands his ground as his frantic mother attempts to race home before Christmas Day. | 3.538 | /onTSipZ8R3bliBdKfPtsDuHTdlL.jpg | 1990-11-16 | Home Alone | False | 7.4 | 10599 | -| 3 | /vKp3NvqBkcjHkCHSGi6EbcP7g4J.jpg | [12, 35, 878] | 196 | en | Back to the Future Part III | The final installment of the Back to the Future trilogy finds Marty digging the trusty DeLorean out of a mineshaft and looking for Doc in the Wild West of 1885. But when their time machine breaks down, the travelers are stranded in a land of spurs. More problems arise when Doc falls for pretty schoolteacher Clara Clayton, and Marty tangles with Buford Tannen. | 28.896 | /crzoVQnMzIrRfHtQw0tLBirNfVg.jpg | 1990-05-25 | Back to the Future Part III | False | 7.5 | 9918 | -| 4 | /3tuWpnCTe14zZZPt6sI1W9ByOXx.jpg | [35, 10749] | 114 | en | Pretty Woman | When a millionaire wheeler-dealer enters a business contract with a Hollywood hooker Vivian Ward, he loses his heart in the bargain. | 97.953 | /hVHUfT801LQATGd26VPzhorIYza.jpg | 1990-03-23 | Pretty Woman | False | 7.5 | 7671 | - -
-
-
-### Response object
-
-The returned object is an instance of a custom class. Its `objects` attribute is a list of search results, each object being an instance of another custom class.
-
-Each returned object will:
-- Include all properties and its UUID by default except those with blob data types.
-- Not include any other information (e.g. references, metadata, vectors.) by default.
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -In Time 2011 -Distance to query: 0.179 - -Gattaca 1997 -Distance to query: 0.180 - -I, Robot 2004 -Distance to query: 0.182 - -Mad Max: Fury Road 2015 -Distance to query: 0.190 - -The Maze Runner 2014 -Distance to query: 0.193 -``` - -
-
-
-
-## Hybrid search
-
-### Code
-
-This example finds entries in "Movie" with the highest hybrid search scores for the term "history", and prints out the title and release year of the top 5 matches.
-
-Example results
- -```text -American History X 1998 -BM25 score: 2.707 - -A Beautiful Mind 2001 -BM25 score: 1.896 - -Legends of the Fall 1994 -BM25 score: 1.663 - -Hacksaw Ridge 2016 -BM25 score: 1.554 - -Night at the Museum 2006 -BM25 score: 1.529 -``` - -
-
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Legends of the Fall 1994 -Hybrid score: 0.016 - -Hacksaw Ridge 2016 -Hybrid score: 0.016 - -A Beautiful Mind 2001 -Hybrid score: 0.015 - -The Butterfly Effect 2004 -Hybrid score: 0.015 - -Night at the Museum 2006 -Hybrid score: 0.012 -``` - -
-
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -Dune 2021 -Distance to query: 0.199 - -Tenet 2020 -Distance to query: 0.200 - -Mission: Impossible - Dead Reckoning Part One 2023 -Distance to query: 0.207 - -Onward 2020 -Distance to query: 0.214 - -Jurassic World Dominion 2022 -Distance to query: 0.216 -``` - -
-
-
-### Response object
-
-Each response object is similar to that from a regular search query, with an additional `generated` attribute. This attribute will contain the generated output for each object.
-
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -In Time -À temps -Looper -Boucleur -I, Robot -Je, Robot -The Matrix -La Matrice -Children of Men -Les enfants des hommes -``` - -
-
-
-### Optional parameters
-
-You can also pass on a list of properties to be used, as the `grouped_properties` parameter. This can be useful to reduce the amount of data passed on to the large language model and omit irrelevant properties.
-
-### Response object
-
-A RAG query with the `grouped_task` parameter will return a response with an additional `generated` attribute. This attribute will contain the generated output for the set of objects.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Example results
- -```text -In Time -Looper -I, Robot -The Matrix -Children of Men -These movies all involve futuristic settings and explore themes related to the manipulation of time, technology, and the potential consequences of advancements in society. They also touch on issues such as inequality, control, and the impact of human actions on the future of humanity. -``` - -
-
-
-Steps to create a collection
- -We will create a simple object collection, with each object containing multiple properties. Each properties will contain the same text, but with different tokenization methods applied. - -
-
-
-
-## Impact on filters
-
-Now that we have added a set of objects to Weaviate, let's see how different tokenization methods impact filtered retrieval.
-
-Each filtered query will look something like this, wherein we filter the objects for a set of query strings.
-
-We'll set up a reusable function to filter objects based on a set of query strings. Remember that a filter is binary: it either matches or it doesn't.
-
-The function will return a list of matched objects, and print the matched objects for us to see.
-
-Steps to add objects
- -Now, we add objects to the collection, repeating text objects as properties. - -
-
-
-Note how the `word` tokenization was the only that consistently returned the matching title, unless the colon (`:`) was included in the query. This is because the `word` tokenization method treats the colon as a separator.
-
-Users may not be expected to include any punctuation in their queries, nor the exact capitalization. As a result, for a typical text filter usage, the `word` tokenization method is a good starting point.
-
-### "**A mouse**" vs "**mouse**" - stop words
-
-Here, we filter for variants of the phrase "computer mouse", where some queries include additional words.
-
-Now, take a look at the results.
-
-**Matches for `"computer mouse"`**
-
-| | `word` | `lowercase` | `whitespace` | `field` |
-|------------------------------|-----------|-------------|--------------|---------|
-| `"computer mouse"` | ✅ | ✅ | ✅ | ✅ |
-| `"a computer mouse"` | ✅ | ✅ | ✅ | ❌ |
-| `"the computer mouse:" ` | ✅ | ✅ | ✅ | ❌ |
-| `"blue computer mouse" ` | ❌ | ❌ | ❌ | ❌ |
-
-**Matches for `"a computer mouse"`**
-
-| | `word` | `lowercase` | `whitespace` | `field` |
-|------------------------------|-----------|-------------|--------------|---------|
-| `"computer mouse"` | ✅ | ✅ | ✅ | ❌ |
-| `"a computer mouse"` | ✅ | ✅ | ✅ | ✅ |
-| `"the computer mouse:" ` | ✅ | ✅ | ✅ | ❌ |
-| `"blue computer mouse" ` | ❌ | ❌ | ❌ | ❌ |
-
-Python query & output
- -
-
-
-The results indicate that adding the word "a" or "the" to the query does not impact the filter results for all methods except `field`. This is because at every tokenization method, the word "a" or "the" is considered a stop word and is ignored.
-
-With the `field` method, the difference is that stop word tokens like "a" or "the" are never produced. An input "a computer mouse" is tokenized to `["a computer mouse"]`, containing one token.
-
-Adding another word, such as "blue", that is not a stop word, causes the query to not match any objects.
-
-### "**variable_name**" vs "**variable name**" - symbols
-
-The `word` tokenization is a good default. However, it may not always be the best choice. Take a look at this example where we filter for different variants of `"variable_name"`, to see if they match the object with the exact string (`"variable_name"`).
-
-| | `word` | `lowercase` | `whitespace` | `field` |
-|------------------------------|-----------|-------------|--------------|---------|
-| `"variable_name"` | ✅ | ✅ | ✅ | ✅ |
-| `"Variable_Name:" ` | ✅ | ✅ | ❌ | ❌ |
-| `"Variable Name:" ` | ✅ | ❌ | ❌ | ❌ |
-| `"a_variable_name"` | ✅ | ❌ | ❌ | ❌ |
-| `"the_variable_name"` | ✅ | ❌ | ❌ | ❌ |
-| `"variable_new_name" ` | ✅ | ❌ | ❌ | ❌ |
-
-Python query & output
- -
-
-
-What is the desired behavior here? Should a filter for `"variable name"` match the object with the property `"variable_name"`?
-
-What about a filter for `"variable_new_name"`? If the goal is to look through, say, a code base, the user might not expect a filter for `"variable_new_name"` to match `"variable_name"`.
-
-In cases such as these, where symbols are important to your data, you should consider using a tokenization method that preserves symbols, such as `lowercase` or `whitespace`.
-
-## Discussions
-
-We've discussed how different tokenization methods impact filters.
-
-For most filtering use, the `word` tokenization method is a good starting point. It is case-insensitive, and treats most symbols as separators.
-
-However, if symbols are important to your data, or if you need to distinguish between different cases, you may want to consider using a different tokenization method.
-
-And what about `field` tokenization? This method is most useful when you have text that should be treated as a single token. This is useful for properties like email addresses, URLs, or identifiers.
-
-A typical filtering strategy with a `field` tokenization method might involve exact matches, or partial matches with wildcards. Do note, however, that wildcard-based filters can be computationally expensive (slow) - so use them judiciously.
-
-Next, we'll discuss how tokenization impacts keyword searches.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Python query & output
- -
-
-
-Here, the same results are returned as in the filter example. However, note differences in the scores.
-
-For example, a search for `"lois clark"` returns a higher score than a search for `"clark"`. This is because the BM25f algorithm considers the number of matching tokens. So, it would be beneficial to include as many matching tokens as possible in the search query.
-
-Another difference is that a keyword search will return objects that match any of the tokens in the query. This is different from a filter, which is sensitive to the filtering operator. Depending on the desired result, you could use an `"Equal"` operator, `"ContainsAny"`, or `"ContainsAll"`, for example.
-
-The next section will demonstrate this, as well as how stop words are treated.
-
-#### "**A mouse**" vs "**mouse**" - stop words
-
-Here, we search for variants of the phrase "computer mouse", where some queries include additional words.
-
-Now, take a look at the results.
-
-**Matches for `"computer mouse"`**
-
-| | `word` | `lowercase` | `whitespace` | `field` |
-|------------------------------|-----------|-------------|--------------|---------|
-| `"computer mouse"` | 0.889 | 0.819 | 1.01 | 0.982 |
-| `"Computer Mouse"` | 0.889 | 0.819 | ❌ | ❌ |
-| `"a computer mouse"` | 0.764 | 0.764 | 0.849 | ❌ |
-| `"computer mouse pad" ` | 0.764 | 0.764 | 0.849 | ❌ |
-
-**Matches for `"a computer mouse"`**
-
-| | `word` | `lowercase` | `whitespace` | `field` |
-|------------------------------|-----------|-------------|--------------|---------|
-| `"computer mouse"` | 0.889 | 0.819 | 1.01 | ❌ |
-| `"Computer Mouse"` | 0.889 | 0.819 | ❌ | ❌ |
-| `"a computer mouse"` | 0.764 | 1.552 | 1.712 | 0.982 |
-| `"computer mouse pad" ` | 0.764 | 0.688 | 0.849 | ❌ |
-
-Python query & output
- -
-
-
-The results here are similar to the filter example, but more nuanced and quite interesting!
-
-Under `word` tokenization, the search for `computer mouse` produces identical results to the search for `a computer mouse`. This is because the stop word `a` is not considered in the search.
-
-But note that the scores are different for returned objects where the only differences are stopwords, such as `"computer mouse"` and `"a computer mouse"`. This is because the BM25f algorithm does [index stopwords](../../../weaviate/config-refs/indexing/inverted-index.mdx#stopwords), and they do impact the score.
-
-As a user, you should keep this in mind, and you can configure the stop words in the collection definition to suit your desired behavior.
-
-Another interesting note is that the `lowercase` and `whitespace` tokenization methods do not remove stop words in the query.
-
-This behavior allows users who want to include stop words in their search queries to do so.
-
-#### "**variable_name**" vs "**variable name**" - symbols
-
-The table below shows keyword search results using the string `"variable_name"` and the resulting scores.
-
-| | `word` | `lowercase` | `whitespace` | `field` |
-|------------------------------|-----------|-------------|--------------|---------|
-| `"variable_name"` | 0.716 | 0.97 | 1.27 | 0.982 |
-| `"Variable_Name:" ` | 0.716 | 0.97 | ❌ | ❌ |
-| `"Variable Name:" ` | 0.716 | ❌ | ❌ | ❌ |
-| `"a_variable_name"` | 0.615 | ❌ | ❌ | ❌ |
-| `"the_variable_name"` | 0.615 | ❌ | ❌ | ❌ |
-| `"variable_new_name" ` | 0.615 | ❌ | ❌ | ❌ |
-
-Python query & output
- -
-
-
-These results are once again similar to the filter example. If your data contains symbols that are important to your search, you should consider using a tokenization method that preserves symbols, such as `lowercase` or `whitespace`.
-
-### Discussions
-
-That's it for keyword searches and tokenization. Similarly to filters, the choice of tokenization method is a big part of your overall search strategy.
-
-Our generally advice for tokenization in keyword searching is similar to [our advice for filtering](./300_filters.mdx#-discussions). Start with `word`, and consider others such as `lowercase` or `whitespace` if symbols, or cases encode important information in your data.
-
-Using `field` tokenization may be too strict for keyword searches, as it will not match any
-objects that do not contain the exact string in the exact order.
-
-Lastly, keep in mind that keyword searches produce ranked results. Therefore, tokenization will not only affect the results set but also their ranking within the set.
-
-With these considerations in mind, you can configure your tokenization strategy to best suit your data and your users' needs.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Python query & output
- -
- 
-
-
-A [vector index](/weaviate/concepts/indexing/vector-index.md) is a specialized type of index that is designed to store and search vectors.
-
-The choice and configuration of your vector index can significantly impact the performance of your imports as well as searches, and the resource requirements of your Weaviate instance.
-
-For this reason, the vector index a critical component in Weaviate.
-
-## Why use a vector index?
-
-Databases can quickly grow very large, to the point where time to search for a specific item can become unacceptably long, or the resources required to search become too high.
-
-A vector index is designed to improve users' experience of searching for items in a large database.
-
-It usually makes a trade-off to balance three key factors: search speed, accuracy, and resource requirements.
-
-## Vector index types
-
-Many different types of vector indexes exist. A majority of them are designed to speed up searches by reducing the number of vectors that need to be compared. However, they do this in different ways, and each has its own strengths and weaknesses.
-
-### Graph indexes
-
-Graph indexes form a network of vectors, such that similar vectors are connected to each other. This allows for fast "traversal" of the graph to find similar vectors to a query vector.
-
-import HNSWDiagram from './_snippets/hnsw_diagram.png';
-
-
-
- 
-
-
-HNSW, or "Hierarchical Navigable Small World", is the most common graph index type. It creates a set of "layers" of vectors, to enable fast traversal of the graph.
-
-They are very scalable, allow incremental updates, and efficient for high-dimensional vectors.
-
-This is the default index type in Weaviate.
-
-### Tree-based indexes
-
-Tree-based indexes divide the vectors into a tree structure.
-
-
- 
-
-
-ANNOY, or "Approximate Nearest Neighbors Oh Yeah", is a well-known tree-based index. It divides the vectors into a binary tree structure.
-
-They can be memory-efficient, and are good for low-dimensional vectors.
-
-Given its nature as a tree, it may be costly to update the index over time. This would depend on whether the tree needs to be rebuilt, or if it can be updated incrementally.
-
-ANNOY itself does not support incremental updates.
-
-### Cluster-based indexes
-
-Cluster-based indexes group vectors based on their similarity. As a result, the search space is reduced to only the cluster(s) that is most likely to contain the nearest neighbors.
-
-Their search accuracy (recall and precision) may generally be lower than graph-based indexes, but they can be more memory-efficient.
-
-### Flat index
-
-A flat index is the simplest type of index. It stores all vectors in a single list, and searches through all of them to find the nearest neighbors.
-
-This is extremely memory-efficient, but does not scale well at all, as the search time grows linearly with the number of vectors.
-
-### Available vector indexes in Weaviate
-
-Weaviate supports multiple types of vector indexes - namely, `hnsw`, `flat`, and `dynamic`.
-
-We will discuss these in more detail in the following sections.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-
-
-
-
-
-This method allows a search to quickly find the right general area of the graph before carrying out a comprehensive search of the dense bottom layer.
-
-Intuitively, an HNSW graph can be thought of as a high-dimensional skip list of sorts, where the higher layers are used to global search, and the lower layers are used for local search.
-
-### Trade-offs
-
-HNSW is a very flexible index type that allows for a wide range of trade-offs.
-
-The key trade-offs are between search speed, accuracy, and resource requirements. These trade-offs can be made by tuning the parameters of the HNSW index, as well as by [quantizing the vectors](../compression/index.md).
-
-### Resource requirements
-
-HNSW is an in-memory index, where each node in the graph as well as each edge between nodes are stored in memory.
-
-This means that the size of the index in memory is directly proportional to the number of vectors in the index, as well as the number of connections between vectors.
-
-The size of an HNSW index is dominated by the number of vectors; take a look at the table below for an example:
-
-| Component | Size derivation | Typical size | Size @1M vectors | Size @100M vectors |
-| --- | --- | --- | --- | --- |
-| Node | 4B (float) x N dimensions | 2-12kB | 2-12GB | 200-1200GB |
-| Edge | 10B x 20 connections | 200B | 200MB | 20GB |
-
-As you can see, the memory requirements of an HNSW index can quickly become a bottleneck. This is where [quantization](../compression/index.md) can be used to reduce the size of the index in memory.
-
-### Distance metric
-
-
-
-The distance metric used in the index determines how the distance between vectors is calculated. In an HNSW index, it impacts where each vector is placed in the graph.
-
-You must choose a metric that suits the vectors in your collection. To find this, consult the documentation of the model that generated your vectors.
-
-Weaviate's default metric is cosine, but you can also use any number of [other available metrics](/weaviate/config-refs/distances.md).
-
-If you are unsure, the cosine distance is a good, robust, default choice that is used by a majority of models.
-
-### Specify HNSW as the index type
-
-HNSW is Weaviate's default vector index type. So, if you do not specify a collection to use a specific index type, it will use HNSW.
-
-But you can explicitly specify it as follows:
-
-
- 
-
-
-The maximum number of connections between nodes (`maxConnections`) determine how densely the graph is connected.
-
-A higher number of connections will allow for more accurate searches, but will also slow down searches, and require more memory.
-
-The default value is `32`. Note that on the bottom layer of the graph each node can have up to (`2 * maxConnections`) connections.
-
-### Dynamic list size
-
-The "dynamic list" in HNSW refers to the list of nodes that are considered by the algorithm. Note that dynamic lists are used in two different contexts:
-
-During search, the dynamic list is used to keep track of the nodes that are being considered, and to ensure that the search is comprehensive.
-
-During index construction, the dynamic list is used to keep track of candidate nodes that are being considered for connection. The HNSW algorithm will then choose the best `maxConnections` connections from the dynamic list, taking into account not only proximity but also aspects such as overall connectivity of the graph.
-
-#### Search dynamic list size
-
-import EfDiagram from './_snippets/ef.png';
-
-
- 
-
-
-You can set the dynamic list size for search statically or dynamically.
-
-To set it statically, provide the `ef` parameter when creating the collection. The default value is `-1`, which defers this to a dynamic setting.
-
-To set it dynamically, provide a combination of `dynamicEfMin`, `dynamicEfMax` and `dynamicEfFactor`.
-
-The dynamic list size will be set as the query limit multiplied by `dynamicEfFactor`, modified by a minimum of `dynamicEfMin` and a maximum of `dynamicEfMax`.
-
-In code, this can be expressed as:
-
-```python
-ef = min(max(dynamicEfMin, queryLimit * dynamicEfFactor), dynamicEfMax)
-```
-
-The default values are `dynamicEfMin=100`, `dynamicEfMax=500`, and `dynamicEfFactor=8`.
-
-#### Index construction dynamic list size
-
-import EfConstructionDiagram from './_snippets/efConstruction.png';
-
-
- 
-
-
-To set the dynamic list size for index construction, provide the `efConstruction` parameter when creating the collection.
-
-This will improve your search performance, at a cost of the index construction process. The default value is `128`.
-
-### Quantization
-
-Enabling quantization with HNSW reduces the size of the index in memory by using compressed vectors. Note that the full vector is still stored on disk, which is used to rescore the vectors after they are fetched from the index.
-
-This can be a powerful way to reduce the memory requirements of your Weaviate instance, especially if you have a large number of vectors.
-
-Learn more about quantization [with this Weaviate Academy course](../compression/index.md).
-
-## Configure HNSW in Weaviate
-
-Each of these parameters can be provided when creating a collection in Weaviate. Note that out of the discussed parameters, only the `dynamicEf` related parameters are mutable.
-
-### Code example
-
-- -Weaviate is an open-source [vector database](https://weaviate.io/blog/what-is-a-vector-database). But what does that mean? Let's unpack it here. - -### Vector database - -Weaviate is a fantastic tool for retrieving the information you need, quickly and accurately. It does this by being an amazing **vector database**. - -You may be familiar with traditional databases such as relational databases that use SQL. A database can catalog, store and retrieve information. A **vector** database can carry out these tasks also, with the key difference being that they can perform these tasks based on similarity. - -#### How traditional searches work - -Imagine that you are searching a relational database containing articles on cities, to retrieve a list of "major" European cities. Using SQL, you might construct a query like this: - -```sql -SELECT city_name wiki_summary -FROM wiki_city -WHERE (wiki_summary LIKE '%major European city%' OR - wiki_summary LIKE '%important European city%' OR - wiki_summary LIKE '%prominent European city%' OR - wiki_summary LIKE '%leading European city%' OR - wiki_summary LIKE '%significant European city%' OR - wiki_summary LIKE '%top European city%' OR - wiki_summary LIKE '%influential European city%' OR - wiki_summary LIKE '%notable European city%') - (… and so on) -``` - -Which would return cities that contained any of these strings (`major`, `important`, `prominent`, ... etc) in the `wiki_summary` column. - -This works well in many circumstances. However, there are two significant limitations with this approach. - -#### Limitations of traditional search - -Using this type of search requires you to identify terms that *may* have been used to describe the concept, which is no easy feat. - -What's more, this doesn't solve the problem of how to rank the list of resulting objects. - -With the above search query, an entry merely containing a mention of a different European city (i.e. not very relevant) would be given equal weighting to an entry for Paris, or Rome, which would be highly relevant. - -A vector database makes this job simpler by enabling searches based on similarity. - -#### Examples of vector search - -So, you could perform a query like this in Weaviate: - -```graphql -{ - Get { - WikiCity ( - nearText: { concepts: ["Major European city"] } - ) { city_name wiki_summary } - } -} -``` - -And it would return a list of entries that are *ranked by their similarity* to the query - the idea of "Major European city". - -What's more, Weaviate "indexes" the data based on their similarity, making this type of data retrieval lightning-fast. - -Weaviate can help you to do all this, and actually a lot more. Another way to think about Weaviate is that it supercharges the way you use information. - -:::info Vector vs semantic search -A vector search is also referred to as a "semantic search" because it returns results based on the similarity of meaning (therefore "semantic"). -::: - -### Open-source - -Weaviate is open-source. In other words, its [codebase is available online](https://github.com/weaviate/weaviate) for anyone to see and use[(1)](#-notes). - -And that is *the* codebase, regardless of how you use it. So whether you run Weaviate on your own computer, on a cloud computing environment, or through our managed service [Weaviate Cloud (WCD)](https://console.weaviate.cloud/), you are using the exact same technology. - -So, if you want, you can run Weaviate for free on your own device, or use our managed service for convenience. You can also take comfort in that you can see exactly what you are running, and be a part of the open-source community, as well as to shape its development. - -It also means that your knowledge about Weaviate is fungible, between local, cloud, or managed instances of Weaviate. So anything you learn here about Weaviate using WCD will be equally applicable to running it locally, and vice versa. 😉 - -### Information, made dynamic - -We are used to thinking of information as static, like a book. But with Weaviate and modern AI-driven language models, we can do much more than just retrieve static information but easily build on top of it. Take a look at these examples: - -#### Question answering - -Given a list of Wikipedia entries, you could ask Weaviate: - -:::note We asked Weaviate: -When was Lewis Hamilton born? -::: - -And it would answer with: - -:::note Weaviate responded: -Lewis Hamilton was born on January 7, 1985. ([check for yourself](https://en.wikipedia.org/wiki/Lewis_Hamilton)) -::: - -
-
-
-#### Generative search
-
-Or you can synthesize passages using retrieved information with Weaviate:
-
-Here is one, where we searched Weaviate for an entry on a "racing driver", and produce the result in the format of:
-
-:::note We asked Weaviate:
-Write a fun tweet encouraging people to read about this: ## \{title} by summarizing highlights from: ## \{wiki_summary}
-:::
-
-Which produces:
-
-:::note Weaviate responded:
-Check out the amazing story of Lewis Hamilton, the 7-time Formula One World Drivers' Championship winner! From his humble beginnings to becoming one of the world's most influential people, his journey is an inspiring one. #LewisHamilton #FormulaOne #Motorsport #Racing
-:::
-
-See the full query & response
- -#### Query - -```graphql -{ - Get { - WikiArticle ( - ask: { - question: "When was Lewis Hamilton born?", - properties: ["wiki_summary"] - }, - limit: 1 - ) { - title - _additional { - answer { - result - } - } - } - } -} -``` - -#### Response - -```json -{ - "data": { - "Get": { - "WikiArticle": [ - { - "_additional": { - "answer": { - "result": " Lewis Hamilton was born on January 7, 1985." - } - }, - "title": "Lewis Hamilton" - } - ] - } - } -} -``` - -
-
-
-
-We will cover these and many more capabilities, such as vectorization, summarization and classification, in our units.
-
-For now, keep in mind that Weaviate is a vector database at its core which can also leverage AI tools to do more with the retrieved information.
-
-## Review
-
-In this section, you learned about what Weaviate is and how it works at a very high level. You have also been introduced to what vector search is at a high level, that it is a similarity-based search method.
-
-### Review exercises
-
-See the full query & response
- -#### Query - -```graphql -{ - Get { - WikiArticle( - nearText: { - concepts: ["Racing Driver"] - } - limit: 1 - ) { - title - wiki_summary - _additional { - generate( - singleResult: { - prompt: """ - Write a fun tweet encouraging people to read about this: ## {title} - by summarizing highlights from: ## {wiki_summary} - """ - } - ) { - singleResult - error - } - } - } - } -} -``` - -#### Response - -```json -{ - "data": { - "Get": { - "WikiArticle": [ - { - "_additional": { - "generate": { - "error": null, - "singleResult": "Check out the amazing story of Lewis Hamilton, the 7-time Formula One World Drivers' Championship winner! From his humble beginnings to becoming a global icon, his journey is an inspiring one. #LewisHamilton #FormulaOne #Motorsport #Racing #Inspiration" - } - }, - "title": "Lewis Hamilton", - "wiki_summary": "Sir Lewis Carl Davidson Hamilton (born 7 January 1985) is a British racing driver currently competing in Formula One, driving for Mercedes-AMG Petronas Formula One Team. In Formula One, Hamilton has won a joint-record seven World Drivers' Championship titles (tied with Michael Schumacher), and holds the records for the most wins (103), pole positions (103), and podium finishes (191), among others.\nBorn and raised in Stevenage, Hertfordshire, Hamilton joined the McLaren young driver programme in 1998 at the age of 13, becoming the youngest racing driver ever to be contracted by a Formula One team. This led to a Formula One drive with McLaren for six years from 2007 to 2012, making Hamilton the first black driver to race in the series. In his inaugural season, Hamilton set numerous records as he finished runner-up to Kimi R\u00e4ikk\u00f6nen by one point. The following season, he won his maiden title in dramatic fashion\u2014making a crucial overtake at the last corner on the last lap of the last race of the season\u2014to become the then-youngest Formula One World Champion in history. After six years with McLaren, Hamilton signed with Mercedes in 2013.\nChanges to the regulations for 2014 mandating the use of turbo-hybrid engines saw the start of a highly successful period for Hamilton, during which he won six further drivers' titles. Consecutive titles came in 2014 and 2015 during an intense rivalry with teammate Nico Rosberg. Following Rosberg's retirement in 2016, Ferrari's Sebastian Vettel became Hamilton's closest rival in two championship battles, in which Hamilton twice overturned mid-season point deficits to claim consecutive titles again in 2017 and 2018. His third and fourth consecutive titles followed in 2019 and 2020 to equal Schumacher's record of seven drivers' titles. Hamilton achieved his 100th pole position and race win during the 2021 season. \nHamilton has been credited with furthering Formula One's global following by appealing to a broader audience outside the sport, in part due to his high-profile lifestyle, environmental and social activism, and exploits in music and fashion. He has also become a prominent advocate in support of activism to combat racism and push for increased diversity in motorsport. Hamilton was the highest-paid Formula One driver from 2013 to 2021, and was ranked as one of the world's highest-paid athletes by Forbes of twenty-tens decade and 2021. He was also listed in the 2020 issue of Time as one of the 100 most influential people globally, and was knighted in the 2021 New Year Honours. Hamilton was granted honorary Brazilian citizenship in 2022.\n\n" - } - ] - } - } -} -``` - -- -We've covered that Weaviate is a vector database, and that a vector search is similarity-based. But, what is a vector? - -A vector in this context is just a series of numbers like `[1, 0]` or `[0.513, 0.155, 0.983, ..., 0.001, 0.932]`. Vectors like these are used to capture meaning. - -This might seem like an odd concept. But in fact, you may have already used vectors to capture meaning without realizing it. If you have tried photo editing, or used MS Paint you might have encountered the RGB color system. - -### How do numbers represent meaning? - -The RGB system uses groups of three numbers to represent colors. For example: - -- (255, 0, 0) = red -- (80, 200, 120) = emerald - -In these examples, each number can be thought of as a dial for how red, green or blue a color is. - -Now, imagine having hundreds, or even thousands, of these dials. That’s how vectors are used to represent meaning. Modern machine learning models such as GPT-x, or those used with Weaviate, use vectors to represent some "essence", or "meaning" of objects. This can be done for any object type, such as text, code, images, videos and more. - -## Vector embeddings in Weaviate - - - -The vector representation of an object's meaning is called a "vector embedding". - -Weaviate enables vector searches by indexing and storing data objects and their corresponding vector embeddings. The vector embeddings come from machine learning models. - -In plain terms, Weaviate processes and organizes your data in such a way that objects can be retrieved based on their similarity to a query. To perform these tasks at speed, Weaviate does two things that traditional databases do not. Weaviate: - -- Quantifies similarity -- Indexes vector data - -These operations enable Weaviate to do what it does. - -### Quantifying similarity - -As we've mentioned, vector searches are similarity-based, but what does that actually mean? How do we determine that two pieces of data are "similar"? What does it mean for two pieces of text, two images, or two objects in general, to be similar? - -This is a relatively simple idea that is actually incredibly interesting and intricate once we start to dive into the details. - -But for now, you should know that machine learning (ML) models are the key to this whole process. The ML models that power vector searches share similarities with those that generate text responses from prompts. Instead of generating new text, these (vectorizer) models capture the "meaning" of text or other media. We will cover this in more detail later on. - -### Indexing (vector) data - -Vector searches can be very intensive computationally. - -To overcome this problem, Weaviate uses a combination of indexes including an approximate nearest neighbor (ANN) index and an inverted index. The ANN index lets Weaviate perform extremely fast vector searches. The inverted index lets Weaviate filter data using Boolean criteria. - -We will get into this in more detail later - but for now, it's enough to know that Weaviate can perform fast vector searches as well as filtering. - -## Review - -In this section, you learned about what vectors are and how Weaviate utilizes them at a very high level. You have also been introduced to two of Weaviate's key capabilities that help it to enable vector search at speed. - -### Review exercise - -:::tip Self-reflection -Can you describe, in your own words, what vectors are? -::: - -
- -Let's take a look at a few more examples of what you can do with Weaviate. - -First, we will try vector searches by searching through our demo database. You will learn how to use Weaviate to retrieve objects based on their similarity, using various query types such as an input text, vector, or object. - -You will also compare vector search with keyword search to compare and contrast the two techniques, before learning how to combine the two techniques through the use of filters. - -### Vector search demo - -For our first example, let's search our demo dataset. It contains a small sample of questions from the quiz show *Jeopardy!*. - -Imagine that you're running a quiz night, and you want to get some questions about "animals in movies". In a traditional database you could look for word matches, perhaps something like: - -```sql -SELECT question, answer -FROM jeopardy_questions -WHERE ( - question LIKE '%animal%' - OR question LIKE '%creature%' - OR question LIKE '%beast%' - ) -AND ( - question LIKE '%movie%' - OR question LIKE '%film%' - OR question LIKE '%picture%' - OR question LIKE '%cinema%' - ) - -``` - -This is a difficult query to write. Even worse, you would probably have to add the names of specific animals to the query as well. - -The Weaviate query is much more intuitive. See what happens when we run the following query: - -:::note We searched Weaviate for: -animals in movies -::: - -
-
-
-Weaviate retrieved these as the top answers:
-
-:::note Weaviate retrieved:
-- **meerkats**: Group of mammals seen here like Timon in *The Lion King*
-- **dogs**: Scooby-Doo, Goofy & Pluto are cartoon versions
-- **The Call of the Wild Thornberrys**: Jack London story about the dog Buck who joins a Nick cartoon about Eliza, who can talk to animals
-:::
-
-See the full query
- -```graphql -{ - Get { - JeopardyQuestion ( - nearText: { - concepts: ["animals in movies"] - } - limit: 3 - ) { - question - answer - } - } -} -``` - -
-
-
-Note how relevant the results are, despite none of them including the word "animal" or the word "movie", let alone both!
-
-This is exactly why vector searches are so useful. They can identify related objects without the need to match exact text.
-
-### Vector similarities demo
-
-If we run *this* query, you might expect to see responses like the ones we saw earlier.
-
-```graphql
-{
- Get {
- JeopardyQuestion (
- nearText: {
- concepts: ["European geography"]
- }
- limit: 3
- ) {
- question
- answer
- _additional {
- distance
- }
- }
- }
-}
-```
-
-But, take a look at this response. Do you notice any additional information?
-
-```json
-{
- "data": {
- "Get": {
- "JeopardyQuestion": [
- {
- "_additional": {
- "distance": 0.15916324
- },
- "answer": "Bulgaria",
- "question": "A European republic: Sofia"
- },
- ...
- ]
- }
- }
-}
-
-```
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "answer": "meerkats", - "question": "Group of mammals seen here: [like Timon in The Lion King]" - }, - { - "answer": "dogs", - "question": "Scooby-Doo, Goofy & Pluto are cartoon versions" - }, - { - "answer": "The Call of the Wild Thornberrys", - "question": "Jack London story about the dog Buck who joins a Nick cartoon about Eliza, who can talk to animals" - } - ] - } - } -} -``` - -
-
-
-The difference is that the response contains a `distance` value.
-
-A `distance` is indicative of the degree of similarity between the returned object and the query.
-
-If you're wondering exactly what that means, and who decides how similar any two objects or concepts are, those are great questions! We will cover those in more detail later.
-
-For now, just keep in mind that smaller distances mean two objects are more similar to each other.
-
-## Review
-
-
-
-
-
-### Key takeaways
-
-- Vector searches can identify related objects without the need for exact text matches.
-- In vector searches, distance values indicate the degree of similarity between the returned object and the query.
-- Smaller distances indicate greater similarity.
-- Vector searches can be combined with keyword searches and filtering techniques for more refined search results.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-See the full JSON response from Weaviate
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "_additional": { - "distance": 0.15916324 - }, - "answer": "Bulgaria", - "question": "A European republic: Sofia" - }, - { - "_additional": { - "distance": 0.16247147 - }, - "answer": "Balkan Peninsula", - "question": "The European part of Turkey lies entirely on this peninsula" - }, - { - "_additional": { - "distance": 0.16832423 - }, - "answer": "Mediterranean Sea", - "question": "It's the only body of water with shores on the continents of Asia, Africa & Europe" - } - ] - } - } -} - -``` - -- -You can do a lot more with Weaviate than simply retrieve static information. - -Let's take a look at a couple of examples where we do more than simply retrieve objects from the database. - -We will extract information from this Wikipedia entry. - -
-
-
-Weaviate creates [data objects](/weaviate/concepts/data) when it processes the Wikipedia entry. The data objects are stored in classes. A class is roughly analogous to a table in a relational database. An object is similar to an entry in that table.
-
-### Question-answering demo
-
-Weaviate can extract knowledge from the text.
-
-```graphql
-{
- Get {
- WikiArticle (
- ask: {
- question: "When did construction for the Sydney Opera House start?",
- properties: ["wiki_summary"]
- },
- limit: 1
- ) {
- title
- _additional {
- answer {
- hasAnswer
- property
- result
- startPosition
- endPosition
- }
- }
- }
- }
-}
-```
-
-Given this query, Weaviate doesn't just identify the most relevant data object. Weaviate also answers the question based on the textual information in the Wikipedia article.
-
-"The Sydney Opera House" Wikipedia summary
- -The Sydney Opera House is a multi-venue performing arts centre in Sydney. Located on the foreshore of Sydney Harbour, it is widely regarded as one of the world's most famous and distinctive buildings and a masterpiece of 20th-century architecture. Designed by Danish architect Jørn Utzon, but completed by an Australian architectural team headed by Peter Hall, the building was formally opened by Queen Elizabeth II on 20 October 1973 after a gestation beginning with Utzon's 1957 selection as winner of an international design competition. The Government of New South Wales, led by the premier, Joseph Cahill, authorised work to begin in 1958 with Utzon directing construction. The government's decision to build Utzon's design is often overshadowed by circumstances that followed, including cost and scheduling overruns as well as the architect's ultimate resignation. The building and its surrounds occupy the whole of Bennelong Point on Sydney Harbour, between Sydney Cove and Farm Cove, adjacent to the Sydney central business district and the Royal Botanic Gardens, and near to the Sydney Harbour Bridge. - -The building comprises multiple performance venues, which together host well over 1,500 performances annually, attended by more than 1.2 million people. Performances are presented by numerous performing artists, including three resident companies: Opera Australia, the Sydney Theatre Company and the Sydney Symphony Orchestra. As one of the most popular visitor attractions in Australia, the site is visited by more than eight million people annually, and approximately 350,000 visitors take a guided tour of the building each year. The building is managed by the Sydney Opera House Trust, an agency of the New South Wales State Government. - -On 28 June 2007, the Sydney Opera House became a UNESCO World Heritage Site, having been listed on the (now defunct) Register of the National Estate since 1980, the National Trust of Australia register since 1983, the City of Sydney Heritage Inventory since 2000, the New South Wales State Heritage Register since 2003, and the Australian National Heritage List since 2005. The Opera House was also a finalist in the New7Wonders of the World campaign list. - -
-
-
-### Generative search
-
-Weaviate can do even more with these entries. You can ask Weaviate to grab an object from its data store and use that object to generate new text. For example, Weaviate can use the object that contains the entry for the Sydney Opera House to derive new text.
-
-```graphql
-{
- Get {
- WikiArticle(
- nearText: {
- concepts: ["Sydney Opera House"]
- }
- limit: 1
- ) {
- title
- wiki_summary
- _additional {
- generate(
- singleResult: {
- prompt: """
- Write a fun tweet encouraging people to read about this: ## {title}
- by summarizing highlights from: ## {wiki_summary}
- """
- }
- ) {
- singleResult
- error
- }
- }
- }
- }
-}
-```
-
-The sample code generates a Tweet based on the Wikipedia entry!
-
-See response
- -:::note Weaviate says: -Construction for the Sydney Opera House started in 1958. -::: - -
-
-
-This process is an example of `generative search`. In a generative search, Weaviate retrieves information, and then leverages a large language model (LLM) to re-shape it. This is a powerful feature that can transform how you deal with information.
-
-You can vary the prompt to generate different results.
-
-### What next?
-
-Tools like Q&A and generative search really start to bring your information to life. In the next sections, you will set up Weaviate and run your own queries.
-
-## Review
-
-
-
-### Key takeaways
-
-- Weaviate can extract knowledge from text using question-answering capabilities, identifying the most relevant object and the actual answer based on the provided text.
-- Generative search allows you to retrieve information and reshape or repurpose the content, such as generating a tweet based on a Wikipedia entry.
-- These advanced capabilities of Weaviate transform how you interact with and utilize information in your data.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-See response
- -:::note Weaviate says: -Explore the world-famous Sydney Opera House and its incredible architecture! From the iconic design to the amazing performances, there's something for everyone to enjoy. #SydneyOperaHouse #Explore #Architecture #Performances #Experience -::: - -- -## Options for running Weaviate - -[Weaviate Cloud (WCD)](https://console.weaviate.cloud/) is a managed service that runs Weaviate for you. You can also manage your own Weaviate instances using Docker, Kubernetes, or Embedded Weaviate. - -No matter how you run Weaviate, the underlying code is identical. However, there are some operational differences to be aware of. - -### WCD (Recommended) - -[Weaviate Cloud (WCD)](https://console.weaviate.cloud/), is a managed SaaS service that requires no maintenance at your end. - -As it is managed by Weaviate (the company - the software is not sentient... *yet*). WCD is the fastest way to create a new instance of Weaviate and requires the least amount of effort for users. - -WCD has Weaviate instances that are pre-configured for your convenience. They include a number of Weaviate modules by default. They also have built-in support for user authentication. - -WCD includes a free "sandbox" tier. The WCD sandbox is our recommended method of running Weaviate in this course. - -### Docker and Kubernetes - -You can run Weaviate instances using containerization solutions such as [Docker](https://docs.docker.com/) and [Kubernetes](https://kubernetes.io/docs/home/). - -Running your own instance provides you with the same Weaviate code base as a WCD instance, but you have to manage configuration and deployment yourself. - -This course doesn't cover self-managed instances. We will cover Docker and Kubernetes in separate course units in the future. - -If you are familiar with either solution and want to use them to install Weaviate now, see the documentation for [Docker-Compose](/deploy/installation-guides/docker-installation.md) or [Kubernetes](/deploy/installation-guides/k8s-installation.md). - -### Embedded Weaviate - -We also have an experimental feature called [Embedded Weaviate](/deploy/installation-guides/embedded.md), where you can directly instantiate a Weaviate database from a client library. - -Currently, Embedded Weaviate is only recommended for evaluation purposes. - -## Get started with WCD - -### Sign in to WCD - -1. To access WCD, go to the [Weaviate Cloud Console](https://console.weaviate.cloud) - -1. If you don't have a WCD account, click on the "Register" button to create a new account. - -1. Click on "Sign in with the Weaviate Cloud" and sign in with your WCD username and password. - -### Create a Weaviate Cluster - -To create a new Weaviate Cluster, click the "Create cluster" button. - -On the Create a Cluster page: - -1. Select the **Free sandbox** plan tier. -1. Provide a *cluster name*. The sandbox URL is based on the cluster-name. WCD adds a suffix to ensure uniqueness. -1. Set the `Enable Authentication?` option to `YES`. -1. Press **Create** to create your sandbox instance. Note that the sandbox will expire after a set number of days. - -
-
-
-There are several ways to work with your sandbox.
-
-- [Client libraries](/weaviate/client-libraries/index.mdx)
-- [RESTful API](/weaviate/api/rest)
-- [GraphQL API](/weaviate/api/index.mdx)
-
-## Install Weaviate client
-
-:::info Academy material in Python
-For the initial release of Weaviate Academy units, our materials are written around Python examples.Sandbox expiration & options
- -import SandBoxExpiry from '/_includes/sandbox.expiry.mdx'; - --We are working to add examples for other client languages, starting with TypeScript. We appreciate your patience as we build up our educational material. -::: - -### Available clients - -Currently, Weaviate clients are available in: - -- Python -- TypeScript -- Java -- Go - -### Client capabilities - -import ClientCapabilitiesOverview from '/_includes/client.capabilities.mdx' - -
- -Now that you've set up your own Weaviate instance and installed a client, let's get hands-on with Weaviate. - -### Client instantiation - -Create a `client` object for persistent access to your Weaviate instance. You can set the following parameters: - -- **Host URL (required)** - - This is the location of your Weaviate instance. The URL resembles - - ``` - https://your-sandbox-uo2vgb1z.weaviate.network - ``` - -- **Authentication details (optional)** - - If authentication is enabled, you MUST provide your authentication information here. Otherwise the Weaviate instance will not provide access. - - ``` - AbCdEfGhIjKlMnOpAXB6gbBIaxHDuuwWj5OD - ``` -- **Additional headers (optional)** - - You can provide additional headers. These headers provide API keys for inference services such as Cohere, Hugging Face or OpenAI. - -A fully configured client resembles this sample code, edited to match your Weaviate instance. - - - -import ClientInstantiationCode from './_snippets/academy.hello.client.instantiation.mdx'; - -
-
-
-What is REST?
- -REST is an acronym for **RE**presentational **S**tate **T**ransfer. - -A REST API provides multiple endpoints, each with its own URL, that can be used to interact with the application. - -The endpoints are organized into a hierarchy, with each endpoint representing a resource. The client can then request information about these resources by sending a request to the server. - -
-
-
-### Access the REST API
-
-You can access the REST API and the GraphQL API with the Weaviate client libraries, or with other tools as long as the tool formats the request properly.
-
-In these examples, the code shown in each set of tabs is functionally the same. Both REST calls request meta information. Both GraphQL calls ask for the same data.
-
-#### Example 1: REST vs client requests
-
-import ClientRESTObjectGet from './_snippets/academy.clients.rest.object.get.mdx';
-
-What is GraphQL?
- -GraphQL is a query language for APIs. - -First released by Facebook in 2015, it is now maintained by the GraphQL Foundation. - -GraphQL is a specification for a query language that can be used to request information from a server. GraphQL is a **strongly typed** language, which means that the client must specify the type of data that it wants to receive. - -
-
-
-### Vector searches
-
-The next query searches the `WikiCity` objects for the cities that are closest to the specified text, which in this case is simply, "Major European city".
-
-To run this query, update the connection file you just created.
-
-- Comment out any lines that request meta information
-- Add the OpenAI authorization key you created earlier
-- Add the following code snippet
-
-import QueryVectorSearch from './_snippets/academy.hello.query.vector.search.mdx';
-
-Connect to the demo instance
- -
-
\ No newline at end of file
diff --git a/docs/academy/py/zero_to_mvp/101_hello_weaviate/_snippets/academy.hello.check.connection.mdx b/docs/academy/py/zero_to_mvp/101_hello_weaviate/_snippets/academy.hello.check.connection.mdx
deleted file mode 100644
index 52fa4f803..000000000
--- a/docs/academy/py/zero_to_mvp/101_hello_weaviate/_snippets/academy.hello.check.connection.mdx
+++ /dev/null
@@ -1,59 +0,0 @@
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-Expected output
- -```json -{ - "hostname": "http://[::]:8080", - "modules": { - "generative-openai": { - "documentationHref": "https://beta.openai.com/docs/api-reference/completions", - "name": "Generative Search - OpenAI" - }, - "qna-openai": { - "documentationHref": "https://beta.openai.com/docs/api-reference/completions", - "name": "OpenAI Question & Answering Module" - }, - "ref2vec-centroid": {}, - "text2vec-cohere": { - "documentationHref": "https://docs.cohere.com/docs/embeddings", - "name": "Cohere Module" - }, - "text2vec-huggingface": { - "documentationHref": "https://huggingface.co/docs/api-inference/detailed_parameters#feature-extraction-task", - "name": "Hugging Face Module" - }, - "text2vec-openai": { - "documentationHref": "https://beta.openai.com/docs/guides/embeddings/what-are-embeddings", - "name": "OpenAI Module" - } - }, - "version": "1.18.2" -} -``` - -
-
\ No newline at end of file
diff --git a/docs/academy/py/zero_to_mvp/101_hello_weaviate/_snippets/academy.hello.client.instantiation.mdx b/docs/academy/py/zero_to_mvp/101_hello_weaviate/_snippets/academy.hello.client.instantiation.mdx
deleted file mode 100644
index f06a773d9..000000000
--- a/docs/academy/py/zero_to_mvp/101_hello_weaviate/_snippets/academy.hello.client.instantiation.mdx
+++ /dev/null
@@ -1,26 +0,0 @@
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-Expected output
- -```json -{ - "hostname": "http://[::]:8080", - "modules": { - "generative-openai": { - "documentationHref": "https://beta.openai.com/docs/api-reference/completions", - "name": "Generative Search - OpenAI" - }, - "qna-openai": { - "documentationHref": "https://beta.openai.com/docs/api-reference/completions", - "name": "OpenAI Question & Answering Module" - }, - "ref2vec-centroid": {}, - "text2vec-cohere": { - "documentationHref": "https://docs.cohere.com/docs/embeddings", - "name": "Cohere Module" - }, - "text2vec-huggingface": { - "documentationHref": "https://huggingface.co/docs/api-inference/detailed_parameters#feature-extraction-task", - "name": "Hugging Face Module" - }, - "text2vec-openai": { - "documentationHref": "https://beta.openai.com/docs/guides/embeddings/what-are-embeddings", - "name": "OpenAI Module" - } - }, - "version": "1.18.2" -} -``` - -
-
\ No newline at end of file
diff --git a/docs/academy/py/zero_to_mvp/101_hello_weaviate/_snippets/academy.hello.query.question.answering.mdx b/docs/academy/py/zero_to_mvp/101_hello_weaviate/_snippets/academy.hello.query.question.answering.mdx
deleted file mode 100644
index f5d3bc889..000000000
--- a/docs/academy/py/zero_to_mvp/101_hello_weaviate/_snippets/academy.hello.query.question.answering.mdx
+++ /dev/null
@@ -1,54 +0,0 @@
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-Expected output
- -```json -{ - "generate": { - "error": null, - "singleResult": " #FunFact: Bangkok is the world's most visited city, with over 22 million visitors in 2019! #Bangkok #Thailand #Travel" - } -} -{ - "generate": { - "error": null, - "singleResult": "Did you know that Ho Chi Minh City is home to many multinational companies and generates nearly a quarter of Vietnam's total GDP? #HCMC #Vietnam #Economy" - } -} -{ - "generate": { - "error": null, - "singleResult": "Surprising fact: Singapore is the only country in Asia with a AAA sovereign credit rating from all major rating agencies. #Singapore #AAA #CreditRating" - } -} -``` - -
-
\ No newline at end of file
diff --git a/docs/academy/py/zero_to_mvp/101_hello_weaviate/_snippets/academy.hello.query.vector.search.mdx b/docs/academy/py/zero_to_mvp/101_hello_weaviate/_snippets/academy.hello.query.vector.search.mdx
deleted file mode 100644
index 312ec9d69..000000000
--- a/docs/academy/py/zero_to_mvp/101_hello_weaviate/_snippets/academy.hello.query.vector.search.mdx
+++ /dev/null
@@ -1,64 +0,0 @@
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
-Expected output
- -```json -{ - "data": { - "Get": { - "WikiCity": [ - { - "_additional": { - "answer": { - "hasAnswer": true, - "property": "wiki_summary", - "result": " 2012" - } - }, - "city_name": "London" - } - ] - } - } -} -``` - -
-
\ No newline at end of file
diff --git a/docs/academy/py/zero_to_mvp/101_hello_weaviate/index.mdx b/docs/academy/py/zero_to_mvp/101_hello_weaviate/index.mdx
deleted file mode 100644
index 4c3fce6ca..000000000
--- a/docs/academy/py/zero_to_mvp/101_hello_weaviate/index.mdx
+++ /dev/null
@@ -1,46 +0,0 @@
----
-title: P3_101 Hello, Weaviate
-description: "Get started with Weaviate: an introductory guide to set up and query data."
-sidebar_position: 101 # Like a subject number (e.g. CS101)
----
-
-import ZeroToMvpDeprecationNote from '/docs/academy/py/zero_to_mvp/_snippets/deprecation.md';
-
-Expected output
- -```json -{ - "data": { - "Get": { - "WikiCity": [ - { - "city_name": "Paris", - "country": "France", - "lat": 48.8566, - "lng": 2.3522 - }, - { - "city_name": "London", - "country": "United Kingdom", - "lat": 51.5072, - "lng": -0.1275 - }, - { - "city_name": "Madrid", - "country": "Spain", - "lat": 40.4167, - "lng": -3.7167 - }, - { - "city_name": "Berlin", - "country": "Germany", - "lat": 52.5167, - "lng": 13.3833 - }, - { - "city_name": "Budapest", - "country": "Hungary", - "lat": 47.4983, - "lng": 19.0408 - } - ] - } - } -} -``` - -- - - -Welcome! - -This is the beginning of your journey through the world of vectors with Weaviate. This unit will provide you with an overview of the fundamentals of Weaviate. - -You'll first gain an understanding of what Weaviate is, and what it can do. You will then learn about what vector database and vector search are before going on to run Weaviate and perform vector searches yourself. - -By the end of this unit, you will have a strong foundation of knowledge that will help you to effectively navigate the rest of the course, and for using Weaviate in your own projects. - -### Prerequisites - -- None - -## Learning objectives - -import LearningGoalsExp from '/src/components/Academy/learningGoalsExp.mdx'; - -
- -## About `Get` queries - -In many use cases, retrieving objects from a Weaviate instance may be the most common operation. - -For example, a user may want to retrieve a list of passages most closely related to the input query, or they may wish to retrieve a list of images which are most similar to another image. It is even possible to retrieve a set of images that best match a given passage. - -In Weaviate, such operations to retrieve objects are performed using the `Get` function. - -### `Get` function syntax - -A basic `Get` function looks as follows: - -import GetSyntax from './_snippets/academy.queries.get.syntax.mdx'; - -
-
-
-### Response object from Weaviate
-
-As you see above, the response object from Weaviate is in JSON format, where the results are obtained via the `data` field. It is then arranged in a corresponding manner to reflect the query as well as the queried object.
-
-In the above example, the `Get` field reflects the query function, and the `JeopardyQuestion` field reflects the queried object class, containing returned objects. Each object contains `answer` and `question` fields, reflecting the requested properties.
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "answer": "Amazon.com", - "question": "On July 16, 1995 this company made its first sale, a science textbook" - }, - { - "answer": "Parfait", - "question": "The French name of this layered ice cream dessert means \"perfect\"" - } - ] - } - } -} -``` - -
-
-
-### `Class` and `properties`
-
-In the above example, we specify a `Class` of `JeopardyQuestion` and `properties` of `questions` and `answer`.
-
-That is possible because those follow the structure of our data in Weaviate. To see the available data classes and properties, you can take a look at the Weaviate schema, as shown below:
-
-Explain this query
- -In this case, Weaviate will return two objects due to the `.with_limit(2)` argument. Without this limit, Weaviate would return a maximum number according to its configuration. - -:::tip Is this query useful? -As this does not apply any user-specific criteria to the search, the specific results will likely not be very useful. - -However, this may be a viable method for "sanity" checks, such as checking that you can connect to a Weaviate instance, or that at least some objects have been imported successfully. -::: - -
-
-
-import ShortSchema from './_snippets/academy.queries.schema.short.mdx';
-
-How do I see the schema?
- -You can fetch the schema like this - try it out! - -```python -client.schema.get() -``` - -
-
-
-### Response object from Weaviate
-
-In this response, you see that the `_additional` field has been added to the response object, with `distance` and `id` nested under it.
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "_additional": { - "distance": 0.1791926, - "id": "b1645a32-0c22-5814-8f35-58f142eadf7e" - }, - "answer": "escaping the Earth's gravity (and go off into outer space, on your way to the moon, for instance)", - "question": "It takes approximately 24,840 MPH to achieve this" - }, - { - "_additional": { - "distance": 0.18123823, - "id": "ef263438-b152-5540-97f7-99f4076bd124" - }, - "answer": "the Milky Way", - "question": "This is the name of our own galaxy" - } - ] - } - } -} -``` - -
-
-
-### `_additonal` properties
-
-As the name suggests, `_additional` properties are separate to those explicitly created as class properties.
-
-The above query included the `distance` property in the `_additional` field. This represents the degree of similarity (or, in this case, the *dissimilarity*) between the input vector and the vector of the object.
-
-:::info Vectors available through `_additional`
-
-The object vector can also be retrieved through an `_additional` field, by specifying `vector` in its sub-field.
-
-Note that the returned vector in many cases will be a very long list of numbers.
-
-:::
-
-:::note Exercise
-Try out the above query again, with these changes.
-- Can you get the `vector` property as well?
-- Earlier, providing an empty list as the main properties under the `.get()` method will have caused an error. What happens if you try it again, now that you are requesting additional properties?
-:::
-
-## Review
-
-### Review exercise
-
-See in-line exercises.
-
-### Key takeaways
-
-- The 'Get' function can be used to retrieve objects in Weaviate.
-- The 'Get' function syntax requires specifying the class, properties, and any additional arguments related to the search criteria.
-- Weaviate responses are in JSON format.
-- Class and properties fields must correspond to the objects and properties defined in the Weaviate schema.
-- 'Get' queries can retrieve additional properties not defined in the schema, which can be inherent to the object or related to the query performed.
-- '_additional' properties can include distance, id, and vector information, providing more context and information about the retrieved objects.
-
-import Quiz from '/src/components/Academy/quiz.js'
-export const nearText = [{
- questionText: 'questionText',
- answerOptions: [
- {
- answerText: 'answerOne',
- isCorrect: false,
- feedback: 'feedbackOne',
- },
- {
- answerText: 'answerTwo',
- isCorrect: false,
- feedback: 'feedbackTwo',
- },
- {
- answerText: 'answerThree',
- isCorrect: false,
- feedback: 'feedbackThree',
- },
- ]
-}];
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Explain this query
- -Here, the `distance` and `id` properties contain the object's distance to the query and its unique ID respectively. - -- -Weaviate offers multiple vector search "operators", through which you can carry out vector searches. Typically, our users use one of `nearVector`, `nearObject` or `near
-
-
-import NearVectorSimple from './_snippets/academy.queries.nearVector.simple.mdx';
-
-See the whole vector
- -[0.023932384327054024, -0.014095712453126907, 0.013304559513926506, -0.01155742909759283, -0.01147831417620182, 0.015321999788284302, -0.025013625621795654, -0.04198386147618294, 0.0006061387248337269, -0.008940030820667744, 0.013475975953042507, 0.0021558923181146383, 0.008148877881467342, -0.0022696207743138075, 0.014623147435486317, 0.0010969009017571807, 0.032199934124946594, -0.016746075823903084, 0.007700557820498943, 0.014293501153588295, -0.004793069791048765, 0.009223527275025845, 0.01840749755501747, -0.007628035265952349, -0.0017108687898144126, -0.001233704504556954, 0.01867121458053589, -0.030696744099259377, 0.012150794267654419, -0.003728309413418174, 0.030512141063809395, -0.004667803645133972, -0.005402917042374611, -0.013779250904917717, -0.013344116508960724, -0.026727791875600815, 0.020596355199813843, -0.008043390698730946, -0.0035865609534084797, -0.007120378781110048, 0.0073379455134272575, 0.031145064160227776, -0.009137819521129131, -0.00028988346457481384, -0.009823485277593136, 0.007482990622520447, 0.0011644785990938544, -0.012856239452958107, 0.001305402722209692, 0.007911532185971737, 0.011432163417339325, 0.033914100378751755, -0.0353645458817482, -0.013845180161297321, -0.009896007366478443, -0.009717998094856739, -0.022824769839644432, 0.008280736394226551, 0.029299039393663406, -0.02130839228630066, 0.008485117927193642, 0.004756808280944824, -0.021071046590805054, 0.03462613746523857, -0.021756712347269058, 0.0076082563027739525, 0.0020471089519560337, -0.013205665163695812, 0.00041803380008786917, 0.0013144679833203554, 0.019475553184747696, 0.0022762136068195105, -0.001709220465272665, -0.0005315560265444219, 0.0039063189178705215, -0.027110181748867035, 0.008676312863826752, -0.009843263775110245, -0.02209954522550106, 0.012394732795655727, 0.016772447153925896, -0.016179082915186882, -0.012078272178769112, 0.018684400245547295, 0.02439389005303383, 0.021321577951312065, -0.012282652780413628, 0.013726507313549519, -0.030169308185577393, -0.007898345589637756, 0.0166669599711895, 0.0005167219205759466, -0.0011076144874095917, 0.028085937723517418, -0.004760105162858963, 0.0016292810905724764, -0.0074698044918477535, 0.02565973438322544, 0.0024525749031454325, -0.036393046379089355, -0.01155742909759283, 0.013937481679022312, -0.025527875870466232, -0.023549992591142654, -0.014201199635863304, -0.005650152452290058, 0.011267339810729027, 0.022152289748191833, 0.025686105713248253, -0.01213101577013731, 0.011418977752327919, 0.021743526682257652, -6.031512748450041e-05, -0.010172910988330841, -0.010120167396962643, -0.026450887322425842, 0.020042547956109047, 0.006981926970183849, -0.012295839376747608, -0.04338156431913376, 0.023958755657076836, 0.014675891026854515, 0.022442378103733063, -0.00022127565171103925, 0.012427697889506817, 0.03164612874388695, -0.020464494824409485, -0.01867121458053589, 0.0167856328189373, -0.02187538519501686, 0.009632290340960026, 0.022297333925962448, 0.007318167015910149, -0.0076148491352796555, -0.01660103164613247, 0.018499799072742462, -0.004489794373512268, 0.007779672741889954, -0.0037349022459238768, -0.022112730890512466, 0.0036524904426187277, 0.012691415846347809, -0.015638461336493492, 0.0141352703794837, -0.017220767214894295, 0.035944726318120956, 0.01574394851922989, 0.005607298109680414, 0.009625696577131748, -0.021611668169498444, 0.013581463135778904, -0.004196408204734325, 0.013541905209422112, -0.007113785482943058, 0.0005550433997996151, 0.012889203615486622, -0.010060830973088741, 0.02295662835240364, -0.0015551104443147779, -0.007786266040056944, -0.003810721216723323, 0.003098683198913932, -0.0011685991194099188, -0.01275075227022171, 0.02200724370777607, 0.03061762824654579, 0.0014512715861201286, -0.004245855379849672, -0.01964697055518627, -0.01104977261275053, -0.018104221671819687, 0.006741284392774105, -0.038871992379426956, 0.011122294701635838, -0.017181210219860077, 0.0328855998814106, 7.489171548513696e-05, 0.004971079062670469, -0.031303294003009796, -0.023906011134386063, -0.005834754556417465, 0.003945876378566027, 0.0012196945026516914, 0.009137819521129131, -0.01106955111026764, -0.028666116297245026, 0.008768614381551743, -0.0029849549755454063, 0.010153132490813732, -0.015308814123272896, -0.0037085304502397776, -0.002592674922198057, -0.010252026841044426, 0.0013853422133252025, -0.6903074979782104, -0.03264825418591499, 0.00757529167458415, -0.021769898012280464, 0.005010636989027262, 0.013528719544410706, 0.006810510065406561, 0.008940030820667744, -0.009856450371444225, -0.00023652183881495148, -0.02009529061615467, 0.023906011134386063, -0.023958755657076836, -0.005762232467532158, -0.011689288541674614, -0.014504474587738514, 0.007080820854753256, -0.018051479011774063, 0.014385801739990711, 0.012856239452958107, -0.02067546918988228, 0.02763761766254902, 0.005946834571659565, 0.020345821976661682, 0.015335185453295708, 0.00432497076690197, 0.03797535225749016, -0.013284780085086823, -0.012803495861589909, -0.00042730511631816626, -0.007094006985425949, 0.015651647001504898, -0.012645265087485313, -0.001946566510014236, 0.05511700361967087, -0.005521589890122414, -0.003125054994598031, 0.019225021824240685, 0.009071889333426952, 0.023009371012449265, -0.008300515823066235, -0.0010746497428044677, 0.021954501047730446, 0.010100388899445534, -0.016970235854387283, 0.008485117927193642, 0.04230032488703728, -0.019541483372449875, -0.009038925170898438, -0.030643999576568604, 0.0005055963410995901, 0.03618207201361656, -0.004559020046144724, 0.014807750470936298, 0.009329014457762241, 0.019000861793756485, 0.020279893651604652, 0.019501926377415657, 0.0006320984102785587, 0.0028465031646192074, -0.000699264055583626, 0.028797974810004234, -0.02056998200714588, 0.001127393334172666, -0.014794564805924892, 0.020464494824409485, -0.0181305930018425, 0.020016174763441086, -0.016653774306178093, -0.01939643919467926, 0.029773730784654617, 0.021071046590805054, -0.013845180161297321, 0.030643999576568604, 0.014952794648706913, 0.026543188840150833, 0.025198228657245636, -0.007535734213888645, -0.017827318981289864, 0.022086359560489655, 0.0005212545511312783, -0.0073313526809215546, -0.04918335750699043, -0.024090614169836044, 0.03494259715080261, -0.004084328189492226, -0.028507886454463005, 0.009137819521129131, 0.022152289748191833, 0.009203748777508736, 0.0019432699773460627, 0.027558501809835434, -0.023760966956615448, -0.02416973002254963, 0.00514249550178647, -0.03014293685555458, -0.018486613407731056, -0.007304980885237455, 0.005656745284795761, -0.008827950805425644, 0.008623569272458553, -0.016587844118475914, 0.028929835185408592, 0.01545385830104351, -0.004028288181871176, -0.012849646620452404, 0.004885370843112469, 0.017576785758137703, 0.012988097965717316, -0.021638039499521255, -0.009553174488246441, -0.008227992802858353, -0.00912463292479515, 0.020583167672157288, -0.0167856328189373, -0.02674097754061222, 0.007858788594603539, -0.014174827374517918, 0.017352625727653503, -0.017154838889837265, 0.04003234952688217, -0.007852194830775261, 0.01170247420668602, 0.004951300099492073, -0.005396323744207621, -0.005828161723911762, 0.007970868609845638, -0.03336029127240181, -0.016930678859353065, -0.0029025431722402573, -0.0031563714146614075, 0.019739272072911263, 0.009797113947570324, -0.0014908292796462774, 0.004908446222543716, 0.008274143561720848, 0.014504474587738514, -0.019409624859690666, -0.022982999682426453, -0.021980872377753258, -0.03172524273395538, -0.0033986622001975775, -0.0009329014574177563, -0.0034052550327032804, -0.008544454351067543, -0.006929183378815651, 0.0042656343430280685, 0.0009007608750835061, -0.006569867953658104, -0.029562756419181824, -0.0016894417349249125, 0.006151216104626656, -0.0074698044918477535, 0.013528719544410706, 0.011966192163527012, -0.018776701763272285, -0.003073959844186902, -0.027743104845285416, -0.02500043995678425, -0.017339440062642097, 0.023892825469374657, 0.0060753971338272095, -0.008775207214057446, -0.0004223604337312281, -0.007219272665679455, -0.010252026841044426, -0.016640588641166687, 0.015124212019145489, -0.007713743485510349, -0.010014680214226246, 0.0026569559704512358, -0.007041263394057751, -0.014913237653672695, 0.018565727397799492, -0.017405370250344276, -0.0010021273046731949, 0.0036030435003340244, 0.00573915708810091, -0.0007532437448389828, -3.229512003599666e-05, -0.010898134671151638, -0.007839009165763855, 0.001443030429072678, 0.004476608242839575, 0.015137397684156895, 0.01714165136218071, 0.03380861133337021, 0.016113152727484703, -0.009447687305510044, 0.022020429372787476, -0.005152385216206312, 0.03586561232805252, 0.006688540801405907, -0.002876171376556158, -0.012816681526601315, -0.0023124748840928078, -0.001210629241541028, -0.020978745073080063, 0.0031217585783451796, 0.018750330433249474, 0.009487245231866837, -0.022152289748191833, 0.007806044537574053, -0.02025352045893669, 0.004793069791048765, -0.016139524057507515, -0.019950246438384056, -0.029773730784654617, 0.01933050900697708, -0.012955132871866226, 0.008115912787616253, 0.0029651762451976538, 0.002904191380366683, 0.012928761541843414, 0.034837111830711365, 0.019198650494217873, -0.017919618636369705, 0.010977250523865223, -0.02038538083434105, -0.005564444232732058, -0.023892825469374657, 0.01181455422192812, 0.01020587608218193, -0.007812637835741043, -0.008181842043995857, 0.02079414203763008, -0.01039707101881504, -0.011564021930098534, 0.002449278486892581, -0.038212697952985764, 0.0007363493205048144, 0.003395365783944726, -0.00129963387735188, 0.027083810418844223, 0.029615500941872597, 0.024631235748529434, -0.008794985711574554, -0.006915997248142958, 0.029167180880904198, 0.007529140915721655, -0.0042854128405451775, 0.005640262737870216, 0.029905589297413826, -0.006101768929511309, -0.004753511864691973, 0.011267339810729027, 0.01545385830104351, -0.004868888296186924, -0.012388139963150024, 0.0011809609131887555, -0.04034881293773651, -0.010627823881804943, -0.026503631845116615, 0.00977733451873064, 0.012394732795655727, -0.0054820324294269085, -0.01822289451956749, 0.006319336127489805, 0.02391919679939747, 0.024723537266254425, 0.005158978048712015, -0.0021575407590717077, 0.007456618826836348, 0.0139111103489995, 0.007502769120037556, -0.008834543637931347, -0.0024707054253667593, 0.01596810854971409, 0.002129520522430539, 0.015823064371943474, 0.00874224305152893, -0.032991088926792145, 0.011155259795486927, -0.012183759361505508, 0.020279893651604652, 0.004845813382416964, -0.0030442913994193077, -0.0077335224486887455, 0.01327818725258112, 0.006016060709953308, 0.0021047971677035093, -0.03892473503947258, 0.007212679833173752, 0.014517661184072495, -0.014082526788115501, -0.027690360322594643, 0.0006259175133891404, 0.02604212611913681, -0.0019498629262670875, 0.0015748892910778522, -0.02111060358583927, 0.001626808661967516, 0.0029388044495135546, -0.020055733621120453, -0.008794985711574554, -0.017102094367146492, 0.008485117927193642, -0.012295839376747608, 0.017985548824071884, 0.011735438369214535, -0.0017636122647672892, 0.020055733621120453, -0.0020108476746827364, -0.0040579563938081264, 0.03449427708983421, 0.020055733621120453, -0.013416639529168606, -0.016930678859353065, 0.006180884316563606, -0.009427908807992935, 0.0046381354331970215, 0.0022531382273882627, -0.013977039605379105, -0.013113363645970821, 0.0014776433818042278, -0.004338156431913376, -0.02455211989581585, -0.02500043995678425, 0.021928129717707634, -0.005890794564038515, -0.00726542342454195, -0.014174827374517918, -0.02203361690044403, -0.018842631950974464, 0.06730076670646667, -0.010416850447654724, -0.008636755868792534, 0.021044675260782242, 0.008320294320583344, -0.004657914396375418, -0.012869425117969513, -0.013871552422642708, -0.008590605109930038, -0.004275523591786623, 0.006902811583131552, 0.003566782223060727, -0.0010655844816938043, 0.005353469867259264, 0.0153747433796525, 0.019185464829206467, 0.010172910988330841, -0.011155259795486927, 0.018869003280997276, -0.014069341123104095, 0.005894090980291367, -0.027584875002503395, 0.01314632873982191, 0.018644843250513077, 0.016047224402427673, -0.029272668063640594, 0.01675926148891449, -0.000627565779723227, -0.005386434495449066, -0.03278011456131935, -0.0004099986399523914, 0.008715870790183544, -0.008214807137846947, -0.007759894244372845, 0.002576192608103156, -0.007581884507089853, 0.001857561757788062, 0.008300515823066235, 0.005468846298754215, -0.027004694566130638, 0.014412174001336098, 0.014623147435486317, -0.0011933227069675922, -0.02038538083434105, 0.0038634645752608776, -0.006042432505637407, -0.019792014732956886, 0.021980872377753258, -0.010489372536540031, -0.0018839335534721613, 0.026991508901119232, 0.009783927351236343, 0.001620215829461813, -0.03183072805404663, 0.006981926970183849, 0.0184734258800745, 0.016099967062473297, -0.0034085516817867756, -0.01962059922516346, -0.03262188285589218, -0.01043003611266613, -0.024130171164870262, 0.02410379983484745, 0.016179082915186882, -0.02067546918988228, -0.03557552024722099, -0.0009081779280677438, -0.008373037911951542, -0.0137396939098835, 0.008986181579530239, -0.008669720031321049, -0.030327538028359413, -0.004924928303807974, -0.019449181854724884, 0.007792858872562647, 0.005449067335575819, 0.02101830206811428, -0.02079414203763008, 0.019805200397968292, 0.0044172718189656734, -0.0031992257572710514, -0.0012485386105254292, 0.004555723629891872, -0.009164190851151943, -0.0008335952297784388, 0.021638039499521255, 0.0077401152811944485, 0.0009032331872731447, -0.0077401152811944485, 0.0019877722952514887, 0.014280314557254314, -0.005017229821532965, 0.007515955250710249, -0.0070148915983736515, 0.015361557714641094, 0.02041175216436386, 0.01213101577013731, -0.0013721563154831529, 0.0015946681378409266, 0.02387963980436325, 0.0044172718189656734, -0.015664832666516304, -0.002815186744555831, -0.0028184831608086824, 0.003744791727513075, 0.01179477572441101, -0.01685156300663948, 0.008946623653173447, -0.010799241252243519, -0.017497671768069267, 0.009216934442520142, -0.02919355221092701, 0.003922800999134779, -0.01774820312857628, 0.0009922379394993186, 0.008234585635364056, 0.0011100867995992303, 0.01084539107978344, -0.010528930462896824, -0.03185710310935974, -0.007311573717743158, -0.02969461679458618, 0.0046315426006913185, 0.03750066086649895, 0.006006170995533466, -0.008267550729215145, -0.009058703668415546, -0.0047699944116175175, 0.0002550644858274609, 0.0018460240680724382, -0.002813538536429405, 0.012335396371781826, -0.009520210325717926, 0.0054523637518286705, -0.03122417815029621, -0.007107192650437355, -0.007080820854753256, -0.0042788200080394745, -0.0025069667026400566, -0.007186308037489653, -0.02092600241303444, 0.0075950706377625465, 0.01189366914331913, -0.015585717745125294, -0.006797324400395155, -0.019515112042427063, -0.0021839123219251633, 0.0021377617958933115, -0.025580618530511856, 0.03014293685555458, 0.00684347515925765, 0.0008265902288258076, -0.019515112042427063, -0.028481515124440193, -0.03541729226708412, -0.0385291613638401, -0.027400271967053413, -0.026622304692864418, 0.033281177282333374, 0.02610805444419384, 0.022152289748191833, -0.017959177494049072, 0.03349215164780617, -0.0019729381892830133, 0.005725970957428217, 0.003622822230681777, 0.022666538134217262, -0.008794985711574554, -0.024288402870297432, 0.010819019749760628, 0.005656745284795761, 0.001946566510014236, 0.010759683325886726, 0.003340973984450102, 0.0007503593224100769, 0.029905589297413826, -0.016482356935739517, 0.002592674922198057, 0.0033063609153032303, -0.027769476175308228, -0.011517872102558613, 0.0021328171715140343, -0.007542327046394348, -0.0033162503968924284, -0.012249688617885113, -0.01758997142314911, 0.03354489430785179, 0.025738850235939026, -0.005623780656605959, -0.0001497834309702739, 0.0182492658495903, -0.011735438369214535, 0.041772887110710144, -0.012506812810897827, 0.02153255231678486, -0.008834543637931347, -0.0038239071145653725, -0.002765739569440484, 0.011616765521466732, -0.0022383041214197874, 0.015823064371943474, 0.002080073580145836, 0.0016836728900671005, 0.02219184674322605, 0.0013350709341466427, 0.0074698044918477535, 0.0013152922037988901, 0.011854112148284912, 0.004334860015660524, -0.022125916555523872, -0.006457787938416004, -0.021888570860028267, -0.00907848309725523, -0.03889836370944977, -0.01592855155467987, -0.011715659871697426, 0.009731183759868145, 0.038397300988435745, -0.03934668377041817, -0.011339861899614334, 0.011821147054433823, -0.011102516204118729, 0.021123789250850677, 0.0019317322876304388, 0.02385326847434044, 0.032068073749542236, -0.011801368556916714, -0.014267128892242908, -0.02254786528646946, -0.017022978514432907, -0.005297429859638214, 0.014346243813633919, 0.013581463135778904, -0.0009889415232464671, -0.024209287017583847, -0.0007462387438863516, 0.0018921747105196118, -0.02705743908882141, -0.022244589403271675, 0.0153747433796525, 0.004483201541006565, 0.025910265743732452, -0.0155989034101367, -0.008373037911951542, -0.008168656378984451, 0.021123789250850677, 0.016231825575232506, -0.00213446537964046, -0.01660103164613247, -0.017128465697169304, -0.01694386452436447, 0.014952794648706913, 0.0010252026841044426, 0.010997029021382332, 0.008623569272458553, -0.006438008975237608, 0.0176822729408741, 0.004687582608312368, 0.009487245231866837, 0.0012749104062095284, -0.012572742998600006, 0.03132966533303261, 0.006352300755679607, 0.03148789703845978, 0.007957682013511658, 0.0002802000963129103, 0.01022565457969904, -0.00923671294003725, 0.0007136861095204949, 0.02903532236814499, -0.015387929044663906, -0.010680567473173141, 0.010515743866562843, 0.0028943021316081285, 0.0184734258800745, -0.005538072437047958, -0.0017174617387354374, 0.015229698270559311, -0.00958613958209753, 0.0035634858068078756, 0.01669333130121231, 0.0015963163459673524, -0.0008438967051915824, -0.019225021824240685, -0.004601874388754368, 0.009467466734349728, -0.025105927139520645, -0.010278398171067238, 0.01583625003695488, -0.026701420545578003, -0.020490868017077446, -0.010074017569422722, 0.0018114111153408885, 0.008584012277424335, -0.02572566457092762, 0.00958613958209753, -0.0032272457610815763, 0.025606991723179817, -0.03539091721177101, 0.016271384432911873, 0.018077850341796875, 0.022297333925962448, -0.020016174763441086, 0.004430457949638367, 0.0136869503185153, -0.011768403463065624, 0.015783505514264107, -0.011867297813296318, -0.0135155338793993, -0.0057028960436582565, -0.012566149234771729, 0.010884949006140232, -0.008748835884034634, -0.005126013420522213, -0.02086007222533226, 0.029404526576399803, -0.0021641335915774107, -0.028217796236276627, -0.012170572765171528, 0.0013952315784990788, 0.007667592726647854, 0.014860494062304497, 0.02108423225581646, -0.039610400795936584, 0.016337312757968903, -0.022442378103733063, 0.011959598399698734, 0.003134944476187229, -0.022112730890512466, 0.01984475925564766, -0.014623147435486317, 0.0172339528799057, 0.0005542192957364023, -0.02216547541320324, -0.0029503421392291784, -0.003777756355702877, -0.014161641709506512, 0.006381968967616558, -0.016904305666685104, 0.006606128998100758, 0.0135155338793993, 0.02887709066271782, 0.025606991723179817, 0.009401536546647549, 0.005946834571659565, 0.0014356133760884404, -9.935771231539547e-05, 0.0010936044855043292, -0.012684823013842106, -0.0027574985288083553, 0.020332636311650276, 0.021967686712741852, 0.014398987405002117, -0.020965559408068657, -0.010469594039022923, 0.001205684500746429, -0.018658028915524483, -0.02998470515012741, 0.0014298445312306285, -0.00483262725174427, 0.010245434008538723, 0.009197155945003033, -0.005307319108396769, 0.02536964602768421, -0.0022926959209144115, 0.01035092119127512, 0.0026800313498824835, -0.009012552909553051, 0.04053341597318649, -0.0031547232065349817, -0.006902811583131552, -0.003113517304882407, -0.031145064160227776, -0.012473848648369312, 0.010522337630391121, 0.002930563176050782, 0.011623358353972435, -0.0023306054063141346, 0.013126550242304802, 0.02718929760158062, -6.428119377233088e-05, -0.015190141275525093, -0.006711616180837154, 0.01685156300663948, 0.008438967168331146, -0.030670370906591415, -0.0014825881225988269, 0.002033923054113984, -0.007278609089553356, 0.0038964294362813234, -0.005923759192228317, 0.019805200397968292, 0.0051227170042693615, 0.004282116424292326, 0.015427486971020699, -0.034230560064315796, 0.017471298575401306, -0.040849875658750534, 0.0021542441099882126, 0.002892653690651059, 0.015321999788284302, -0.0010029515251517296, 0.010080610401928425, -0.00695555517449975, 0.01894811913371086, 0.0034645916894078255, 0.011412384919822216, -0.005719378124922514, -0.004397492855787277, 0.002235007705166936, -0.0046249497681856155, -0.011412384919822216, -0.008656534366309643, -0.00885432306677103, 0.008715870790183544, -0.014029783196747303, 0.012632079422473907, -0.007707150653004646, 0.01650873012840748, 0.023365391418337822, -0.006589646451175213, -0.022982999682426453, -0.0005290837143547833, -0.008709277957677841, -0.015704389661550522, -0.0062962607480585575, 0.0015468692872673273, -0.005379841662943363, 0.00585123710334301, -0.001642466988414526, -0.0037744599394500256, -0.010548708960413933, -0.00768737168982625, 0.008808172307908535, 0.016970235854387283, 0.023826897144317627, 0.22004607319831848, -0.025303715839982033, -0.015018724836409092, 0.03336029127240181, -0.014517661184072495, 0.018895374611020088, 0.02387963980436325, -0.004390900023281574, 0.006002874579280615, 0.014794564805924892, -0.017695458605885506, 0.027110181748867035, 0.004262337926775217, -0.00915759801864624, -0.005735860671848059, 0.0004409030661918223, -0.01917227916419506, -0.013950667344033718, 0.0026338808238506317, -0.01136623416095972, 0.007792858872562647, -0.0023685148917138577, -0.011972784996032715, -0.013192479498684406, 0.009711405262351036, -0.012730972841382027, -0.0030822008848190308, 0.0018278934294357896, 0.019027233123779297, 0.01020587608218193, 0.011155259795486927, -0.014201199635863304, -0.009902600198984146, -0.000843484653159976, 0.017563600093126297, 0.011623358353972435, 0.021387508139014244, -0.012440883554518223, 0.0011603579623624682, 0.009197155945003033, -0.022745653986930847, 0.006935776211321354, -0.009137819521129131, -0.0063094464130699635, -0.004460126161575317, -0.0011661268072202802, -0.01901404745876789, 0.010667381808161736, 0.011939819902181625, 0.009737776592373848, -0.034520652145147324, -0.014504474587738514, 0.010515743866562843, 0.04034881293773651, -0.029483642429113388, -0.0035371140111237764, 0.018750330433249474, 0.008234585635364056, -0.013752879574894905, 0.007370910607278347, 0.0017322958447039127, 0.016640588641166687, -0.016205454245209694, -0.0019531594589352608, 0.002096555894240737, 0.020266707986593246, -0.018842631950974464, -0.017669087275862694, 0.028613373637199402, 0.01806466467678547, 0.01203871425241232, 0.0010144890984520316, -0.03011656366288662, -0.0017850393196567893, -0.024538934230804443, -0.0027921113651245832, 0.011603579856455326, 0.0057984935119748116, 0.008096134290099144, 0.018262453377246857, -0.005514997057616711, -0.004799662623554468, -0.002014144090935588, -0.023760966956615448, -0.026886021718382835, -0.031303294003009796, 0.011577208526432514, -0.00917737651616335, -0.015664832666516304, 0.007819230668246746, 0.0028349654749035835, 0.006714912597090006, 0.015440672636032104, -0.004921631887555122, 0.008412595838308334, -0.0027228854596614838, 0.00585123710334301, 0.01127393264323473, -0.006378672551363707, -0.015440672636032104, -0.027611246332526207, 0.0023882936220616102, 0.0029948444571346045, -0.0015147286467254162, -0.014240757562220097, 0.0010804185876622796, -0.006546792574226856, -0.01507146842777729, 0.0015773616032674909, 0.00542269553989172, -0.005983096081763506, -0.007806044537574053, 0.008972995914518833, 0.00010152102186111733, -0.01977882906794548, 0.02140069380402565, 0.018341567367315292, -0.0009535044082440436, 0.025290530174970627, -0.0007726105395704508, -0.00203557126224041, -0.006622611545026302, 0.006935776211321354, 0.006556681822985411, -0.006658872589468956, -0.026859650388360023, -0.008096134290099144, 0.017273511737585068, -0.004041474312543869, -0.029325410723686218, -0.0063918582163751125, -0.0029025431722402573, 0.019159093499183655, -0.02378733828663826, 0.006316039711236954, 0.01930413767695427, 0.004011806100606918, -0.009355386719107628, -0.010166318155825138, -0.013416639529168606, -0.007753300946205854, 0.018170151859521866, 0.002277861814945936, 0.004948003683239222, 0.0019218429224565625, -0.01971290074288845, 0.017893247306346893, -0.002117983065545559, 0.0033508634660393, -0.02022714912891388, 0.0006386913591995835, -0.022692909464240074, -0.012302432209253311, 0.0026816795580089092, 0.03362400829792023, -0.00992237962782383, -0.01948874071240425, -0.028507886454463005, -0.0013861663173884153, 0.00684347515925765, -0.01653510145843029, 0.007542327046394348, 0.016455985605716705, -0.014372616074979305, -0.010370699688792229, 0.0004413151182234287, -0.17067810893058777, 0.013779250904917717, -0.0009724590927362442, -0.011168445460498333, 0.015308814123272896, -0.012632079422473907, -0.009408130310475826, -0.0020273299887776375, 0.006988519802689552, 0.002660252619534731, 0.006431416142731905, 0.0009312531910836697, -0.05643559247255325, -0.008748835884034634, 0.013581463135778904, 0.013344116508960724, -0.011517872102558613, 0.011768403463065624, 0.01758997142314911, -0.009282863698899746, 0.03307020291686058, -0.029615500941872597, -0.006721505429595709, -0.02079414203763008, 0.007700557820498943, -0.009526803158223629, -0.0151505833491683, 0.0028679303359240294, -0.013297966681420803, 0.0027294785249978304, -0.018552541732788086, -0.0032585621811449528, 0.04222120717167854, 0.004740326199680567, 0.03230542317032814, 0.009025739505887032, -0.008293922059237957, 0.0026849762070924044, -0.017273511737585068, 0.016034036874771118, 0.0011323379585519433, 0.020490868017077446, 0.014583590440452099, -0.0017075722571462393, -0.019963432103395462, -0.003761274041607976, -0.004628246184438467, -0.011682695709168911, 0.004687582608312368, 0.003373938612639904, 0.006368783302605152, -0.03275374323129654, -0.017220767214894295, -0.0034547022078186274, 0.037922609597444534, 0.014227570965886116, 0.024763094261288643, 0.006058914586901665, 0.007766487076878548, -0.02238963544368744, -0.028059566393494606, -0.018011920154094696, 0.001116679748520255, -0.03926756978034973, -0.0018295417539775372, -0.02423565834760666, -0.01583625003695488, 0.005432585254311562, -0.012599114328622818, 0.024116985499858856, -0.0019993099849671125, 0.0008875749772414565, 0.0007441784837283194, -0.0030162713956087828, 0.011867297813296318, 0.025422388687729836, -0.012374954298138618, 0.020504053682088852, -0.00885432306677103, -0.02500043995678425, -0.03937305510044098, 0.038871992379426956, 0.01020587608218193, -0.0008513137581758201, -0.002080073580145836, 0.0025910267140716314, -0.01583625003695488, 0.007654407061636448, -0.02194131538271904, -0.004206297919154167, 0.017181210219860077, -0.026846464723348618, 0.004921631887555122, -0.013713321648538113, 0.025870708748698235, 0.024763094261288643, 0.012236502021551132, 0.010179503820836544, 0.023576363921165466, -0.007674186024814844, -0.004203001037240028, 0.0139111103489995, 0.006550088990479708, 0.0015740651870146394, 0.031171435490250587, 0.016587844118475914, 0.014702263288199902, 3.2604162697680295e-05, 0.017985548824071884, -0.019923873245716095, 0.008293922059237957, -0.02209954522550106, -0.00284155854023993, 0.024182915687561035, -0.011148666962981224, 0.02397194132208824, -0.014095712453126907, -0.01443854533135891, 0.017537228763103485, 0.009678440168499947, 0.04754830524325371, -0.019027233123779297, -0.03043302521109581, -0.0134034538641572, -0.017260326072573662, -0.0012691415613517165, -0.09361979365348816, -0.02658274583518505, -0.006484159734100103, 0.0014405581168830395, 0.0032832857687026262, 0.005874312482774258, -0.005188646260648966, -0.00885432306677103, -0.016495544463396072, 0.014319872483611107, 0.006022653542459011, -0.02358955144882202, -0.012783716432750225, -0.020372195169329643, 0.039926864206790924, -0.0139111103489995, 0.004470015410333872, -0.013884738087654114, -0.007779672741889954, 0.018196523189544678, -0.00850489642471075, 0.002144354861229658, 0.011372826993465424, 0.008003832772374153, -0.019027233123779297, -0.006408340763300657, -0.030960461124777794, 0.028217796236276627, 0.020820515230298042, -0.007845601998269558, -0.027294784784317017, -0.002777277259156108, 0.0045425379648804665, -0.015124212019145489, -0.023985126987099648, -0.004918335471302271, 0.0015443969750776887, -0.021347949281334877, 0.028534257784485817, -0.04522759094834328, -0.012249688617885113, 0.01586262136697769, -0.0170097928494215, -0.02047768048942089, -0.005610594525933266, -0.013001283630728722, -0.0027789254672825336, 0.03014293685555458, 0.005267761647701263, -0.02734752744436264, -0.03275374323129654, -0.020082104951143265, -0.037474289536476135, 0.014148456044495106, 0.008748835884034634, 0.010443221777677536, 0.005818272475153208, 0.00779945170506835, 0.0001228965847985819, 1.5928653738228604e-05, -0.006002874579280615, 0.01020587608218193, -0.029773730784654617, 0.027822220697999, -0.007080820854753256, -0.00799723993986845, -0.03539091721177101, -0.0040579563938081264, 0.015163769014179707, -0.003744791727513075, -0.009678440168499947, 0.02302255667746067, -0.012427697889506817, 0.001398528111167252, -0.05298089236021042, -0.007001705467700958, -0.0019564558751881123, -0.006263296119868755, 0.04557042196393013, -0.008577419444918633, -0.02419610135257244, -0.00706763518974185, 0.030195679515600204, 0.00038033039891161025, 0.024314774200320244, 0.0385291613638401, 0.0186975859105587, -0.008564232848584652, 0.013647392392158508, 0.012684823013842106, -0.0033146021887660027, -0.0013803974725306034, 0.013647392392158508, -0.04330245032906532, -0.009335607290267944, 0.007839009165763855, -0.020662283524870872, -0.0015452210791409016, 0.026253100484609604, 0.020820515230298042, -0.02635858580470085, -0.0058116791769862175, -0.06608766317367554, 0.02813868224620819, 0.006896218750625849, 0.015229698270559311, 0.010258619673550129, -0.010990436188876629, 0.019673341885209084, -0.0059303524903953075, -0.0018839335534721613, 0.007944496348500252, -0.0061709946021437645, 0.01243429072201252, -0.0002400243392912671, -0.004941410850733519, 0.0034975563175976276, -0.0031481303740292788, 0.015796691179275513, 0.008458745665848255, 0.005561147350817919, 0.0002383761020610109, -0.010627823881804943, 0.021374322474002838, 0.01955466903746128, 0.002106445375829935, -0.0025646549183875322, 0.0060819899663329124, -0.01062123104929924, -0.001493301591835916, -0.01608678139746189, 0.002991548040881753, 0.014728634618222713, -0.021796269342303276, -0.024657607078552246, 0.028481515124440193, -0.028534257784485817, -0.018750330433249474, -0.014623147435486317, 0.009322421625256538, 0.005152385216206312, 0.03275374323129654, -0.007509362418204546, 0.0011076144874095917, 0.02980010211467743, 0.0032157080713659525, -0.015335185453295708, 0.01662740297615528, 0.00958613958209753, 0.022363262251019478, 0.002279510023072362, 0.011234374716877937, -0.010383885353803635, 0.026068497449159622, -0.02327308990061283, -0.004743622615933418, 0.003174502169713378, -0.006975333672016859, 0.018038293346762657, -0.0016432910924777389, -0.0019201947143301368, -0.006556681822985411, 0.02840239927172661, 0.0139111103489995, 0.017036164179444313, 0.00675776693969965, -0.0052117216400802135, -0.01860528625547886, -0.023259904235601425, -0.001598788658156991, 0.012091457843780518, -0.04541219398379326, 0.0060325427912175655, -0.014583590440452099, 0.009038925170898438, 0.0018427276518195868, -0.017603158950805664, -0.00652701361104846, 0.006922590080648661, 0.0038964294362813234, -0.029114436358213425, 0.01930413767695427, 0.03523268923163414, -0.01031795609742403, -0.036366675049066544, 0.0017339440528303385, 0.022059988230466843, 0.006006170995533466, 0.006197366397827864, 0.028771603479981422, -0.0073313526809215546, 0.010074017569422722, -0.00547873554751277, 0.003995323553681374, -0.0017388887936249375, -0.00240147951990366, -0.008880694396793842, 0.04406723380088806, -0.012790309265255928, 0.002683327766135335, 0.020833700895309448, 0.03188347443938255, -0.004288709722459316, -0.006672058254480362, 0.013304559513926506, -0.003932690713554621, -0.012315617874264717, 0.03494259715080261, -0.03125055134296417, -0.036234814673662186, -0.007239051628857851, 0.0047205472365021706, -2.2856394934933633e-05, 0.014280314557254314, 0.00726542342454195, 0.006513827946037054, -0.014280314557254314, -0.00041597351082600653, 0.005867719184607267, -0.03974226117134094, -0.02165122516453266, 0.02232370525598526, 0.0074632116593420506, 0.04622971639037132, -0.012348582036793232, -0.026028938591480255, 0.03272736817598343, 0.0184734258800745, 0.04172014445066452, 0.0034052550327032804, 0.00010394187120255083, 0.008972995914518833, 0.0014356133760884404, -0.015229698270559311, 0.002439389005303383, 0.0035041493829339743, 0.002167430007830262, 0.012684823013842106, -0.003751384560018778, 0.011755217798054218, -0.031145064160227776, 0.05263805761933327, 0.007628035265952349, -0.011175038293004036, 0.025765221565961838, -0.016165897250175476, -0.00207842537201941, 0.023233531042933464, 0.01222990918904543, -0.02795407921075821, -0.03425693139433861, 0.003929394297301769, 0.006675355136394501, 0.0353645458817482, -0.008788392879068851, -0.006975333672016859, 0.002627287758514285, -0.00023528565361630172, -0.009223527275025845, -0.013344116508960724, -0.015031910501420498, 0.027136553078889847, -0.0034876668360084295, 0.016904305666685104, 0.01662740297615528, -0.027558501809835434, -0.006975333672016859, 0.01412208378314972, 0.020359007641673088, 0.00684347515925765, -0.02347087673842907, -0.0034876668360084295, 0.00757529167458415, -0.021796269342303276, -0.026991508901119232, 0.002261379500851035, -0.0137396939098835, 0.0013045786181464791, -0.030775858089327812, 0.008452152833342552, 0.01209805067628622, 0.006411637179553509, 0.0358128659427166, -0.015242884866893291, -0.003301416290923953, -0.004516166169196367, 0.012559556402266026, -0.028323283419013023, -0.0023882936220616102, -0.0357337519526481] - -
-
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "answer": "escaping the Earth's gravity (and go off into outer space, on your way to the moon, for instance)", - "question": "It takes approximately 24,840 MPH to achieve this" - }, - { - "answer": "the Milky Way", - "question": "This is the name of our own galaxy" - } - ] - } - } -} -``` - -
-
-
-## `nearObject`
-
-The `nearObject` operator can be used to look for objects which are the most similar to an existing Weaviate object.
-
-The object's identity is passed using the `id` argument as shown below.
-
-### Example
-
-import NearObjectSimple from './_snippets/academy.queries.nearObject.simple.mdx';
-
-Explain this query
- -- Because there was a `limit` of 2, the results contained (maximum of) two objects. -- Each object included `question` and `answer` properties, as specified in the `Get` function. -- And since the vector was provided externally, we have no way of telling what it represents. (Although we can infer from the results that it relates to something to do with space.) - -
-
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "answer": "escaping the Earth's gravity (and go off into outer space, on your way to the moon, for instance)", - "question": "It takes approximately 24,840 MPH to achieve this" - }, - { - "answer": "66,000", - "question": "Of 6,000, 66,000 or 666,000 MPH, the closest to the speed of the Earth around the sun" - } - ] - } - } -} -``` - -
-
-
-## `nearText`
-
-The `nearText` operator can be used to look for objects which are the most similar to a text input.
-
-The text is passed using the `concept` argument as shown below.
-
-### Example
-
-import NearTextSimple from './_snippets/academy.queries.nearText.simple.mdx';
-
-Explain this query
- -- When it comes to `nearObject` queries, Weaviate uses the vector of the object used as the query vector. - -
-
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "answer": "escaping the Earth's gravity (and go off into outer space, on your way to the moon, for instance)", - "question": "It takes approximately 24,840 MPH to achieve this" - }, - { - "answer": "the Milky Way", - "question": "This is the name of our own galaxy" - } - ] - } - } -} -``` - -
-
-
-### `nearExplain this query
- -With `nearText` Weaviate converts the input text to a vector. - -Accordingly, `nearText` requires some means of converting an input medium to a vector. This is called a `vectorizer` in Weaviate. - -:::info How to vectorizers work? -In Weaviate, vectorizers are added as optional `modules`. You will learn more about modules and vectorizers in later units. -::: - -
-
-
-:::tip Exercise: Threshold distances
-Try changing the one of the above queries to:
-- Use a `certainty` threshold rather than `distance`.
-- Return `certainty` as well as `distance`.
-Do they behave as you expected?
-:::
-
-In this context, `distance` measures the degree of difference. You might also see `certainty` used in some contexts. This is the opposite of `distance`, where higher values imply larger differences in meaning.
-
-We will cover later on what these values mean, exactly, and where they come form. But for now, remember that:
-
-- `distance` is a measure of dissimilarity (lower is more similar), and
-- `certainty` is a measure of similarity (higher is more similar)
-
-### `limit` threshold
-
-Several of the earlier examples use the `limit` clause. Use `limit` to limit the number of objects returned.
-
-In this Python example, `with_limit()` sets a limit of two responses.
-
-```
-response = client.query.get(
- "JeopardyQuestion",
- ["question", "answer"]
-).with_limit(2)
-```
-
-
-## Review
-
-### Review exercise
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "_additional": { - "distance": 0.1791926, - "id": "b1645a32-0c22-5814-8f35-58f142eadf7e" - }, - "answer": "escaping the Earth's gravity (and go off into outer space, on your way to the moon, for instance)", - "question": "It takes approximately 24,840 MPH to achieve this" - }, - { - "_additional": { - "distance": 0.18123823, - "id": "ef263438-b152-5540-97f7-99f4076bd124" - }, - "answer": "the Milky Way", - "question": "This is the name of our own galaxy" - } - ] - } - } -} -``` - -- -Now that you have seen how to retrieve individual objects with `Get`, let's take a look at how to compile information with `Aggregate`. - -`Aggregate` is a powerful function that allows you to combine information from multiple objects into a single result to get a quick overview of the results. - -## About `Aggregate` queries - -### `Aggregate` function syntax - -While the overall structure of `Aggregate` queries is similar to `Get` queries, there are some important differences as the queries relate to sets of results. - -The basic syntax for `Aggregate` queries is as follows: - -import AggregateSyntax from './_snippets/academy.queries.aggregate.syntax.mdx'; - -
-
-
-## Standalone `Aggregate` queries
-
-### Example 1
-
-Take a look at this query:
-
-import AggregateStandalone from './_snippets/academy.queries.aggregate.standalone.mdx';
-
-See relevant schema
- -import ShortSchema from './_snippets/academy.queries.schema.short.mdx'; - -
-
-
-### Response object from Weaviate
-
-This response includes an object to represent the `meta` information requested from `JeopardyQuestion` class. The `meta` object contains a `count` property, which is the total number of objects in the class.
-
-See the JSON response
- -```json -{ - "data": { - "Aggregate": { - "JeopardyQuestion": [ - { - "meta": { - "count": 10000 - } - } - ] - } - } -} -``` - -
-
-
-### `meta` property
-
-In the above `Aggregate` query we requested a `meta` property, for the count of the objects. Note that this is not an available property of the object class itself. This is a key difference between `Aggregate` and `Get` queries.
-
-A `Get` query retrieves a set of individual results, so we can select properties (e.g. `id`, or one of the properties unique to the data, such as `answer`) that apply to each of those individual results.
-
-An `Aggregate` query, on the other hand, returns an aggregation of the results. Accordingly, we must specify a sub-property that applies to the entire set of results.
-
-The `meta` property is one such property. It is available for all data types, and can be used with the `count` sub-property to return the number of retrieved objects.
-
-
-### Example 2
-
-Take a look at this query:
-
-import AggregateStandalone2 from './_snippets/academy.queries.aggregate.standalone.2.mdx';
-
-Explain this query
- -This query aggregates the objects in the `JeopardyQuestion` class to obtain the total count. Since there are no restrictions, it returns the total number of objects which is 10,000. - -
-
-
-See the JSON response
- -```json -{ - "data": { - "Aggregate": { - "JeopardyQuestion": [ - { - "answer": { - "count": 10000, - "topOccurrences": [ - { - "occurs": 19, - "value": "Australia" - }, - { - "occurs": 18, - "value": "Hawaii" - }, - { - "occurs": 16, - "value": "Boston" - }, - { - "occurs": 15, - "value": "French" - }, - { - "occurs": 15, - "value": "India" - } - ] - } - } - ] - } - } -} -``` -
-
-
-:::tip Available properties
-The list of available properties can be found on [this page](/weaviate/api/graphql/aggregate.md) in our documentation.
-:::
-
-## `Aggregate` with a search operator
-
-As we did with `Get` queries, we can also use search operators such as `nearText` in an `Aggregate` query. Take a look:
-
-### Example (with `nearText`)
-
-For example, let's say that now instead of individual questions, we would like to know something more holistic about the answers. Like how many questions might be related to this query:
-
-import AggregateWithNearText from './_snippets/academy.queries.aggregate.with.neartext.mdx';
-
-Explain this query
- -This response includes an object to represent aggregations from the `answer` property requested from `JeopardyQuestion` class. Because the property contains textual information, we can aggregate `topOccurrences` information, such as the `value` property, which is the token, as well as the number of times it `occurs`. - -
-
-
-See the JSON response
- -```json -{ - "data": { - "Aggregate": { - "JeopardyQuestion": [ - { - "meta": { - "count": 9 - } - } - ] - } - } -} -``` - -
-
-
-### Limit search space
-
-In order to produce meaningful aggregations with a vector search, you must limit the search space.
-
-This is different from aggregations in, say, a relational database. In a relational database, grouping or aggregating data can be done using `groupby` with functions such as SUM, AVG, MIN, MAX, etc. This allows you to find a result set and then aggregate the results.
-
-However, a vector search does not inherently exclude any results. This is because a vector search retrieves results based on *degrees* of similarity.
-
-**Accordingly, the search space must be limited** so that only relevant results are included in the aggregation. This can be done by setting an explicit `limit` or a threshold (`distance` or `certainty`) in the query.
-
-## `Aggregate` with `groupBy`
-
-So far, we have seen how to use `Aggregate` queries to compile information relating one set of results. This can be extended with the `groupBy` argument to compile information from multiple, subsets of results.
-
-### Example
-
-For example, let's say we want to know how many questions there are for each available `value` property. We can do this by adding the `groupBy` argument to the query:
-
-import AggregateGroupby from './_snippets/academy.queries.aggregate.groupby.mdx';
-
-Explain this query
- -This query aggregates the results that were restricted using the `distance` argument. This argument is a threshold that restricts the returned results to those that are relevant to the input. Without it, the search results would potentially include the entire class. - -This is called "limiting the search space". - -
-
-
-See the JSON response
- -```json -{ - "data": { - "Aggregate": { - "JeopardyQuestion": [ - { - "groupedBy": { - "path": [ - "round" - ], - "value": "Double Jeopardy!" - }, - "meta": { - "count": 5 - } - }, - { - "groupedBy": { - "path": [ - "round" - ], - "value": "Jeopardy!" - }, - "meta": { - "count": 3 - } - }, - { - "groupedBy": { - "path": [ - "round" - ], - "value": "Final Jeopardy!" - }, - "meta": { - "count": 1 - } - } - ] - } - } -} -``` - -
-
-
-### `groupBy` + `groupedBy`
-
-Results identified by an `Aggregate` query can be further grouped by using a `groupBy` argument. This argument takes a list of properties as an argument, and will group the results by the values of those properties.
-
-This is a particularly useful query pattern for identifying characteristics for subsets of results of a vector search.
-
-When the `groupBy` argument is used, additional property `groupedBy` is made available. This property and its sub-properties can be used to identify the group that the result belongs to.
-
-:::note Exercise
-Try out the above query again, with these changes.
-- Instead of `round` try grouping by the `points` property.
-- Instead of `distance`, try adding an `.with_object_limit(9)` in the method chain. Are the results the same?
-:::
-
-## Review
-
-### Review exercise
-
-Try out the above `nearText` query again, with these changes.
-- Change the distance to another value - say, to 0.1, 0.19, 0.21 or 0.25 - how do the results change? Are they in line with your expectations?
-
-### Key takeaways
-
-- The `Aggregate` function is used to compile information from multiple objects, providing an overview.
-- Search operators, like `nearText`, can be used in `Aggregate` queries.
- - To produce meaningful aggregations, the search space must be limited by setting an explicit limit or a threshold (distance or certainty) in the query.
-- The `groupBy` argument can be used to compile information from multiple subsets of results, refining the aggregation.
-- When using the groupBy argument, the additional property groupedBy is made available, helping to identify the group that the result belongs to.
-
-import Quiz from '/src/components/Academy/quiz.js'
-export const varName = [{
- questionText: 'questionText',
- answerOptions: [
- {
- answerText: 'answerOne',
- isCorrect: false,
- feedback: 'feedbackOne',
- },
- {
- answerText: 'answerTwo',
- isCorrect: false,
- feedback: 'feedbackTwo',
- },
- {
- answerText: 'answerThree',
- isCorrect: false,
- feedback: 'feedbackThree',
- },
- ]
-}];
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-Explain this query
- -This query supplies an additional `groupedBy` argument, as a result of which the counts are of each `round`. The query also requests `groupedBy` a property so that each count is identifiable by the `value` of each `round` group. - -- -## Available operators - -So far, you've seen different query functions such as `Get`, and `Aggregate`, and search operators such as `nearVector`, `nearObject` and `nearText`. - -Now, let's take a look at filters. - -A filter is a way to specify additional criteria to be applied to the results. There are a number of available filters in Weaviate. - -### Available filters - -There exist many available filters, but we do not need to cover them all at this moment. For now, let's explore a few of the most commonly used filters: - -- `where`: Apply a Boolean condition to filter the available data. -- `limit`: Restrict the maximum objects to be retrieved. -- `offset`: For pagination of search results. - -## Filter data with `where` - -The `where` filter is analogous to the `WHERE` clause in a SQL query. As in the SQL clause, the `where` filter can be used to apply a boolean conditional to the data. - -### Single operand example - -We ran an example query like this earlier: - -import GetWithAdditional from './_snippets/academy.queries.get.with.additional.mdx'; - -
-
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "_additional": { - "distance": 0.18400955, - "id": "ddcc3f06-5410-5944-85c4-3cb56ab27088" - }, - "answer": "space shuttles", - "question": "These transports, first sent up in 1981, lift off like a rocket & land like a plane" - }, - { - "_additional": { - "distance": 0.2267003, - "id": "36ffe6ca-9b73-5a54-80eb-a93f01822956" - }, - "answer": "Robert Goddard", - "question": "He's been called the \"Father of Modern Rocketry\"" - } - ] - } - } -} -``` - -
-
-
-We can apply the query to filter the data in any number of ways. For example, consider this query:
-
-import FilterWhereGreater from './_snippets/academy.queries.filter.where.greater.mdx';
-
-Explain this query
- -Observe that the results have changed. The previous results have been removed as they do not contain the text `rocket` in the `question` property. - -This approach of combining a vector search with a filter is a powerful way to find objects that are similar to a given input, but also meet additional criteria as you see. And while filtering may remove some objects which might otherwise be "closer" to the query vector than the remaining ones, it provides a powerful strategy to find the most relevant objects by removing false positive. - -
-
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "_additional": { - "distance": 0.18251508, - "id": "15f06117-012c-506d-b5c5-24df2e750f35" - }, - "answer": "the Milky Way", - "points": 400, - "question": "Into the 20th Century it was thought the universe was one big galaxy--this one" - }, - { - "_additional": { - "distance": 0.19289112, - "id": "584a6c68-0ebe-561f-b32a-3a735eadf02e" - }, - "answer": "Asteroid", - "points": 400, - "question": "A 1991 photo of Gaspra taken by the Galileo probe was the first close-up of one of these minor planets" - } - ] - } - } -} -``` - -
-
-
-:::note Exercise
-Try filtering for `JeopardyQuestion` objects with:
-- a `points` value equal to 200
-- a `points` value greater than or equal to 600
-
-You can find the list of available operators on [this page](/weaviate/api/graphql/filters#filter-structure).
-:::
-
-### Multiple operands example
-
-The query syntax can extend to beyond a single operand to take advantage of multiple conditions:
-
-import FilterMultipleOperands from './_snippets/academy.queries.filter.multiple.operands.mdx';
-
-Explain this query
- -This query has been modified to only return `JeopardyQuestion` objects with a `points` value greater than 200. - -Accordingly, the returned data set is very different. - -
-
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "_additional": { - "distance": 0.2267003, - "id": "a488fbe5-c2c6-50ad-8938-4b9f20dc56d1" - }, - "answer": "Robert Goddard", - "points": 400, - "question": "He's been called the \"Father of Modern Rocketry\"" - }, - { - "_additional": { - "distance": 0.24946856, - "id": "c00decd4-4cf1-5b03-a789-a57077e082fb" - }, - "answer": "Huntsville", - "points": 1000, - "question": "A campus for the University of Alabama is here, nicknamed \"Rocket City, U.S.A.\"" - } - ] - } - } -} -``` - -
-
-
-You can apply these filters to an `Aggregate` query also. Try it yourself.
-
-:::note Exercise
-Try these:
-- adding a `where` filter to an `Aggregation` query, following the above pattern.
-:::
-
-## Result pagination with `offset`
-
-When you query for data, you can use the `offset` operator to skip a number of results. This is useful for pagination, where you want to show a certain number of results per page.
-
-The `offset` operator works in conjunction with the existing `limit` operator to shows results from the `offset+1` to `offset+1+limit`.
-
-For example, to list the first ten results, set `limit`: 10. Then, to "display the second page of 10", set `offset`: 10, `limit`:10 and so on.
-
-The syntax, using `offset` is as follows:
-
-import FilterPaginationNeartext from './_snippets/academy.queries.filter.pagination.nearText.mdx';
-
-Explain this query
- -This query has been modified to only return `JeopardyQuestion` objects with a `points` value great than than 400, AND include the text `rocket` in the `question` field. - -
-
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "answer": "the Milky Way", - "question": "Into the 20th Century it was thought the universe was one big galaxy--this one" - }, - { - "answer": "space shuttles", - "question": "These transports, first sent up in 1981, lift off like a rocket & land like a plane" - } - ] - } - } -} -``` - -
-
-
-:::tip
-So, the `n` th page would have `offset`: `n*m`, `limit`: `m`, where `m` is the number of results per page.
-:::
-
-The `offset` operator is available with all vector search functions including `Get` and `Aggregate`.
-
-## Review
-
-Explain this query
- -This query retrieves the next 2 results (`limit`: 2) after the first 2 results (`offset`: 2). - -We can confirm this by comparing the results of two queries with different result limits. The query below retrieves the top 4 results. The last two results from that query are the same as the result in the query that uses limit with pagination. - -```graphql -{ - Get { - JeopardyQuestion(limit: 4) { - answer - question - } - } -} -``` - -import FilterPaginationRelated from './_snippets/academy.queries.filter.pagination.related.mdx'; - -- -In this unit, we have looked at various ways to construct a vector search. - -We started by learning how to use `Get` and `Aggregate` vector search functions, adding search operators such as `nearVector`, `nearObject` and `nearText`, before wrapping up with various filters such as `where`, `limit` and `offset`. - -Now let's stop to take a look at how these searches are actually performed, starting with the vector search process. -## The vector search process - -### Vector search, behind the scenes - -As the name suggests, vector search relies on vectors to perform its operations. When you perform a vector search in Weaviate, regardless of whether `nearVector`, `nearObject` or `nearText` is used, the input is converted to a vector. - -The input vector is then compared to the stored vectors in Weaviate to return the most relevant objects. - -For queries using the `nearVector` operator, Weaviate simply takes the provided vector and performs the vector search. - -For queries such as `nearObject` and `nearText` where the vector is not directly provided, Weaviate obtains the vector using a suitable method. - -#### `nearObject` - -If the `nearObject` operator is used, Weaviate retrieves the associated vector for the object, which becomes the input vector. - -Let's confirm this by performing a vector search using the `nearObject` operator, and replicating it with an equivalent `nearVector` operator. - -Here is the `nearObject` query: - -import NearobjectExample from './_snippets/academy.queries.nearobject.example.mdx'; - -
-
-
-Now, try running this `nearVector` query and compare the results:
-
-import NearobjectEquivalentNearvector from './_snippets/academy.queries.nearobject.equivalent.nearvector.mdx';
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "_additional": { - "distance": 3.5762787e-07, - "id": "d53fd7ea-35c1-5f8d-a35a-e53511db1a2a" - }, - "answer": "meerkats", - "question": "Group of mammals seen here: [like Timon in The Lion King]" - }, - { - "_additional": { - "distance": 0.12663543, - "id": "9eaf38fe-e7f0-5da3-b703-6b44c49faf7d" - }, - "answer": "hyena", - "question": "It's the amused African mammal heard here" - } - ] - } - } -} -``` - -
-
-
-See the full `meerkat_vector`
- -import MeerkatsVector from './_snippets/meerkats.vector.mdx'; - -
-
-
-#### `nearText`
-
-If the `nearText` operator is used, Weaviate converts the input text to a vector to use as the input. The specific method would depend on the `vectorizer` applicable to the relevant class. Depending on the setting, Weaviate may use the `text2vec-openai` module or the `text2vec-transformers` module to do this.
-
-:::info Vectorizer setting
-We will cover how to set the vectoriser in the next section.
-:::
-
-import NeartextExample from './_snippets/academy.queries.neartext.example.mdx';
-
-Explain this query
- -If you run the second (`nearVector`) query, you will see that it returns the same results as the `nearObject` query. The distances are also identical. This is because the vector of object in the `nearObject` query is identical to the vector you specify in the `nearVector` query. - -
-
-
-Now, try running this `nearVector` query and compare the results:
-
-import NeartextEquivalentNearvector from './_snippets/academy.queries.neartext.equivalent.nearvector.mdx';
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "_additional": { - "distance": 0.1800943, - "id": "92710bd6-de3c-5220-a60a-d386b2748e28" - }, - "answer": "Two Guys and a Girl", - "question": "In 1999 an ABC sitcom dropped \"a Pizza Place\" from its name, which changed to this" - }, - { - "_additional": { - "distance": 0.18657643, - "id": "7e7a6572-02bd-529f-8943-38ccd4a2a90b" - }, - "answer": "Beavis & Butthead of the Class", - "question": "2 MTV cartoon teens join Howard Heh-Heh-Hesseman's honors program on '80's TV" - } - ] - } - } -} -``` - -
-
-
-See the full `popular_90s_comedy_vector`
- -import Popular90sComedyVector from './_snippets/popular.90s.comedy.vector.mdx'; - -
-
-
-### Filtering
-
-Filters can be applied during the vector search process to further refine the results.
-
-Under the hood, Weaviate implements "pre-filtering". This means that Weaviate applies the filter on the entire database to end up with an "allow list", from which vector search results are returned.
-
-This, combined with efficient filtering methods, allows Weaviate to easily return the right number of results even under challenging conditions, such as when restrictive filters are applied to a large database.
-
-### Conceptual diagram (filtering + vector search)
-
-The conceptual diagram below shows how the vector search process works. When an input is provided containing filters and search operators, Weaviate first performs a pre-filtering step to obtain an "allow list" of objects.
-
-The input is used to determine an input vector, which may involve an additional step, such as retrieving the object's vector with `nearObject` or vectorizing the input text with `nearText`. The input vector is then compared to perform a vector search, returning the most relevant objects from the allow list.
-
-import SearchConceptualImg from './images/search-conceptual-dark.png';
-
-Explain this query
- -When you run the second (`nearVector`) query, once again the query returns the same objects as the `nearText` query as well as distances. - -In this case, Weaviate vectorizes the input text using the `text2vec-openai` module, which relies on the OpenAI inference API to convert the input to a vector. The vectorization process is deterministic, meaning that the same input text will always result in the same vector. For the `nearVector` query, we generated the vector directly using the OpenAI API and used it as the input vector, leading to the same results. - -- -In this unit, you will learn how to efficiently retrieve relevant objects or aggregated information from Weaviate. - -You have already encountered some examples of vector searches. In this section, we will delve deeper by reviewing the various vector search methods available in Weaviate, such as `nearVector`, `nearObject`, and `nearText`. - -Along with vector search methods, you will also discover filters that can be employed to accompany search operators. For instance, you will learn how to search for data objects that exclude specific criteria. - -As we examine these capabilities, we will simultaneously use them as a means to gain insight into the inner workings of Weaviate. - -Upon completing this unit, you will possess a thorough understanding of how to effectively query Weaviate to obtain desired results, as well as the underlying mechanisms that make it all possible. - -### Prerequisites - -- (**Required**) A Python (3) environment with `weaviate-client` installed. -- (**Required**) Complete [101A Weaviate Academy Preparation](../setup.mdx) -- (*Recommended*) Complete [Hello, Weaviate](../101_hello_weaviate/index.mdx) - -## Learning objectives - -import LearningGoalsExp from '/src/components/Academy/learningGoalsExp.mdx'; - -
- -You've seen some of the powerful queries that Weaviate enables. But how does Weaviate actually store data such that it can supports these queries? - -In this section, we'll take a look at some of the key components that allow Weaviate to perform these queries at speed. In particular, we'll take a look at indexes, which are the backbone of Weaviate's data structure, and the schema, which acts as a blueprint for your data. - -## Indexes - -An index is a data structure that allows for efficient retrieval of data. In Weaviate, there are two main indexes: the inverted index and the vector index. - -The **inverted index** is the kind of index that you may be familiar with. You can think of it as a reference table that for example allows you to quickly look up a term and find objects that contain that term. - -The **vector index** allows for efficient retrieval of vectors based on similarity. This is the index that allows Weaviate to perform vector searches fast. Let's dig in a little more. - -### Inverted index - -An inverted index deconstructs text into a set of constituent terms and stores them in a data structure, such as a hash table. Take, for example, a data object containing the text "Conference (Chicago, IL)". - -The user might want to search for this object based on any of the contained terms such as "conference", "Chicago", or "IL". The inverted index allows Weaviate to quickly retrieve the ID of the object containing the term. - -This is done by mapping the object ID to "tokens", where a token is a term that has been extracted from the text. By default, Weaviate uses a `word` tokenization, where only alpha-numeric characters are kept, converted into lowercase, and then split into tokens based on whitespace. - -So an input text `Conference (Chicago, IL)` is indexed by three tokens: `conference`, `chicago`, `il`. - -We will cover more about different available tokenization methods later on. - -### Vector index - -Each object in Weaviate can be associated with a vector. These vectors are what enables similarity searches that you have seen before. As we mentioned, however, brute-force similarity searches are computationally expensive, as well as growing linearly with the size of the dataset. - -To tackle this problem Weaviate uses vector indexes that utilize an Approximate Nearest Neighbor (ANN) algorithm. The ANN algorithm enables each vector index to organize a set of vectors, so that similar ones to a query can be retrieved at lightning-fast speeds. Weaviate currently uses an HNSW-based ANN index. - -import HNSWLayersImg from '/docs/weaviate/concepts/img/hnsw-layers.svg'; - -
-
-
-This is a lot of information, and can be quite intimidating. Let's break it down.
-
-First of all, you see that the first level key in the object is `classes`, which contains a list of classes. In this case, there is only one class, `Article`.
-
-The schema specifies for each class:
-- The metadata such as its name (`class`),
-- Its data `properties`,
-- The `vectorizer`,
-- Module configurations (`moduleConfig`),
-- The index configurations (for inverted `invertedIndexConfig` and vector `vectorIndexConfig` indexes),
-- and more.
-
-:::info Auto-schema
-Any missing information required for schema definition will be automatically inferred by Weaviate based on default values and the imported data.
-:::
-
-## Review
-
-### Review exercise
-
-Example schema
- -```json -{ - "classes": [ - { - "class": "Article", - "invertedIndexConfig": { - "bm25": { - "b": 0.75, - "k1": 1.2 - }, - "cleanupIntervalSeconds": 60, - "stopwords": { - "additions": null, - "preset": "en", - "removals": null - } - }, - "moduleConfig": { - "text2vec-openai": { - "model": "ada", - "modelVersion": "002", - "type": "text", - "vectorizeClassName": true - } - }, - "properties": [ - { - "dataType": [ - "text" - ], - "moduleConfig": { - "text2vec-openai": { - "skip": false, - "vectorizePropertyName": false - } - }, - "name": "title", - "tokenization": "word" - }, - { - "dataType": [ - "text" - ], - "moduleConfig": { - "text2vec-openai": { - "skip": false, - "vectorizePropertyName": false - } - }, - "name": "body", - "tokenization": "word" - } - ], - "replicationConfig": { - "factor": 1 - }, - "shardingConfig": { - "virtualPerPhysical": 128, - "desiredCount": 1, - "actualCount": 1, - "desiredVirtualCount": 128, - "actualVirtualCount": 128, - "key": "_id", - "strategy": "hash", - "function": "murmur3" - }, - "vectorIndexConfig": { - "skip": false, - "cleanupIntervalSeconds": 300, - "maxConnections": 32, - "efConstruction": 128, - "ef": -1, - "dynamicEfMin": 100, - "dynamicEfMax": 500, - "dynamicEfFactor": 8, - "vectorCacheMaxObjects": 1000000000000, - "flatSearchCutoff": 40000, - "distance": "cosine", - "pq": { - "enabled": false, - "segments": 0, - "centroids": 256, - "encoder": { - "type": "kmeans", - "distribution": "log-normal" - } - } - }, - "vectorIndexType": "hnsw", - "vectorizer": "text2vec-openai" - } - ] -} -``` - -- -A `schema` in Weaviate is the blueprint that defines its data structure for each `class` of objects. A class is a collection of objects of the same type. - -In this section, you will learn how to define a schema and gain insight into some key considerations while doing so. - -## How to define a schema - -As you learned earlier, a schema definition includes a great deal of information. Let's cover a few of those properties in this section, starting with: -- The metadata such as its name (`class`), -- Its data `properties`, -- The `vectorizer`, and -- Module configurations (`moduleConfig`). - -### Metadata definition - -You can define for each class and property a name and description. - -For classes, these are called: -- `class` (required), and -- `description` (optional). - -For properties, these are called: -- `name` (required), and -- `description` (optional). - -In defining a class, the only required parameter is `class`, as the rest can be inferred by Weaviate. However, it is recommended to include a description for each class and property, as this will help you and others understand the data structure. - -To define a class, you can use this syntax. - -
-
-
-Note that most data types can include one such instance, or an array of instances, such as `text` or `text[]`.
-
-Available data types in Weaviate
-
-
-
-See the JSON response
- -```json -{ - "generative-openai": { - "documentationHref": "https://beta.openai.com/docs/api-reference/completions", - "name": "Generative Search - OpenAI" - }, - "qna-openai": { - "documentationHref": "https://beta.openai.com/docs/api-reference/completions", - "name": "OpenAI Question & Answering Module" - }, - "ref2vec-centroid": {}, - "text2vec-cohere": { - "documentationHref": "https://docs.cohere.com/docs/embeddings", - "name": "Cohere Module" - }, - "text2vec-huggingface": { - "documentationHref": "https://huggingface.co/docs/api-inference/detailed_parameters#feature-extraction-task", - "name": "Hugging Face Module" - }, - "text2vec-openai": { - "documentationHref": "https://beta.openai.com/docs/guides/embeddings/what-are-embeddings", - "name": "OpenAI Module" - } -} -``` - -- -So far, you've learned that data in Weaviate is represented by objects, which belong to a `class`, and have a set of `properties`. In the previous section on the `schema`, you learned how to create a framework for this structure. - -In this section, you will learn how to import data into Weaviate, including our recommended best practices, and some key considerations. Once you're done with this section, you will be ready to import a real dataset into Weaviate by putting together what we've learned about the schema and imports. - -## Imports: A high-level view - -### Object creation - -To create a Weaviate object, you must: - -- Load your source data, -- Build a data object with any desired properties, and -- Add it to your desired class in Weaviate. - -Optionally, you can manually specify: - -- An object ID, and -- A vector - -:::info Optional properties - -- An object ID is required. So, if one is not specified, Weaviate will create one. -- If a vector is not specified, Weaviate will create one if a vectorizer is set for the class. - -We'll cover these in more detail later on. -::: - -### Use batch imports - -Weaviate offers a "batch import" process to maximize the import speed. Take a look at this diagram that shows the object creation process: - -import ImportConceptualImg from './images/academy_103_import_process_conceptual.png'; - -
- -It's time to put what we've learned into action! In this section, we will: - -- Download a small dataset, -- Build a schema corresponding to the dataset, and -- Import it to your WCD instance. - -### Dataset used - -We are going to use data from a popular quiz game show called *Jeopardy!*. - -The original dataset can be found [here on Kaggle](https://www.kaggle.com/datasets/tunguz/200000-jeopardy-questions), but we'll use a [small subset from it, just containing 100 rows](https://raw.githubusercontent.com/weaviate-tutorials/edu-datasets/main/jeopardy_100.json). - -Here's a preview of a few rows of data. - -| | Air Date | Round | Value | Category | Question | Answer | -|---:|:-----------|:-----------------|--------:|:-----------------|:---------------------------------------------------------------------------------------------------------------------|:-------------------------------------------| -| 0 | 2006-11-08 | Double Jeopardy! | 800 | AMERICAN HISTORY | Abraham Lincoln died across the street from this theatre on April 15, 1865 | Ford's Theatre (the Ford Theatre accepted) | -| 1 | 2005-11-18 | Jeopardy! | 200 | RHYME TIME | Any pigment on the wall so faded you can barely see it | faint paint | -| 2 | 1987-06-23 | Double Jeopardy! | 600 | AMERICAN HISTORY | After the original 13, this was the 1st state admitted to the union | Vermont | - -For now, let's keep it simple by populating Weaviate with just the `Round`, `Value`, `Question` and `Answer` columns. - -:::note Exercise -Can you remember what the next steps should be? -::: - -## Build a schema - -The next step is to build a schema, making some decisions about how to represent our data in Weaviate. - -### Add class names & properties - -First of all, we'll need a name. The name refers to each row or item (*note: singular*), so I called it `JeopardyQuestion`. Then, I need to define properties and types. - -You saw above that we'll be populating Weaviate with `Round`, `Value`, `Question` and `Answer` columns. We need names for Weaviate `properties` - these names are sensible, but we follow the GraphQL convention of capitalizing classes and leaving properties as lowercases, so the names will be `round`, `value`, `question` and `answer`. - -Then, we should select datatypes. All of `round`, `question` and `answer` are text, so we can simply choose `text` as our datatype. `value` is a number, but I know that values in *Jeopardy!* represent dollar amounts, meaning that they are always integers. So we'll use `int`. - -
-
-
-:::note The retrieved schema is even longer!
-Although we've defined a lot of details here, the retrieved schema is still longer. The additional details relate to the vector index, the inverted index, sharding and tokenization. We'll cover many of those as we go.
-:::
-
-If you see a schema that is close to the example response - awesome! You're ready to import the data.
-
-## Import data
-
-Here, we'll show you how to import the requisite data, including how to configure and use a batch.
-
-### Load data
-
-We've made the data available online - so, fetch and load it like so:
-
-See the full schema response
- -
-
-
-### Specify object UUID
-
-You could specify an object UUID at import time to serve as the object identifier. The Weaviate Python client, for example, provides a function to create a deterministic UUID based on an object. So, it could be added to our import script as shown below:
-
-The answer is...
- -That you will end up with duplicate items!- - - -The previous units introduced you to the fundamentals of Weaviate, its capabilities, and how to execute basic queries on existing data. Now that you're familiar with the various query types, it's time to explore how to populate your own Weaviate instance with data. - -In this unit, we'll examine how to use Weaviate to effectively structure your data so that you can retrieve the right information the way you want. We'll delve into defining a schema for your data and importing data into Weaviate. - -By the end of this unit, Weaviate's overall data architecture will start to become clearer in your mind. This will start to empower you to build a vector database that really suits your needs and goals. - -Let's get started. - -### Prerequisites - -- (**Required**) A Python (3) environment with `weaviate-client` installed. -- (**Required**) Complete [101A Weaviate Academy Preparation](../setup.mdx) -- (*Recommended*) Complete [Hello, Weaviate](../101_hello_weaviate/index.mdx) -- (*Recommended*) Complete [Queries 1](../102_queries_1/index.mdx) - -## Learning objectives - -import LearningGoalsExp from '/src/components/Academy/learningGoalsExp.mdx'; - -
-
-
-:::tip Exercise
-Try varying the boost factor, and the query string. What happens to the results?
-:::
-
-## Tokenization and `bm25` searches
-
-### Why tokenization matters
-
-In an earlier unit, we briefly discussed [the inverted index](../103_schema_and_imports/10_data_structure.mdx#-inverted-index), and that it stores a "tokenized" index of data.
-
-When a BM25 query is submitted, Weaviate will search each property according to its tokenization property. For example, if a property is tokenized with the `word` tokenization option, it will tokenize the query string into its constituent, lowercase, words, and search for each word in the index. On the other hand, if a property uses a `field` tokenization, Weaviate will look for the entire query string in the index.
-
-:::tip This applies to the inverted index only
-This is different to any tokenization in the context of, for example, language models or vectorization models. Tokenization in the context of the current section only applies to the inverted index.
-:::
-
-More concretely, let's take a look at some examples.
-
-### `word` tokenization
-
-In this example, we search through the `question` property with the query string `Jeopardy`. The `question` property is tokenized with the `word` tokenization option.
-
-See the JSON response
- -
-
-
-Now, let's take a look at the same query, but with the `field` tokenization option.
-
-### `field` tokenization
-
-In this example, the query string remains the same (`Jeopardy`), however we are now searching the `round` property, which is tokenized with the `field` tokenization option.
-
-See the JSON response
- -
-
-
-:::tip Exercise
-Try changing the query to `Jeopardy!`. What happens to the results?
-:::
-
-### Rules of thumb
-
-The [full list of tokenization options](/weaviate/config-refs/collections.mdx#tokenization) are `word`, `whitespace`, `lowercase` and `field`. A rule of thumb on when to use each option is to use `word` for long text where you want to retrieve partial matches, and `field` for short text where you only want to retrieve exact matches. The others are somewhere in between, and may be useful in specific situations, where for example you want case to matter (`whitespace`) or special characters to be respected (`lowercase`).
-
-## BM25F scoring
-
-The exact algorithm used for scoring and ranking the results is called the BM25F algorithm. The details are beyond the scope of this course, but the gist is that the BM25F algorithm is a variant of the BM25 algorithm, where the `F` stands for "field". It is used to score and rank results based on the fields that are searched.
-
-If you would like to delve into the details of the exact algorithm, you can review [this Wikipedia page](https://en.wikipedia.org/wiki/Okapi_BM25#Modifications).
-
-## Review
-
-See the JSON response
- -
-
-
-## `hybrid` search parameters
-
-A `hybrid` search includes multiple parameters, some of which you may be familiar with from the earlier `bm25` search discussions.
-
-The `query` parameter and `properties` parameter are the same as in a `bm25` search, with the exception that currently, the boost parameter is not supported in a `hybrid` search. Some of the parameters, however, are unique to a `hybrid` search.
-
-[//]: # (### `vector`)
-
-[//]: # ()
-[//]: # (The `vector` parameter is optional. If you do not include a `vector` parameter, the `hybrid` search will generate a vector from the query string. If you do include a `vector` parameter, the `hybrid` search will use the vector you provide.)
-
-[//]: # ()
-[//]: # (In this way, you may be able to perform a hybrid search where the `bm25` search and the vector search are based on different concepts. For example, you could perform a `bm25` search with the query string `italian`, and have the vector search be based on a vector of `food`.)
-
-[//]: # ()
-[//]: # (:::warning)
-
-[//]: # (TODO - complete this section after I get responses on the slack comment/question)
-
-[//]: # (:::)
-
-### `alpha`
-
-The optional `alpha` parameter determines the weighting of the BM25 search ranking and the vector search ranking. If you do not include an `alpha` parameter, the `hybrid` search will use a default value of `0.5`, which weights each equally.
-
-Otherwise, an `alpha` value of 1 is the same as a pure vector search, whereas an `alpha` value of 0 is the same as a pure BM25 search.
-
-:::tip Exercise
-Try varying the `alpha` parameter above. What happens to the results?
-:::
-
-## Review
-
-See the JSON response
- -
-
-
-#### Grouped Task
-
-A grouped task search does not require any properties to be specified. It can be carried out as follows:
-
-See the JSON response
- -
-
-
-## Object properties
-
-You already saw that `single prompt` generative searches require properties to be specified. You can also specify specific properties to be used with `grouped task` searches.
-
-### Why specify properties?
-
-In `single prompt` examples, each property serves to form the model prompt, replacing placeholder text like (`{summary}`) with the retrieve summary text.
-
-You can also specify the properties to be used for each `grouped task`, so that the generative model only receives the data you want to pass to it.
-
-#### Context window length
-
-Another reason for specifying the properties is to reduce the chance of exceeding the model context length. Generative models are typically transformer-based, and many have a limited context window.
-
-As a result, only passing the required properties to the generative module may allow you to include results from more objects without exceeding the context window limit.
-
-### How to specify properties
-
-:::tip Generative search properties unrelated to returned properties
-In both `single prompt` and `grouped task` searches, the properties to be used in the generative search do not need to be specified in the properties to be returned.
-:::
-
-You saw earlier that for `single prompt` searches, you can specify in the prompt the specific properties to be used.
-
-For `grouped task` searches, the properties to be used must be passed as an additional parameter. Take a look at the following example, where we ask the generative model to simply repeat the provided prompt:
-
-
-See the JSON response
- -
-
-
-## Generative search parameters
-
-### Optional model parameters
-
-Generative modules use what are called "large language models", or LLMs, to produce these outputs from search results and prompts.
-
-Weaviate allows you to tune the behavior of these models through additional, optional parameters made available by their providers.
-
-While we cannot cover every single parameter, some groups of commonly available parameters are described below.
-
-:::tip Parameter names will vary
-Each model will use different parameter names, so you should consult the exact module documentation and the corresponding model documentation.
-:::
-
-- `model`: Determines the actual language model to be used.
-- `temperature`, `k` and `p`: Determine how "random" the model will behave. At one extreme, the model will behave deterministically, whereas at the other extreme, it will produce more unpredictable (potentially incoherent) outputs. Some providers such as OpenAI recommend using only one of these settings.
-- `max_tokens`: Determines the maximum length of a generated output. The longer the value, the more likely you are to potentially exceed the context limit in combination with the input length.
-- `xxx_penalty`: Determines how much to penalise certain aspects such as the same tokens appearing again, or the number of times that the same tokens appear.
-
-This example specifies various parameters for the `generative-openai` module:
-
-```json
-{
- "classes": [
- {
- "class": "JeopardyQuestion",
- "moduleConfig": {
- "generative-openai": {
- "model": "gpt-3.5-turbo",
- "temperatureProperty": 0.3,
- "maxTokensProperty": 512,
- "frequencyPenaltyProperty": 0.1,
- "presencePenaltyProperty": -0.1
- }
- }
- }
- ]
-}
-```
-
-As a starting point, we recommend that you try using the default options, including the model, if possible. Then if something is not working to your satisfaction, you could try a different model parameter or approach.
-
-## Review
-
-### Key takeaways
-
-- Generative search transforms data before delivery, turning a database into a more active participant in data processing.
-- To use generative search, a generative module must be enabled in the Weaviate instance; generative modules are enabled by default in WCD.
-- A generative search involves performing a search and then calling a generative model using the search results and a user-provided prompt.
-- Single prompt search generates a response for each result, while grouped task search generates one response for the whole result set.
-- Object properties are used in generative searches; they form the model prompt in single prompt examples and can be specified in grouped tasks.
-- Generative modules use large language models (LLMs), and Weaviate exposes optional parameters for tuning their behavior.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-See the JSON response
- -
-
-
-### How to specify properties
-
-You can specify the object properties in which the QnA module is to search through for the answer. Similarly to the case of a generative query, this may be useful if you want to reduce the length of the input, or be very specific about where the information should come from.
-
-:::note `nearText` search unaffected
-Specifying properties to search only affects the answer extraction part of the query, as the underlying object vectors do not change.
-:::
-
-See the JSON response
- -
-
-
-### Object limits
-
-As this is also a two-step search, you can specify the number of objects for Weaviate to initially retrieve before attempting to extract the answer.
-
-By setting the number of objects, you may increase the chance of retrieving the object that contains the answer to the specific question.
-
-## Review
-
-### Key takeaways
-
-- Question and Answer (QnA) search is another two-step search, which attempts to extract an answer to a specific question from the retrieved data before delivery.
-- To perform a QnA search, a `qna-xxx` module must be enabled in the Weaviate instance. The `qna-openai` module is enabled by default in WCD.
-- The QnA module will look for an answer in each retrieved object, returning the answer as an additional property.
-- If the QnA module does not identify an answer, it will indicate so in the response.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-See the JSON response
- -
-
-
-
-```python
-response = client.query.get(
- "JeopardyQuestion",
- ["question", "answer", "points"]
-).with_limit(2).with_near_text(
- {"concepts": "Intergalactic travel"}
-).with_additional(
- ["distance", "id"]
-).with_where({
- "operator": "And",
- "operands": [
- {
- "path": ["question"],
- "operator": "Like",
- "valueText": "*rocket*"
- },
- {
- "operator": "Or",
- "operands": [
- {
- "path": ["points"],
- "operator": "Equal",
- "valueInt": 200
- },
- {
- "path": ["points"],
- "operator": "Equal",
- "valueInt": 1000
- }
- ]
- }
- ]
-}).do()
-
-print(json.dumps(response, indent=2))
-```
-
-
-
-
-```graphql
-{
- Get {
- JeopardyQuestion (
- limit: 2
- nearText: {
- concepts: ["Intergalactic travel"],
- }
- where: {
- operator: And,
- operands: [
- {
- path: ["question"],
- operator: Like,
- valueText: "*rocket*"
- }
- {
- operator: Or,
- operands: [
- {
- path: ["points"],
- operator: Equal,
- valueInt: 200
- },
- {
- path: ["points"],
- operator: Equal,
- valueInt: 1000
- }
- ]
- },
- ]
-
- }
- ) {
- question
- answer
- points
- _additional {
- distance
- id
- }
- }
- }
-}
-```
-
-
-
-
-
-
-it should produce a result like this:
-
-- Nested where filter example -
- -
-
-
-:::
diff --git a/docs/academy/py/zero_to_mvp/_109_tmp_leftovers/_graphql.mdx b/docs/academy/py/zero_to_mvp/_109_tmp_leftovers/_graphql.mdx
deleted file mode 100644
index 4ae1a93c2..000000000
--- a/docs/academy/py/zero_to_mvp/_109_tmp_leftovers/_graphql.mdx
+++ /dev/null
@@ -1,159 +0,0 @@
----
-title: A little more about GraphQL
-sidebar_position: 99
----
-
-import imageUrl from '../../tmp_images/academy_placeholder.jpg';
-
-## About GraphQL
-
-See the JSON response
- -```json -{ - "data": { - "Get": { - "JeopardyQuestion": [ - { - "_additional": { - "distance": 0.18400955, - "id": "48fe1f8f-8e09-5aee-afe9-1728ab2fe8a0" - }, - "answer": "space shuttles", - "points": 200, - "question": "These transports, first sent up in 1981, lift off like a rocket & land like a plane" - }, - { - "_additional": { - "distance": 0.24946856, - "id": "c00decd4-4cf1-5b03-a789-a57077e082fb" - }, - "answer": "Huntsville", - "points": 1000, - "question": "A campus for the University of Alabama is here, nicknamed \"Rocket City, U.S.A.\"" - } - ] - } - } -} -``` - -
-
-
-See the JSON response
- -```json -{ - "data": { - "Aggregate": { - "JeopardyQuestion": [ - { - "meta": { - "count": 10 - }, - "round": { - "topOccurrences": [ - { - "occurs": 5, - "value": "Double Jeopardy!" - }, - { - "occurs": 4, - "value": "Jeopardy!" - }, - { - "occurs": 1, - "value": "Final Jeopardy!" - } - ], - "type": "text" - } - } - ] - } - } -} -``` - -
-
-
-### Available sub-properties
-
-Sub-properties allow further granular aggregations of data from Weaviate. These properties that can be queried will vary according to the data type that is being aggregated.
-
-Along with the `string` data type, `text` properties may be queried to retrieve one or more of:
-
-- count
-- type
-- topOccurrences
-
-However, `int` data types, for example, can be aggregated to retrieve:
-
-- count
-- type
-- minimum
-- maximum
-- mean
-- median
-- mode
-- sum
-
-:::note Exercise
-Try out the above query again, with these changes.
-- Can you aggregate the results based on the data from the `answer` property?
-- Try modifying the query to group the results by `round`, and then aggregate top occurrences from `answer`.
-:::
-
-:::info
-Add summary
-:::
-
-import Quiz from '/src/components/Academy/quiz.js'
-const queryMethods = [{
- questionText: 'Which of these is not a valid way of running queries with Weaviate?',
- answerOptions: [
- {
- answerText: 'Sending a HTTP GET request.',
- isCorrect: true,
- feedback: 'In Weaviate, the REST interface is not used for queries.',
- },
- {
- answerText: 'Using the Weaviate Console.',
- isCorrect: false,
- feedback: 'You can use the Weaviate Console to directly enter GraphQL queries',
- },
- {
- answerText: 'With the Weaviate Python client.',
- isCorrect: false,
- feedback: 'In fact, you can send raw GraphQL queries or use native Python methods to perform queries with the Python client.',
- },
- ]
-}];
-const functionExplanations = [{
- questionText: 'Which of the following are correct?',
- answerOptions: [
- {
- answerText: 'You can use the Get function to retrieve summary information about a group of objects.',
- isCorrect: false,
- feedback: 'The Get function is used to retrieve individual objects.',
- },
- {
- answerText: 'The Aggregate function will return objects from Weaviate.',
- isCorrect: false,
- feedback: 'The Aggregate function will return summary, or aggregated, information about retrieved objects.',
- },
- {
- answerText: 'The Get function can retrieve objects from multiple classes in one query.',
- isCorrect: false,
- feedback: 'Each Get query can only search one class of objects.',
- },
- {
- answerText: 'None of the above.',
- isCorrect: true,
- feedback: 'All of the above are false!',
- },
- ]
-}];
\ No newline at end of file
diff --git a/docs/academy/py/zero_to_mvp/_109_tmp_leftovers/tmp.mdx b/docs/academy/py/zero_to_mvp/_109_tmp_leftovers/tmp.mdx
deleted file mode 100644
index 55e72ac65..000000000
--- a/docs/academy/py/zero_to_mvp/_109_tmp_leftovers/tmp.mdx
+++ /dev/null
@@ -1,77 +0,0 @@
-# Leftover information
-
-### `Get` with cross-references
-
-And before we wrap up this section, let's take a look at how to perform a `Get` query with cross-references. While we have not yet covered cross-references in detail in this unit, for now it is sufficient to know that they are a way to connect objects to each other.
-
-In the example below, we will perform a `Get` query on the `Question` class, but this time we will include the `hasCategory` property in the query. This property is a cross-reference to the `Category` class, and will be used to return the category of each question.
-
-```graphql
-{
- Get {
- Question (
- limit: 1
- nearText: {
- concepts: ["Seafaring technology from Scandanavia."],
- }
- ) {
- question
- answer
- hasCategory {
- ... on Category {
- title
- _additional {
- id
- }
- }
- }
- _additional {
- distance
- id
- }
- }
- }
-}
-```
-
-This produces a result like the following:
-
-Explain this query
- -The sub-properties under `round` in the query requests additional aggregations here, resulting in the top occurring answers being returned as well as their counts. - -
-
-
-Where the `hasCategory` property is now an array of objects, containing objects of type `Category` that the Question is related to and their requested properties. The `title` property of the `Category` class is returned as well, and the `_additional` field contains the `id` of the category.
-
-Cross-references provide you with a great deal of flexibility and power in your data model, and are a great way to connect objects to each other. We will cover them in more detail in our unit on schemas.
-
-:::info `...` pattern
-You may have noticed the `... on Category` pattern in the above query. This syntax is called a "fragment" in GraphQL terms. Populated this way, the fragment specifies that the `hasCategory` property is of type `Category`, and that the results should be returned as such.
-:::
diff --git a/docs/academy/py/zero_to_mvp/_999_vectorizer_selection_1/combining_models.md b/docs/academy/py/zero_to_mvp/_999_vectorizer_selection_1/combining_models.md
deleted file mode 100644
index d4b1d11c3..000000000
--- a/docs/academy/py/zero_to_mvp/_999_vectorizer_selection_1/combining_models.md
+++ /dev/null
@@ -1,12 +0,0 @@
----
-title: Combining models
-sidebar_position: 3
----
-
-Inspired by discussions here https://weaviate.slack.com/archives/C017EG2SL3H/p1677534406507339
-
-- How to combine models for things like Generative search + vectorizer
-
-- CLIP + Q&a
-
-etc etc
\ No newline at end of file
diff --git a/docs/academy/py/zero_to_mvp/_999_vectorizer_selection_1/common_model_types.md b/docs/academy/py/zero_to_mvp/_999_vectorizer_selection_1/common_model_types.md
deleted file mode 100644
index 3aa1c8070..000000000
--- a/docs/academy/py/zero_to_mvp/_999_vectorizer_selection_1/common_model_types.md
+++ /dev/null
@@ -1,100 +0,0 @@
----
-title: Common model types
-sidebar_position: 2
----
-
-## Types of vector representations
-
-:::warning TODO
-Intro video here
-:::
-
-When we talk about vectors, we are typically referring to vectors that are derived by machine-learning models. More specifically, we refer to vectors that are derived from neural networks, called "dense" vectors.
-
-However, there are other vector representations that are used to represent meaning, especially in relation to textual meaning. They include:
-
-- One-hot encoding
-- TF-IDF (term frequency-inverse document frequency) vectors
-- BM25 vectors
-
-Let's take a brief look at each one, as well as dense vectors.
-
-### One-hot encoding
-
-One-hot encodings represent text as a collection of 0s and 1s, where each 1 represents the presence of a word in the text. Sometimes this is also referred to as a "bag of words" representation.
-
-Accordingly, this representation ends up being very sparse, with most of the vector being 0s. This is because most words are not present in a given text.
-
-A limitation of this method is that it is not able to capture similarity of words, as each word is simply represented as being present or not present. Additionally, it is not able to take into account the relative importance of words in a text.
-
-### TF-IDF vectors
-
-A TF-IDF representation improves on the one-hot encoding by taking into account the relative importance of words in a text.
-
-TF-IDF stands for "term frequency-inverse document frequency". It is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus.
-
-The TF-IDF value increases proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the corpus. This means that common words across all documents such as "the", "a", "is" and "are" are penalized, whereas words that are rare across all documents are given more weight.
-
-Intuitively, this means that TF-IDF is able to capture the relative importance of words in a text by weighting rare words more heavily.
-
-### BM25 vectors
-
-BM25 vectors are similar to TF-IDF vectors, but they are able to take into account the length of a document. This is important because longer documents will have more words, and therefore a higher TF-IDF score, even if they are not more relevant than shorter documents.
-
-Accordingly, BM25 vectors normalize the TF-IDF score by the length of the document.
-
-### Dense Vectors
-
-#### Word vectors
-
-Word vectors are a type of vector representation that is derived from neural networks. They are able to capture the meaning of words by learning the context in which they appear.
-
-Models such as "word2vec" and "GloVe" popularized this type of representational learning. One key shortcoming of word-based vectors is that they are not able to take into account local context, such as the context of a word as a part of its parent sentence.
-
-This meant that where words need to be disambiguated, such as in the case of homonyms, word vectors were not able to capture the meaning of the word in the context of the sentence. (For example, the word "bank" can mean a financial institution or a river bank.)
-
-:::note Word vectors + weighting
-Word vectors in a text can be combined with a weighting method such as TF-IDF or BM25 to capture the relative importance of words in the text. The resulting vector can be used to represent the entire text.
-:::
-
-#### Transformer-derived vectors
-
-Most modern vector database use vectors that are derived from what are called "transformer" models.
-
-Transformers are a type of neural network that are able to take into account the context of its parent sentence in determining the meaning of each word. This means that they are able to disambiguate words that have multiple meanings, such as the word "bank" in the example above.
-
-Their current key shortcoming is their resource-intensive nature, especially as the input size (e.g. text length) increases.
-
-Query results
- -```json -{ - "data": { - "Get": { - "Question": [ - { - "_additional": { - "distance": 0.14927602, - "id": "5d9d5c8d-13e7-5791-b2f9-3724a1a1d301" - }, - "answer": "Oslo", - "hasCategory": [ - { - "_additional": { - "id": "fec50326-dfa1-53c9-90e8-63d0240bd933" - }, - "title": "MUSEUMS" - } - ], - "question": "1,200-year-old Viking ships are on display at the Vikingskiphuset in this Norwegian capital" - } - ] - } - } -} -``` - -- -We will discuss this a lot more in a later section related to AI model selection. - -::: - -Now, it may seem somewhat intuitive that a model can take a series of numbers describing features of a house, and predict its value (regression) or the probabilities of it belonging to a particular class. - -But how does this concept lead to generative AI models such as `gpt` models, or embedding models? It’s actually more similar than you might think. - -Let’s take a look at this in the next section. - -## Questions and feedback - -import DocsFeedback from '/_includes/docs-feedback.mdx'; - -
- -Many modern generative models are actually multi-modal. There are "vision language models" that can take not just text, but also images as parts of their inputs. And other types of generative models such as Stable Diffusion, or Sora can produce visual outputs, such as images or videos. -
- -These models fundamentally work similarly to what was described above, but with modality-specific aspects that allow them to go even further. - -::: - -## Questions and feedback - -import DocsFeedback from '/_includes/docs-feedback.mdx'; - -
- -Just like generative models, the world of embedding models is very big and interesting. Multi-modal embedding models can take various input types and produce compatible embeddings in the same space. And modern embeddings may come in different formats, using techniques such as multi-vector embeddings (e.g. ColBERT) or adaptive-length embeddings. -
- -We may touch on them later on, when we get to further in-depth discussions about specific modalities or model selection. - -::: - -## Questions and feedback - -import DocsFeedback from '/_includes/docs-feedback.mdx'; - -
- -The key here is to use “gradient descent”, to update parameters over time using numerical differentiation of the loss function. This technique calculates the direction in which parameters should change to reduce the model's error, and then makes small adjustments in that direction. -
- -But it is slightly outside the scope of our discussion here, and perhaps not as critical for those looking to simply use models. If you are interested in learning about model training in detail, there’s a lot of great resources out there. -
- -Here are a couple of our favorites: -- [Video: Let’s reproduce GPT-2 by Andrej Karpathy](https://www.youtube.com/watch?v=l8pRSuU81PU) -- [Fine-tune large language models: Hugging Face](https://huggingface.co/learn/nlp-course/en/chapter11/1) - -::: - - -import NNBackpropagation from './_img/nn_explained_55_backprop.png'; -import NNBackpropagationDark from './_img/nn_explained_55_backprop_dark.png'; - -
- -Weaviate is an open-source [vector database](https://weaviate.io/blog/what-is-a-vector-database). But what does that mean? Let's unpack it here. - -### Vector database - -Weaviate is a fantastic tool for retrieving the information you need, quickly and accurately. It does this by being an amazing **vector database**. - -You may be familiar with traditional databases such as relational databases that use SQL. A database can catalog, store and retrieve information. A **vector** database can carry out these tasks also, with the key difference being that they can perform these tasks based on similarity. - -#### How traditional searches work - -Imagine that you are searching a relational database containing articles on cities, to retrieve a list of "major" European cities. Using SQL, you might construct a query like this: - -```sql -SELECT city_name wiki_summary -FROM wiki_city -WHERE (wiki_summary LIKE '%major European city%' OR - wiki_summary LIKE '%important European city%' OR - wiki_summary LIKE '%prominent European city%' OR - wiki_summary LIKE '%leading European city%' OR - wiki_summary LIKE '%significant European city%' OR - wiki_summary LIKE '%top European city%' OR - wiki_summary LIKE '%influential European city%' OR - wiki_summary LIKE '%notable European city%') - (… and so on) -``` - -Which would return cities that contained any of these strings (`major`, `important`, `prominent`, ... etc) in the `wiki_summary` column. - -This works well in many circumstances. However, there are two significant limitations with this approach. - -#### Limitations of traditional search - -Using this type of search requires you to identify terms that *may* have been used to describe the concept, which is no easy feat. - -What's more, this doesn't solve the problem of how to rank the list of resulting objects. - -With the above search query, an entry merely containing a mention of a different European city (i.e. not very relevant) would be given equal weighting to an entry for Paris, or Rome, which would be highly relevant. - -A vector database makes this job simpler by enabling searches based on similarity. - -#### Examples of vector search - -Instead of searching for an exact match, you could perform a query to find objects that are "nearest" to "Major European city". - -What it would then return is a list of entries that are *ranked by their similarity* to the query. - -In other words, the results would reflect their similarity to the idea, or meaning, of "Major European city". - -What's more, Weaviate "indexes" the data based on their similarity, making this type of data retrieval lightning-fast. - -Weaviate can help you to do all this, and actually a lot more. Another way to think about Weaviate is that it supercharges the way you use information. - -:::info Vector vs semantic search -A vector search is also referred to as a "semantic search" because it returns results based on the similarity of meaning (therefore "semantic"). -::: - -### Open-source - -Weaviate is open-source. In other words, its [codebase is available online](https://github.com/weaviate/weaviate) for anyone to see and use[(1)](#-notes). - -And that is *the* codebase, regardless of how you use it. So whether you run Weaviate on your own computer, on a cloud computing environment, or through our managed service [Weaviate Cloud (WCD)](https://console.weaviate.cloud/), you are using the exact same technology. - -So, if you want, you can run Weaviate for free on your own device, or use our managed service for convenience. You can also take comfort in that you can see exactly what you are running, and be a part of the open-source community, as well as to shape its development. - -It also means that your knowledge about Weaviate is fungible, between local, cloud, or managed instances of Weaviate. So anything you learn here about Weaviate using WCD will be equally applicable to running it locally, and vice versa. 😉 - -### Information, made dynamic - -We are used to thinking of information as static, like a book. But with Weaviate and modern AI-driven language models, we can do much more than just retrieve static information but easily build on top of it. Take a look at these examples: - -#### Question answering - -Given a list of Wikipedia entries, you could ask Weaviate: - -:::note We asked Weaviate: -When was Lewis Hamilton born? -::: - -And it would answer with: - -:::note Weaviate responded: -Lewis Hamilton was born on January 7, 1985. ([check for yourself](https://en.wikipedia.org/wiki/Lewis_Hamilton)) -::: - -#### Generative search - -Or you can synthesize passages using retrieved information with Weaviate: - -Here is one, where we searched Weaviate for an entry on a "racing driver", and produce the result in the format of: - -:::note We asked Weaviate: -Write a fun tweet encouraging people to read about this: ## \{title} by summarizing highlights from: ## \{wiki_summary} -::: - -Which produces: - -:::note Weaviate responded: -Check out the amazing story of Lewis Hamilton, the 7-time Formula One World Drivers' Championship winner! From his humble beginnings to becoming one of the world's most influential people, his journey is an inspiring one. #LewisHamilton #FormulaOne #Motorsport #Racing -::: - -We will cover these and many more capabilities, such as vectorization, summarization and classification, in our units. - -For now, keep in mind that Weaviate is a vector database at its core which can also leverage AI tools to do more with the retrieved information. - -## Review - -In this section, you learned about what Weaviate is and how it works at a very high level. You have also been introduced to what vector search is at a high level, that it is a similarity-based search method. - -### Review exercises - -
- -We've covered that Weaviate is a [vector database](https://weaviate.io/blog/what-is-a-vector-database), and that a vector search is similarity-based. But, what is a vector? - -A vector in this context is just a series of numbers like `[1, 0]` or `[0.513, 0.155, 0.983, ..., 0.001, 0.932]`. Vectors like these are used to capture meaning. - -This might seem like an odd concept. But in fact, you may have already used vectors to capture meaning without realizing it. If you have tried photo editing, or used MS Paint you might have encountered the RGB color system. - -### How do numbers represent meaning? - -The RGB system uses groups of three numbers to represent colors. For example: - -- (255, 0, 0) = red -- (80, 200, 120) = emerald - -In these examples, each number can be thought of as a dial for how red, green or blue a color is. - -Now, imagine having hundreds, or even thousands, of these dials. That’s how vectors are used to represent meaning. Modern machine learning models such as GPT-x, or those used with Weaviate, use vectors to represent some "essence", or "meaning" of objects. This can be done for any object type, such as text, code, images, videos and more. - -## Vector embeddings in Weaviate - - - -The vector representation of an object's meaning is called a "vector embedding". - -Weaviate enables vector searches by indexing and storing data objects and their corresponding vector embeddings. The vector embeddings come from machine learning models. - -In plain terms, Weaviate processes and organizes your data in such a way that objects can be retrieved based on their similarity to a query. To perform these tasks at speed, Weaviate does two things that traditional databases do not. Weaviate: - -- Quantifies similarity -- Indexes vector data - -These operations enable Weaviate to do what it does. - -### Quantifying similarity - -As we've mentioned, vector searches are similarity-based, but what does that actually mean? How do we determine that two pieces of data are "similar"? What does it mean for two pieces of text, two images, or two objects in general, to be similar? - -This is a relatively simple idea that is actually incredibly interesting and intricate once we start to dive into the details. - -But for now, you should know that machine learning (ML) models are the key to this whole process. The ML models that power vector searches share similarities with those that generate text responses from prompts. Instead of generating new text, these (vectorizer) models capture the "meaning" of text or other media. We will cover this in more detail later on. - -### Indexing (vector) data - -Vector searches can be very intensive computationally. - -To overcome this problem, Weaviate uses a combination of indexes including an approximate nearest neighbor (ANN) index and an inverted index. The ANN index lets Weaviate perform extremely fast vector searches. The inverted index lets Weaviate filter data using Boolean criteria. - -We will get into this in more detail later - but for now, it's enough to know that Weaviate can perform fast vector searches as well as filtering. - -## Review - -In this section, you learned about what vectors are and how Weaviate utilizes them at a very high level. You have also been introduced to two of Weaviate's key capabilities that help it to enable vector search at speed. - -### Review exercise - -:::tip Self-reflection -Can you describe, in your own words, what vectors are? -::: - -
- -Let's take a look at a few more examples of what you can do with Weaviate. - -First, we will try vector searches by searching through our demo database. You will learn how to use Weaviate to retrieve objects based on their similarity, using various query types such as an input text, vector, or object. - -You will also compare vector search with keyword search to compare and contrast the two techniques, before learning how to combine the two techniques through the use of filters. - -### Vector search demo - -For our first example, let's search our demo dataset. It contains a small sample of questions from the quiz show *Jeopardy!*. - -Imagine that you're running a quiz night, and you want to get some questions about "animals in movies". In a traditional database you could look for word matches, perhaps something like: - -```sql -SELECT question, answer -FROM jeopardy_questions -WHERE ( - question LIKE '%animal%' - OR question LIKE '%creature%' - OR question LIKE '%beast%' - ) -AND ( - question LIKE '%movie%' - OR question LIKE '%film%' - OR question LIKE '%picture%' - OR question LIKE '%cinema%' - ) - -``` - -This is a difficult query to write. Even worse, you would probably have to add the names of specific animals to the query as well. - -The Weaviate query is much more intuitive. See what happens when we run the following query: - -:::note We searched Weaviate for: -animals in movies -::: - -Weaviate retrieved these as the top answers: - -:::note Weaviate retrieved: -- **meerkats**: Group of mammals seen here like Timon in *The Lion King* -- **dogs**: Scooby-Doo, Goofy & Pluto are cartoon versions -- **The Call of the Wild Thornberrys**: Jack London story about the dog Buck who joins a Nick cartoon about Eliza, who can talk to animals -::: - -Note how relevant the results are, despite none of them including the word "animal" or the word "movie", let alone both! - -This is exactly why vector searches are so useful. They can identify related objects without the need to match exact text. - -### Vector similarities - -In vector searches, you can retrieve the actual quantified similarity between the query and the returned objects. This is called the `distance`. - -A `distance` is indicative of the degree of similarity between the returned object and the query. - -If you're wondering exactly what that means, and who decides how similar any two objects or concepts are, those are great questions! We will cover those in more detail later. - -For now, just keep in mind that smaller distances mean two objects are more similar to each other. - -## Review - - - - - -### Key takeaways - -- Vector searches can identify related objects without the need for exact text matches. -- In vector searches, distance values indicate the degree of similarity between the returned object and the query. -- Smaller distances indicate greater similarity. -- Vector searches can be combined with keyword searches and filtering techniques for more refined search results. - -## Questions and feedback - -import DocsFeedback from '/_includes/docs-feedback.mdx'; - -
- -You can do a lot more with Weaviate than simply retrieve static information. - -Let's take a look at a couple of examples where we do more than simply retrieve objects from the database. - -We will extract information from this Wikipedia entry. - -
-
-
-Weaviate creates [data objects](/weaviate/concepts/data) when it processes the Wikipedia entry. The data objects are stored in classes. A class is roughly analogous to a table in a relational database. An object is similar to an entry in that table.
-
-
-### Generative search
-
-Weaviate can do even more with these entries. You can ask Weaviate to grab an object from its data store and use that object to generate new text. For example, Weaviate can use the object that contains the entry for the Sydney Opera House to derive new text.
-
-Here is a `GraphQL` query example of a generative search.
-
-```graphql
-{
- Get {
- WikiArticle(
- nearText: {
- concepts: ["Sydney Opera House"]
- }
- limit: 1
- ) {
- title
- wiki_summary
- _additional {
- generate(
- singleResult: {
- prompt: """
- Write a fun tweet encouraging people to read about this: ## {title}
- by summarizing highlights from: ## {wiki_summary}
- """
- }
- ) {
- singleResult
- error
- }
- }
- }
- }
-}
-```
-
-The sample code generates a Tweet based on the Wikipedia entry!
-
-"The Sydney Opera House" Wikipedia summary
- -The Sydney Opera House is a multi-venue performing arts centre in Sydney. Located on the foreshore of Sydney Harbour, it is widely regarded as one of the world's most famous and distinctive buildings and a masterpiece of 20th-century architecture. Designed by Danish architect Jørn Utzon, but completed by an Australian architectural team headed by Peter Hall, the building was formally opened by Queen Elizabeth II on 20 October 1973 after a gestation beginning with Utzon's 1957 selection as winner of an international design competition. The Government of New South Wales, led by the premier, Joseph Cahill, authorised work to begin in 1958 with Utzon directing construction. The government's decision to build Utzon's design is often overshadowed by circumstances that followed, including cost and scheduling overruns as well as the architect's ultimate resignation. The building and its surrounds occupy the whole of Bennelong Point on Sydney Harbour, between Sydney Cove and Farm Cove, adjacent to the Sydney central business district and the Royal Botanic Gardens, and near to the Sydney Harbour Bridge. - -The building comprises multiple performance venues, which together host well over 1,500 performances annually, attended by more than 1.2 million people. Performances are presented by numerous performing artists, including three resident companies: Opera Australia, the Sydney Theatre Company and the Sydney Symphony Orchestra. As one of the most popular visitor attractions in Australia, the site is visited by more than eight million people annually, and approximately 350,000 visitors take a guided tour of the building each year. The building is managed by the Sydney Opera House Trust, an agency of the New South Wales State Government. - -On 28 June 2007, the Sydney Opera House became a UNESCO World Heritage Site, having been listed on the (now defunct) Register of the National Estate since 1980, the National Trust of Australia register since 1983, the City of Sydney Heritage Inventory since 2000, the New South Wales State Heritage Register since 2003, and the Australian National Heritage List since 2005. The Opera House was also a finalist in the New7Wonders of the World campaign list. - -
-
-
-This process is an example of `generative search`. In a generative search, Weaviate retrieves information, and then leverages a large language model (LLM) to re-shape it. This is a powerful feature that can transform how you deal with information.
-
-You can vary the prompt to generate different results.
-
-### What next?
-
-Tools like Q&A and generative search really start to bring your information to life.
-
-## Review
-
-
-
-### Key takeaways
-
-- Weaviate can extract knowledge from text using question-answering capabilities, identifying the most relevant object and the actual answer based on the provided text.
-- Generative search allows you to retrieve information and reshape or repurpose the content, such as generating a tweet based on a Wikipedia entry.
-- These advanced capabilities of Weaviate transform how you interact with and utilize information in your data.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-See response
- -:::note Weaviate says: -Explore the world-famous Sydney Opera House and its incredible architecture! From the iconic design to the amazing performances, there's something for everyone to enjoy. #SydneyOperaHouse #Explore #Architecture #Performances #Experience -::: - -- - - -Welcome! - -This is the beginning of your journey through the world of vectors with Weaviate. This unit will provide you with an overview of the fundamentals of Weaviate. - -You'll first gain an understanding of what Weaviate is, and what it can do. You will then learn about what vector database and vector search are before going on to run Weaviate and perform vector searches yourself. - -By the end of this unit, you will have a strong foundation of knowledge that will help you to effectively navigate the rest of the course, and for using Weaviate in your own projects. - -### Prerequisites - -- None - -## Learning objectives - -import LearningGoalsExp from '/src/components/Academy/learningGoalsExp.mdx'; - -
.default})
-
-Now, let’s see what happens when we try to retrieve the best matching objects using two different embedding models. We will use the following two models:
-
-- `FastText (fasttext-en-vectors)` (from 2015; [model card](https://huggingface.co/facebook/fasttext-en-vectors))
-- `snowflake-arctic-embed-l-v2.0` (from 2024; [model card](https://huggingface.co/Snowflake/snowflake-arctic-embed-l-v2.0))
-
-Here is a summary of results from a search, using the `FastText` model from 2015:
-
-.default})
-
-The top result identified by the FastText is quite relevant, as it discusses how to correct some potential issues with cookie making. However, it’s less relevant than the idea result, which is a step-by-step recipe.
-
-The other two, however, are not relevant to the query. While they are recipes, they are not for baking cookies.
-
-It would be fair to say that there’s quite a bit of room for improvement.
-
-Here are the results from the `snowflake-arctic-embed-l-v2.0` model, from 2025:
-
-.default})
-
-We see that the `arctic` embeddings correctly identified the ideal top-ranked result. In fact, the top two expected results are included in the top three results for the `arctic` embeddings. Even the other result is relevant to chocolate chip cookies - although perhaps slightly off topic.
-
-### Evaluation criteria
-
-We could even compare these models using a standard metric, such as `nDCG@k`.
-
-For this scenarios, the two models scored:
-
-| Model | nDCG@10 |
-| --- | --- |
-| `FastText` | 0.595 |
-| `snowflake-arctic-embed-l-v2.0` | 0.908 |
-
-
-
-
-The size of embeddings produced is another key factor.
-
-Embeddings can vary greatly in size, from around 300 dimensions to thousands. Imagine a service provider that hosts an AI bot that answers questions about legal cases. A vector database with 1 million documents*, one embedding model (`nv-embed-v2`) could require as much memory as 3.3 TB of memory, while another (`embed-english-light-v3.0` ) might only require 300 GB of memory. (The following chart takes some popular models, and compares how each one would affect memory requirements.)
-
-What is nDCG@k?
- -`nDCG` is a metric used to evaluate the returned results in information retrieval. It rewards the model for returning the most relevant results at the top of the list. The `@k` indicates that only the top `k` results are considered. - -[Read more](https://weaviate.io/blog/retrieval-evaluation-metrics#normalized-discounted-cumulative-gain-ndcg) - -.default})
-
-These simple examples illustrate some of the impact of embedding model selection. The choice of embedding models can make a huge difference in the quality of your search, your resource requirements, and many more factors.
-
-There have been huge advancements in the landscape of embedding models over the last 10 to 15 years. In fact, innovations in embedding models continue to occur today. You might have heard of some of these names: word2vec, FastText, GloVe, BERT, CLIP, OpenAI ada, Cohere multi-lingual, Snowflake Arctic, ColBERT, and ColPali.
-
-Each model (or architecture) brings with it some improvements. It may be in model architecture, training data, training methodology, modality, or efficiency, for instance.
-
-So in the next few sections, let’s begin to explore a workflow for embedding model selection.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-.default})
-
-The chart shows a clear positive relationship between model size and higher performance. This also means that generally, models with better performance will be larger. They will require more memory and compute, which means higher costs and slower speeds.
-
-In other words, a larger model such as `nv-embed-v2` may perform better at retrieval than a smaller model such as `snowflake-arctic-embed-m-v1.5`, but may cost more to run and/or use.
-
-But there are many other dimensions to consider. For example:
-
-- A proprietary model such as a modern `gemini` model may show promising performance, but may not meet a user’s preference for local inference.
-- While a model may perform well at a standard benchmark, it may not perform as well if given material from a specialized domain, such as legal, medical, or coding tasks.
-- A local model may be cheaper to run, but the organization may lack the expertise and resources for long-term infrastructure maintenance.
-
-In the face of this complexity, a systematic approach can help you to make an informed decision based on your specific requirements. This is one such approach:
-
-.default})
-
-### Data Characteristics
-
-| Factor | Key Questions | Why It Matters |
-| --- | --- | --- |
-| **Modality** | Are you dealing with text, images, audio, or multimodal data? | Models are built for specific modality/modalities. |
-| **Language** | Which languages must be supported? | Models are trained & optimized for specific language(s), leading to trade-offs in performance. |
-| **Domain** | Is your data general or domain-specific (legal, medical, technical)? | Domain-specific models (e.g. [medical](https://huggingface.co/blog/abhinand/medembed-finetuned-embedding-models-for-medical-ir)) understand specialized vocabulary and concepts. |
-| **Length** | What's the typical length of your documents and queries? | Input token context windows vary between models, from as small as `256` tokens to `8192` tokens for example. However, longer context windows typically require exponentially higher compute and latency. |
-| **Asymmetry** | Will your queries differ significantly from your documents? | Some models are built for asymmetric query to document comparisons. So queries like `laptop won't turn on` can easily identify documents like `Troubleshooting Power Issues: If your device fails to boot...`. |
-
-### Performance Needs
-
-| Factor | Key Questions | Why It Matters |
-| --- | --- | --- |
-| **Accuracy** (recall) | How critical is it that all the top results are retrieved? | Higher accuracy requirements may justify more expensive or resource-intensive models. |
-| **Latency** | How quickly must queries be processed? | Larger models with better performance often have slower inference times. For inference services, faster services will cost more. |
-| **Throughput** | What query volume do you anticipate? Will there be traffic spikes? | Larger models with better performance often have lower processing capacity. For inference services, increased throughput will increase costs. |
-| **Volume** | How many documents will you process? | Larger embedding dimensions increase memory requirements for your vector store. This will impact resource requirements and affect costs at scale. |
-| **Task type** | Is retrieval the only use case? Or will it also involve others (e.g. clustering or classification) ? | Models have strengths and weaknesses; a model excellent at retrieval might not excel at clustering. This will drive your evaluation & selection criteria. |
-
-### Operational Factors
-
-| Factor | Key Questions | Why It Matters |
-| --- | --- | --- |
-| **Hardware limitations** | What computational resources are available for hosting & inference? | Hardware availability (costs, GPU/TPU availability) will significantly affect your range of choices. |
-| **API rate limits** | If using a hosted model, what are the provider's limits? | Rate limits can bottleneck applications, or limit potential growth. |
-| **Deployment & maintenance** | What technical expertise and resources are required? | Is self-hosting a model an option, or should you look at API-based hosted options? |
-
-### Business Requirements
-
-| Factor | Key Questions | Why It Matters |
-| --- | --- | --- |
-| **Hosting options** | Do you need self-hosting capabilities, or is a cloud API acceptable? | Self-hosting ➡️ more control at higher operational complexity; APIs ➡️ lower friction at higher dependencies. |
-| **Licensing** | What are the licensing restrictions for commercial applications? | Some model licenses or restrictions may prohibit certain use cases. |
-| **Long-term support** | What guarantees exist for the model's continued availability? | If a model or business is abandoned, downstream applications may need significant reworking. |
-| **Budget** | What are your cost limits and expenditure preferences? | Embedding costs can add up over time, but self-hosting can incur high upfront costs. |
-| **Privacy & Compliance** | Are there data privacy requirements or industry regulations to consider? | Some industries require specific models. And privacy requirements may impose hosting requirements. |
-
-Documenting these requirements creates a clear profile of your ideal embedding model, which will guide your selection process and help you make informed trade-offs.
-
-## Compile candidate models
-
-After identifying your needs, create a list of potential embedding models to evaluate. This process helps focus your detailed evaluation on the most promising candidates.
-
-There are hundreds of embedding models available today, with new ones being released regularly. For this many models, even a simple screening process would be too time-consuming.
-
-As a result, we suggest identifying an initial list of models with a simple set of heuristics, such as these:
-
-### Account for model modality
-
-This is a critical, first-step filter. A model can only support the modality/modalities that it is designed and trained for.
-
-Some models (e.g. Cohere `embed-english-v3.0`) are multimodal, while others (e.g. Snowflake’s `snowflake-arctic-embed-l-v2.0`) are unimodal.
-
-No matter how good a model is, a text-only model such as `snowflake-arctic-embed-l-v2.0` will not be able to perform image retrieval. Similarly, a `ColQwen` model cannot be used for plain text retrieval.
-
-### Favor models already available
-
-If your organization already uses embedding models for other applications, these are great starting points. They are likely to have been screened, evaluated and approved for use, and accounts/billing already configured. For local models, this would mean that the infrastructure is already available.
-
-This also extends to models available through your other service providers.
-
-You may be already using generative AI models through providers such as Cohere, Mistral or OpenAI. Or, perhaps your hyperscaler partners such as AWS, Microsoft Azure or Google Cloud provide embedding models.
-
-In many cases, these providers will also provide access to embedding models, which would be easier to adopt than those from a new organization.
-
-### Try well-known models
-
-Generally, well-known or popular models are popular for a reason.
-
-Industry leaders in AI such as Alibaba, Cohere, Google, NVIDIA and OpenAI all produce embedding models for different modalities, languages and sizes. Here are a few samples of their available model families:
-
-| Provider | Model families |
-| --- | --- |
-| Alibaba | `gte`, `Qwen` |
-| Cohere | `embed-english`, `embed-multilingual` |
-| Google | `gemini-embedding`, `text-embedding` |
-| NVIDIA | `NV-embed` |
-| OpenAI | `text-embedding`, `ada` |
-
-There are also other families of models that you can consider.
-
-For example, the `ColPali` family of models for image embeddings and `CLIP` / `SigLIP` family of models for multimodal (image and text) are well-known in their respective domains. Then, `nomic`, `snowflake-arctic`, `MiniLM` and `bge` models are some examples of well-known language retrieval models.
-
-These popular models tend to be well-documented, discussed and widely supported.
-
-As a result, they tend to be easier than the more obscure models to use, evaluate, troubleshoot and use.
-
-### Benchmark leaders
-
-Models that perform well on standard benchmarks may be worth considering. Resources like [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) can help identify high-performing models.
-
-As an example, the screenshot below shows models on MTEB at a size of fewer than 1 billion parameters, sorted by their `retrieval` performance.
-
-.default})
-
-It shows some models that we’ve already discussed - such as the `showflake-arctic`, Alibaba’s `gte`, or BAAI’s `bge` models.
-
-But additionally, you can see already a number of high-performing models that we hadn’t discussed. Microsoft research's `intfloat/multilingual-e5-large-instruct` or JinaAI’s `jinaai/jina-embeddings-v3` model are both easily discoverable here.
-
-Note that as of 2025, the MTEB contains different benchmarks to assess different capabilities, such as the linguistic or modality needs.
-
-When viewing benchmarks, make sure to view the right set of benchmarks, and the appropriate columns. In the example below, note that the page shows results for MIEB (image retrieval), with results sorted by *Any to Any Retrieval*.
-
-.default})
-
-The MTEB is filterable and sortable by various metrics. So, you can arrange it to suit your preferences and add models to your list as you see fit.
-
-You should be able to compile a manageable list of models relatively quickly using these techniques. This list can then be manually reviewed for detailed screening.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-.default})
-
-Some of the readily screenable factors, and how to screen models are shown below:
-
-## Screening factors
-
-### Context length
-
-Input context length is a critical factor to ensure that meaning from the whole document chunks taken into account. Maximum input context lengths vary widely between models, as shown in these examples:
-
-- `all-MiniLM-L6-v2`: 256 tokens
-- Cohere `embed-english-v3.0`: 512 tokens
-- `snowflake-arctic-embed-l-v2.0`: 8192 tokens
-
-Input text exceeding the context length will be ignored. On the other hand, higher allowable context lengths typically require exponentially higher compute and latency. As a result, this is an important tradeoff that includes an interplay with your text chunking strategy.
-
-:::tip
-
-Consider what a “chunk” of information to retrieve looks like for your use case. Typically, a model with 512 tokens or higher is sufficient for most use cases.
-
-:::
-
-### Model goals & training methodology
-
-Different embedding models are optimized for different use cases. This informs the model architecture, training data and training methodology.
-
-Reviewing the model provider’s descriptions and published training details can provide key insights into its suitability for your use case.
-
-- **Linguistic capabilities**: Some models (e.g. Snowflake’s `snowflake-arctic-embed-l-v2.0`) are multi-lingual, while others are primarily uni-lingual (e.g. Cohere’s `embed-english-v3.0`). These linguistic capabilities come largely from the training data and methodology selection.
-- **Domain exposure**: Models trained on specialized domains (e.g., legal, medical, financial) typically perform better for domain-specific applications.
-- **Primary tasks**: The provider may have been building a general-purpose embedding model, or one that is particularly focussed on particular tasks. Google’s `gemini-embedding` model appears to be designed with a goal of being a jack-of-all-trades type, state of the art model in all tasks and domains ([release blog](https://developers.googleblog.com/en/gemini-embedding-text-model-now-available-gemini-api/)). On the other hand, Snowflake’s `arctic-embed` 2.0 models appear to be focussed on retrieval tasks ([release blog](https://www.snowflake.com/en/engineering-blog/snowflake-arctic-embed-2-multilingual/)).
-- **Base model**: In many cases, an embedding model is trained from an existing model. Any advantages, or shortcomings, of the base model will often carry over to the final model, especially if it is an architectural one such as its context window size or pooling strategy.
-- **Training methods (advanced)**: If you have more experience with model training techniques, this is an area that you can use as heuristics as well. For example, models trained with contrastive learning often perform better for retrieval tasks. Additionally, hard negative mining is a technique that is valuable to enhance contrastive learning.
-
-:::tip
-
-Select a model whose capabilities align with your goals. For example, if your application requires retrieving paragraphs of text chunks in English, French, German, Mandarin Chinese and Japanese, check the model card and training information. Look for its retrieval performance, and whether these languages were included in the training corpus.
-
-:::
-
-### Dimensionality and optimization options
-
-The dimensionality of embeddings affects both performance and resource requirements.
-
-As a rule of thumb, your memory requirements for a vector database (any quantization notwithstanding) may be: `4 bytes` * `n dimensions` * `m objects` * `1.5` where `m` is the size of your database, and `n` is the vector dimensionality (`1.5` to account for overhead).
-
-This means that for, say, 10 million objects, the memory requirements for given models’ full outputs will be:
-
-- NVIDIA `NV-embed-v2`: `246 GB`
-- OpenAI `text-embedding-3-large`: `184 GB`
-- `snowflake-arctic-embed-l-v2.0`: `61 GB`
-- `all-MiniLM-L6-v2`: `23 GB`
-
-As you might imagine, this can add significant costs to your infrastructure needs for the vector database.
-
-At the database end, there are quantization strategies which will reduce the footprint and therefore costs, which we will cover in another course.
-
-However, certain models can also help in this regard as well. [Matryoshka Representation Learning (MRL)](https://weaviate.io/blog/openais-matryoshka-embeddings-in-weaviate) models like `jina-embeddings-v2` or `snowflake-arctic-embed-l-v2.0` allow for flexible dimensionality reduction by simply truncating the vector. In the case of `snowflake-arctic-embed-l-v2.0`, it can be truncated to `256` dimensions from its original `1024` dimensions, reducing its size to a quarter without much loss in performance.
-
-:::tip
-
-Consider how big your dataset is likely to get to, then select your model accordingly, keeping the resulting system requirements in mind. If the requirements are too high and thus out-of-budget, it may set you back to square one when you need to scale up and go to production.
-
-:::
-
-### Model size and inference speed
-
-Model size directly impacts inference speed, which is critical for applications with latency requirements. Larger models generally offer better performance but at the cost of increased computational demands.
-
-When screening models, consider these aspects:
-
-| Factor | Implications |
-| --- | --- |
-| Parameter count | More parameters typically mean better quality but slower inference and higher memory usage |
-| Architecture efficiency | Some models are optimized for faster inference despite their size |
-| Hardware requirements | Larger models may require specialized hardware (GPUs/TPUs) |
-
-:::tip
-
-Given that the inference speed is a function of the model, inference hardware as well as the network latencies, review these factors as a system when screening models’ suitability.
-
-:::
-
-### Pricing, availability, and licensing
-
-The practical aspects of model adoption extend beyond technical considerations.
-
-Providers offer various pricing structures:
-
-- **API-based pricing**: Pay-per-token (OpenAI, Cohere)
-- **Compute-based pricing**: Based on hardware utilization (Cloud providers)
-- **Tiered licensing**: Different capabilities at different price points
-- **Open-source**: Free to use, but self-hosting costs apply
-
-Choice of model and inference type will affect model availability:
-
-- **Geographic availability**: Some providers don't operate in all regions
-- **SLA guarantees**: Uptime commitments and support levels
-- **Rate limiting**: Constraints on throughput that may affect your application
-- **Version stability**: How frequently models are deprecated or updated
-
-Additionally, licensing terms vary significantly:
-
-- **Commercial use restrictions**: Some open models prohibit commercial applications
-- **Data usage policies**: How your data may be used by the provider
-- **Export restrictions**: Compliance with regional regulations
-- **Deployment flexibility**: Whether the model can be deployed on-premises or edge devices
-
-Always review the specific terms for each model. For example, while models like CLIP are openly available, they may have usage restrictions that affect your application.
-
-:::tip
-
-These practical considerations can sometimes outweigh performance benefits. A slightly less accurate model with favorable licensing terms and lower costs might be preferable for many production applications.
-
-:::
-
-### Creating your candidate shortlist
-
-After considering these factors, you can create a prioritized shortlist of models to evaluate in more detail. A good approach is to include a mix of:
-
-1. **Benchmark leaders**: High-performing models on standard metrics
-2. **Resource-efficient options**: Models with smaller footprints or faster inference
-3. **Specialized models**: Models that might be particularly well-suited to your domain
-4. **Different architectures**: Including diverse approaches increases the chance of finding a good fit
-
-Aim for 3-5 models in your initial shortlist for detailed evaluation. Including too many models can make the evaluation process unwieldy and time-consuming.
-
-In the next section, we'll explore how to perform detailed evaluations of these candidate models to determine which one best meets your specific requirements.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-.default})
-
-The next image includes tasks that uses more specialized domain data. These benchmarks span a range of areas, including legal, medical, programming, and government data.
-
-.default})
-
-The chart shows that certain embedding models at the top of the table, such as `gemini-embedding-exp-03-07` perform quite well across the board compared to the others. But this doesn’t tell the whole story, as a given model often outperforms its average score in particular tasks.
-
-For example, the `snowflake-arctic-embed` models perform very well with the `LEMBPasskeyRetrieval` task, which is designed to test recall of specific text buried in a longer embedding. And Cohere’s `Cohere-embed-multilingual-v3.0` performs quite well in the MIRACL task, which is a highly multilingual task.
-
-In fact, it is interesting to note that even though we are looking at MTEB’s multilingual task set, it includes includes a number of tasks with an English-only (or majority) corpus.
-
-.default})
-
-So, you may benefit from deriving your own metric that blends these task scores, based on how well each specific task corresponds to your needs.
-
-You might consider:
-
-1. **Task relevance**: Does the task match your use case?
-2. **Data distribution**: Does the data represent your domain?
-3. **Metric relevance**: Are the reported metrics aligned with your requirements?
-4. **Recency**: Are the results recent enough to reflect current model capabilities?
-5. **Fairness**: Were all models evaluated under comparable conditions?
-
-For example, if you know that your data definitely will include a blend of languages, you may weight the multi-lingual datasets more heavily than the mono-lingual datasets. And similarly with domain-specific data, such as legal, medical, programming and so on.
-
-The resulting score may be different from the official blended number, but may be more relevant to your particular use case.
-
-### Limitations of standard benchmarks
-
-These third-party benchmarks are very useful, but there are a few limitations that you should keep in mind. The main two limitations are data leakage, and correlation to your needs.
-
-**Data leakage**
-
-Because these benchmarks are publicly available, there is a risk that some of the benchmark data ends up in the training data used to build models. This can happen for a number of reasons, especially because there is simply so much data being used in the training process.
-
-This means that the benchmark result may not be a fair representation of the model’s capabilities, as the model is “remembering” the training data.
-
-**Correlation to your needs**
-
-Another limitation is that the standard benchmarks don’t accurately reflect your needs. As you saw, we can aim to find a benchmark that is as close as possible to your actual use case. But it is unlikely that the task, data distribution and metrics are fully aligned with your needs.
-
-**Mitigation**
-
-As a result, it is important to take these standard benchmarks with a grain of salt. And in terms of getting further signals, a good complementary exercise is to perform your own benchmarks, which we will look at in the next section.
-
-## Model evaluation: custom benchmarks
-
-
-While standard benchmarks provide valuable reference points, creating your own custom evaluation can be a fantastic complementary tool to address their limitations.
-
-Running your own benchmark can sound quite intimidating, especially given how extensive benchmarks such as MTEB are. But it doesn’t need to be. You can do this by following these steps:
-
-### Set benchmark objectives
-
-By now, you should have an idea of any gaps in your knowledge set, as well as your given tasks. It might be something like:
-
-- Which model best retrieves the appropriate related customer reviews about coffee, written primarily in English, French, and Korean?
-- Does any model work well across code retrieval in Python and Golang for back-end web code chunks, as well as related documentation snippets?
-
-The custom benchmark should be designed with an idea of addressing particular questions.
-
-### Determine metrics to use
-
-Once the overall goals are defined, the corresponding metrics can follow.
-
-For example, retrieval performance is commonly measured by one or more of precision, recall, MAP, MRR, and NDCG.
-
-Each of these measure slightly different aspects of retrieval performance. However, using NDCG is a good starting point.
-
-NDCG measures the system's ability to correctly sort items based on relevance. Given a query and a dataset ranked for this query, NDCG will reward results for having the higher ranked items higher in the search results.
-
-It is measured on a score of 0 to 1, where 0 means no ranked items were retrieved, and 1 means all top ranked items were retrieved, and ordered correctly.
-
-### Curate a benchmark dataset
-
-A suitable dataset is critical for a benchmark to be meaningful. While such a dataset may already exist, it is common to build or reshape a dataset to suit the benchmark goals and metrics.
-
-The dataset should aim to:
-
-- Reflect the retrieval task
-- Reflect the task difficulty
-- Capture the data distribution
-- Include sufficient volume
-
-This may be the most time consuming part of the process. However, a pragmatic approach can help to make it manageable. A benchmark with as few as 20 objects and a handful of queries can produce meaningful results.
-
-### Run benchmark
-
-At this point, run the benchmark using your candidate models.
-
-As with many other scientific projects, reproducibility and consistency is key here. It is also worth keeping in mind that you may come back to this later on to assess new models, or with updated knowledge about your needs.
-
-In programming terms, you might modularize aspects, such as embedding creation, dataset loading, metric evaluation, and result presentation.
-
-### Evaluate the results
-
-Once the benchmarks are run, it is important to assess the results using quantitative (e.g. NDCG@k numbers) and qualitative (e.g. which objects were retrieved where) means.
-
-The quantitative results will produce a definitive ranking that you can use, for example to order the models. However, this is subject to many factors, such as dataset composition and metric being used.
-
-The qualitative results may provide more important insights, such as patterns of failure. For example, may see an embedding model:
-
-- Regularly fail to retrieve certain types of objects, such as shorter, but very relevant text, favoring longer ones
-- Perform better with positively phrased text but not ones with negation in the sentence.
-- Struggle with your domain-specific jargon.
-- Work well with English and Mandarin Chinese, but not so well with Hungarian, which may be a key language for your data.
-
-To some extent, these insights may be only discoverable to those with domain familiarity, or those with context on the system being built. Accordingly, qualitative assessment is critically important.
-
-## Questions and feedback
-
-import DocsFeedback from '/_includes/docs-feedback.mdx';
-
-- return
+const regularItems = Object.fromEntries( +Object.entries(secondaryNavbarItems).filter(([, value]) => !value.isSmall) +); + +{" "} +
+return
})()} ## What is Weaviate? @@ -64,6 +77,12 @@ Weaviate is an **open-source vector database** designed to store and index both - **[Agent-driven workflows](../agents/index.md)** Its flexible API and integration with modern AI models make Weaviate suitable for powering applications that rely on intelligent agents. These agents can leverage semantic insights to make decisions or trigger actions based on the data stored in Weaviate. +import AcademyAdmonition from '@site/src/components/AcademyAdmonition'; + +
+
+
## Questions and feedback
import DocsFeedback from '/_includes/docs-feedback.mdx';
diff --git a/docs/weaviate/quickstart/index.md b/docs/weaviate/quickstart/index.md
index 9dcba7d30..b4237344d 100644
--- a/docs/weaviate/quickstart/index.md
+++ b/docs/weaviate/quickstart/index.md
@@ -483,7 +483,7 @@ Try these additional resources to learn more about Weaviate:
Answer
+ +Weaviate Academy is a full-fledged learning platform available at [academy.weaviate.io](https://academy.weaviate.io). + +:::note + +If you need resources from the previous version of Weaviate Academy, check out the [documentation archive](https://archive.docs.weaviate.io/academy) + +::: + +- Check out the Starter guide: retrieval augmented generation, and the Weaviate Academy unit on chunking. + Check out the Starter guide: retrieval augmented generation.
- Check out the Starter guide: retrieval augmented generation, and the Weaviate Academy unit on chunking. + Check out the Starter guide: retrieval augmented generation.
-
- )
-};
-
-export default AcademyCard;
\ No newline at end of file
diff --git a/src/components/Academy/courseData.js b/src/components/Academy/courseData.js
deleted file mode 100644
index 102790212..000000000
--- a/src/components/Academy/courseData.js
+++ /dev/null
@@ -1,366 +0,0 @@
-export const courseData = {
- "starter_text_data": {
- title: "Text data with Weaviate",
- courseId: "PY_101T",
- body: "Project-based learning where you'll learn how to build with Weaviate and any text data. Weaviate generates the vectors for you.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_text_data",
- badgeType: "course",
- isCourse: true,
- units: [
- "text_setup_weaviate", "text_collections", "text_searches", "text_rag"
- ],
- learningGoals: [
- "How to create a Weaviate instance, add data to it to enable semantic searching, and use AI through retrieval augmented generation."
- ],
- learningOutcomes: [
- "Create an instance of Weaviate for you to use",
- "Produce, store and index semantic (vector) data from source text",
- "Perform semantic, keyword and hybrid searches",
- "Use AI (large language models) to augment and transform retrieved data",
- ],
- note: "Python client (v4); project-based"
- },
- "starter_custom_vectors": {
- title: "Your own vectors with Weaviate",
- courseId: "PY_101V",
- body: "Project-based learning where you'll learn how to build with Weaviate and your own data and vectors. This version is for those who prefer to use your own vectors built outside of Weaviate.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_custom_vectors",
- badgeType: "course",
- isCourse: true,
- units: [
- "byov_setup_weaviate", "byov_collections", "byov_searches", "byov_rag"
- ],
- learningGoals: [
- "How to create a cloud Weaviate instance, add data to it to enable semantic searching, and use AI through retrieval augmented generation."
- ],
- learningOutcomes: [
- "Create a instance of Weaviate for you to use",
- "Produce, store and index data with your own vectors",
- "Perform vector, keyword and hybrid searches",
- "Use AI (large language models) to augment and transform retrieved data",
- ],
- note: "Python client (v4); project-based"
- },
- "starter_multimodal": {
- title: "Multimodal data with Weaviate",
- courseId: "PY_101M",
- body: "Project-based learning where you'll learn how to build with Weaviate and multi-modal data. Weaviate generates the vectors for you.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_multimodal_data",
- badgeType: "course",
- isCourse: true,
- units: [
- "docker_mm_basics", "mm_collections", "mm_searches", "mm_rag"
- ],
- learningGoals: [
- "How to create a local Weaviate instance, add data to it to enable multi-modal searching, and use AI through retrieval augmented generation."
- ],
- learningOutcomes: [
- "Create a local instance of Weaviate with a multimodal vectorizer module",
- "Produce, store and index multimodal data",
- "Perform multimodal searches",
- "Use AI (large language models) to augment and transform retrieved data",
- ],
- note: "Python client (v4); project-based"
- },
- "named_vectors": {
- title: "Flexible data representation: Named vectors",
- courseId: "PY_220",
- body: "Learn how named vectors can provide a flexible way to represent your data in Weaviate.",
- buttonType: "Click here",
- buttonURL: "/academy/py/named_vectors",
- badgeType: "course",
- isCourse: true,
- units: [
-
- ],
- learningGoals: [
- "What named vectors can be used for, and how to add them to your data collection."
- ],
- learningOutcomes: [
- "Describe use cases for named vectors",
- "Create a collection with named vectors",
- "Add objects with multiple vector embeddings per object",
- "Perform searches on named vectors",
- ],
- note: "Python client (v4); project-based"
- },
- "vector_index": {
- title: "Vector indexes",
- courseId: "PY_230",
- body: "Vector indexes are the key components for vector search. Learn what they are, and how to use them effectively to suit your needs.",
- buttonType: "Click here",
- buttonURL: "/academy/py/vector_index",
- badgeType: "course",
- isCourse: true,
- units: [
- "vindex_overview", "vindex_hnsw", "vindex_flat", "vindex_dynamic"
- ],
- learningGoals: [
- "What vector index types are available, when to select each one and how to configure them."
- ],
- learningOutcomes: [
- "Name available vector index types in Weaviate.",
- "Select an appropriate index type for a given use case.",
- "Recite relationships between HNSW parameters and search performance.",
- "Describe how quantization affects each index type.",
- "Create collections with your chosen vector index type and preferred parameters.",
- ],
- note: "Python client (v4)"
- },
- "compression": {
- title: "Vector compression for improved efficiency",
- courseId: "PY_250",
- body: "Vectors can be compressed to reduce memory requirements or improve retrieval speeds. Find out how to get the most out of this feature.",
- buttonType: "Click here",
- buttonURL: "/academy/py/compression",
- badgeType: "course",
- isCourse: true,
- units: [
- "compression_pq", "compression_bq", "compression_strategy"
- ],
- learningGoals: [
- "What vector compression algorithms are available, how to use them and when to use them."
- ],
- learningOutcomes: [
- "Name available vector compression algorithms in Weaviate.",
- "Create collections with vector compression enabled.",
- "Configure vector compression parameters.",
- "Select a compression algorithm for a given use case.",
- ],
- note: "Python client (v4)"
- },
- "tokenization": {
- title: "Text tokenization",
- courseId: "PY_275",
- body: "What happens when text is indexed, and searched, or converted into a vector? They are 'tokenized'. Learn what this is, and how you can make it work for you.",
- buttonType: "Click here",
- buttonURL: "/academy/py/tokenization",
- badgeType: "course",
- isCourse: true,
- units: [
- "tokenization_basics", "tokenization_options", "tokenization_filters", "tokenization_searches"
- ],
- learningGoals: [
- "What tokenization is, and why it is required."
- ],
- learningOutcomes: [
- "Identify tokenized text from raw text.",
- "Name different tokenization options in Weaviate.",
- "Select an appropriate tokenization option for a given use case.",
- "Name languages for which specific tokenization options are available.",
- ],
- note: "Python client (v4)"
- },
- "setup_weaviate_typescript": {
- title: "Set up TypeScript (or Javascript) for Weaviate",
- courseId: "TS_100",
- body: "A quick run through of how to set up and install the Weaviate TypeScript client.",
- buttonType: "Click here",
- buttonURL: "/academy/js/set_up_typescript",
- badgeType: "course",
- isCourse: true,
- units: [
- "setup_weaviate_typescript"
- ],
- learningGoals: [
- "Setup Weaviate to get started building TypeScript (or JavaScript) apps."
- ],
- learningOutcomes: [
- "Install Node.js",
- "(Optionally) Install and set up TypeScript",
- "Install the Weaviate client",
- ],
- note: "TS clients"
- },
- "starter_multimodal_typescript": {
- title: "Multimodal data with Weaviate",
- courseId: "TS_101M",
- body: "Project-based learning where you'll learn how to build with Weaviate and multi-modal data. Weaviate generates the vectors for you.",
- buttonType: "Click here",
- buttonURL: "/academy/js/starter_multimodal_data",
- badgeType: "course",
- isCourse: true,
- units: [
- "docker_mm_basics_ts", "mm_collections_ts", "mm_searches_ts", "mm_rag_ts"
- ],
- learningGoals: [
- "How to create a Weaviate instance, add data to it to enable multi-modal searching, and use AI through retrieval augmented generation."
- ],
- learningOutcomes: [
- "Create an instance of Weaviate with a multimodal vectorizer module",
- "Produce, store and index multimodal data",
- "Perform multimodal searches",
- "Use AI (large language models) to augment and transform retrieved data",
- ],
- note: "TS clients; project-based"
- },
- "multi-tenancy": {
- title: "Multi-tenancy",
- courseId: "PY_280",
- body: "Learn how to implement and manage multi-tenancy in Weaviate for efficient data isolation and resource management.",
- buttonType: "Click here",
- buttonURL: "/academy/py/multitenancy",
- badgeType: "course",
- isCourse: true,
- units: [
- "mt_overview", "mt_setup", "mt_tenant_data", "mt_manage_tenants"
- ],
- learningGoals: [
- "Understand multi-tenancy concepts and their application in Weaviate",
- "Learn how to set up and manage multi-tenant collections for scalable applications",
- "Master techniques for efficient resource management in multi-tenant environments"
- ],
- learningOutcomes: [
- "Explain the concept of multi-tenancy and its benefits in Weaviate",
- "Set up a Weaviate instance and configure collections for multi-tenancy",
- "Create, manage, and remove tenants in a multi-tenant collection",
- "Perform data operations and queries specific to individual tenants",
- "Implement efficient resource management using tenant activity statuses",
- "Utilize advanced features like auto-tenant creation, activation, and offloading",
- "Apply multi-tenancy concepts to real-world scenarios for improved scalability and data isolation"
- ],
- note: "Python client (v4)"
- },
- "starter_text_data_typescript": {
- title: "Text data with Weaviate",
- courseId: "TS_101T",
- body: "Project-based learning where you'll learn how to build with Weaviate and any text data. Weaviate generates the vectors for you.",
- buttonType: "Click here",
- buttonURL: "/academy/js/starter_text_data",
- badgeType: "course",
- isCourse: true,
- units: [
- "text_setup_weaviate_ts", "text_collections_ts", "text_searches_ts", "text_rag_ts"
- ],
- learningGoals: [
- "How to create a Weaviate instance, add data to it to enable semantic searching, and use AI through retrieval augmented generation."
- ],
- learningOutcomes: [
- "Create an instance of Weaviate for you to use",
- "Produce, store and index semantic (vector) data from source text",
- "Perform semantic, keyword and hybrid searches",
- "Use AI (large language models) to augment and transform retrieved data",
- ],
- note: "TS clients; project-based"
- },
- // "building_with_weaviate": {
- // title: "Additional topics",
- // courseId: "PY_200",
- // body: "Expand on the `Getting to MVP` course for deeper dives into key topics vectorizer selection, multi-modal models, and best practices.",
- // buttonType: "Click here",
- // buttonURL: "/academy/py/building_with_weaviate",
- // badgeType: "course",
- // isCourse: true,
- // units: [
- // "which_search",
- // "schema_design",
- // "vectorizer_selection",
- // "indexing",
- // ],
- // learningGoals: [
- // "In-depth material and best practices to help you build with Weaviate, such as vectorization options, which searches to perform and how to work with your indexes."
- // ],
- // learningOutcomes: [
- // "Select a suitable vectorizer for your given goals and situation.",
- // "Understand practical differences between search methods and suggest a suitable technique for a given situation.",
- // "Compare types of indexes used by Weaviate, and modify parameters to balance speed and recall.",
- // ]
- // },
- // "configuring_weaviate_instance": {
- // title: "Customization using modules",
- // courseId: "2",
- // body: "",
- // buttonType: "Notify",
- // badgeType: "course",
- // isCourse: true,
- // units: [
- // "t2v_under_hood", "vectorizer_selection_2", "custom_models", "module_building"
- // ],
- // learningGoals: [
- // "TBC"
- // ],
- // learningOutcomes: [
- // "TBC"
- // ]
- // },
- // "to_production": {
- // title: "Getting to Production",
- // courseId: "3",
- // body: "Speed to production with authentication & authorization, backups, monitoring and replication.",
- // buttonType: "Notify",
- // badgeType: "course",
- // isCourse: true,
- // units: [
- // "backups", "auth", "scaling", "replication", "migration", "kubernetes"
- // ],
- // learningGoals: [
- // "TBC"
- // ],
- // learningOutcomes: [
- // "TBC"
- // ]
- // },
- "kubernetes_intro": {
- title: "Run Weaviate on Kubernetes",
- courseId: "D100",
- body: "Learn how to run Weaviate on a local kubernetes cluster with Minikube.",
- buttonType: "Click here",
- buttonURL: "/academy/deployment/k8s",
- badgeType: "course",
- isCourse: false,
- units: [
- "kubernetes_intro"
- ]
- },
- "standalone": {
- title: "Standalone units",
- courseId: "0",
- body: "Bite-sized, standalone units that can be reviewed by themselves.",
- buttonType: "Notify",
- badgeType: "course",
- isCourse: false,
- units: [
- "which_search",
- "chunking"
- ]
- },
- "standalone_js": {
- title: "Standalone units (JS/TS)",
- courseId: "0",
- body: "Bite-sized, standalone units that can be reviewed by themselves.",
- buttonType: "Notify",
- badgeType: "course",
- isCourse: false,
- units: [
- "which_search",
- "client_server",
- "using_ml_models",
- ]
- },
- "zero_to_mvp": {
- title: "Zero to MVP: The basics",
- courseId: "P3_1",
- body: "Start here: Get started with all the core knowledge and essential skills for building with Weaviate. Learn how to build a Weaviate database and effectively perform queries to find the right data.",
- buttonType: "Click here",
- buttonURL: "/academy/py/zero_to_mvp",
- badgeType: "course",
- isCourse: true,
- units: [
- "hello_weaviate", "queries_1", "schema_and_import", "queries_2"
- ],
- learningGoals: [
- "How to build a Weaviate instance and populate it with vectorized data, as well as how to construct queries to efficiently retrieve relevant data."
- ],
- learningOutcomes: [
- "Use Weaviate Cloud to create an instance of Weaviate",
- "Use appropriate query types and syntax to retrieve desired objects",
- "Outline what vector search is and how it works",
- "Demonstrate how to efficiently populate a Weaviate instance with data",
- "Differentiate BM25 and hybrid search techniques from vector search techniques",
- ],
- note: "Python client (v3)"
- }
-};
diff --git a/src/components/Academy/courseUnits.jsx b/src/components/Academy/courseUnits.jsx
deleted file mode 100644
index 40366705d..000000000
--- a/src/components/Academy/courseUnits.jsx
+++ /dev/null
@@ -1,29 +0,0 @@
-import React from "react";
-import UnitCardSet from "./unitcards";
-import { unitData } from '/src/components/Academy/unitData.js'
-import './academy.css'
-
-function Units(props) {
-
- let cardData = props.courseData;
- let courseName = props.courseName;
- let cardItems = [];
-
- for (let k in cardData) {
- if (k == courseName || courseName == null) {
- let units = cardData[k].units.map(d => unitData[d])
-
- cardItems.push(
-
-
- {props.title}
-
- {badgeTxt} {props.note != null? {props.note}: null }
-
-
- - {props.body} -
-
-
- )
- }
- };
-
- return (
-
-
- {cardItems}
)
-}
-
-export default Units;
\ No newline at end of file
diff --git a/src/components/Academy/coursecards.jsx b/src/components/Academy/coursecards.jsx
deleted file mode 100644
index 8b2803f88..000000000
--- a/src/components/Academy/coursecards.jsx
+++ /dev/null
@@ -1,30 +0,0 @@
-import React from 'react';
-import AcademyCard from './card';
-import './academy.css'
-
-function CourseCardSet(props) {
-
- let cardData = props.cardData;
- let cardItems = [];
-
- for (let k in cardData) {
- if (cardData[k].isCourse) {
- cardItems.push(
-
-
- )
- }
- };
-
- return ({cardItems}
)
-}
-
-export default CourseCardSet;
diff --git a/src/components/Academy/courses.jsx b/src/components/Academy/courses.jsx
deleted file mode 100644
index 698fc775e..000000000
--- a/src/components/Academy/courses.jsx
+++ /dev/null
@@ -1,14 +0,0 @@
-import React from 'react';
-import CourseCardSet from './coursecards';
-import './academy.css'
-
-function Courses(props) {
-
- return (
-
-
-
- )
-};
-
-export default Courses;
\ No newline at end of file
diff --git a/src/components/Academy/learningGoals.jsx b/src/components/Academy/learningGoals.jsx
deleted file mode 100644
index 95e733bdc..000000000
--- a/src/components/Academy/learningGoals.jsx
+++ /dev/null
@@ -1,53 +0,0 @@
-import React from 'react';
-import { unitData } from '/src/components/Academy/unitData.js';
-import { courseData } from '/src/components/Academy/courseData.js';
-import './academy.css'
-
-function LearningGoals(props) {
-
- let data = ""
- if (props.unitName !== undefined) {
- data = unitData[props.unitName]
- } else if (props.courseName !== undefined) {
- data = courseData[props.courseName]
- }
-
- let goalsArray = [];
- let outcomesArray = [];
-
- for (let i = 0; i < data.learningGoals.length; i++) {
- goalsArray.push(
-
-
- )
-};
-
-export default LearningGoals;
diff --git a/src/components/Academy/learningGoalsExp.mdx b/src/components/Academy/learningGoalsExp.mdx
deleted file mode 100644
index 1fec4e800..000000000
--- a/src/components/Academy/learningGoalsExp.mdx
+++ /dev/null
@@ -1,8 +0,0 @@
-
-
-
-
-
-
- Here, we will cover:
- Learning Goals -- {goalsArray}
-
-
- By the time you are finished, you will be able to:
- Learning Outcomes -- {outcomesArray}
-
diff --git a/src/components/Academy/quiz.js b/src/components/Academy/quiz.js
deleted file mode 100644
index e3bb4d8c4..000000000
--- a/src/components/Academy/quiz.js
+++ /dev/null
@@ -1,70 +0,0 @@
-import React, { useState } from 'react';
-import './academy.css'
-
-export default function Quiz(props) {
-
- const questions = props.questions
-
- const [currentQuestion, setCurrentQuestion] = useState(0);
- const [showFeedback, setShowFeedback] = useState(false);
- const [feedback, setFeedback] = useState("");
- const [score, setScore] = useState(0);
-
- const handleAnswerOptionClick = (isCorrect, feedback) => {
- if (isCorrect) {
- setScore(score + 1);
- setFeedback("That's right!\n\n" + feedback);
- } else {
- setFeedback("That's not right.\n\n" + feedback);
- }
-
- const nextQuestion = currentQuestion + 1;
- if (nextQuestion < questions.length) {
- setCurrentQuestion(nextQuestion);
- } else {
- setShowFeedback(true);
- }
- };
-
- const handleRetryClick = () => {
- setScore(0)
- setShowFeedback(false);
- };
-
- let btnClass = "quiz-button"
- if (props.isCode == true) {
- btnClass = "quiz-button code_answer"
- }
-
- return (
- What are these?
- Each unit includes a set of Learning Goals and Learning Outcomes which form the unit's guiding principles. --
-
- Learning Goals describe the unit's key topics and ideas. -
- Learning Outcomes on the other hand describe tangible skills that the learner should be able to demonstrate -
- {showFeedback ? (
-
- );
-}
\ No newline at end of file
diff --git a/src/components/Academy/unitData.js b/src/components/Academy/unitData.js
deleted file mode 100644
index 7bdf10b85..000000000
--- a/src/components/Academy/unitData.js
+++ /dev/null
@@ -1,1189 +0,0 @@
-export const unitData = {
- "text_setup_weaviate": {
- title: "Set up Weaviate",
- body: "Set up a Weaviate instance and connect to it.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_text_data/setup_weaviate",
- badgeType: "practical",
- learningGoals: [
- "Set up a Weaviate instance and connect to it with the Python client."
- ],
- learningOutcomes: [
- "Install the latest Weaviate Python client.",
- "Create a Weaviate instance.",
- "Connect to the Weaviate instance using the Python client and communicate with it."
- ]
- },
- "text_setup_weaviate_ts": {
- title: "Set up Weaviate",
- body: "Set up a Weaviate instance and connect to it.",
- buttonType: "Click here",
- buttonURL: "/academy/js/starter_text_data/setup_weaviate",
- badgeType: "practical",
- learningGoals: [
- "Set up a Weaviate instance and connect to it with the JavaScript/Typescript client."
- ],
- learningOutcomes: [
- "Install the latest Weaviate JavaScript/Typescript client.",
- "Create a Weaviate instance.",
- "Connect to the Weaviate instance using the JavaScript/Typescript client and communicate with it."
- ]
- },
- "text_collections": {
- title: "Populate the database",
- body: "Create a collection and import data, and have Weaviate create vectors for you.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_text_data/text_collections",
- badgeType: "practical",
- learningGoals: [
- "Configure and create a collection then import text data using batch imports."
- ],
- learningOutcomes: [
- "Configure a collection with typical settings and vectorizer set.",
- "Create a collection and work with a collection object.",
- "Import data using batch imports."
- ]
- },
- "text_collections_ts": {
- title: "Populate the database",
- body: "Create a collection and import data, and have Weaviate create vectors for you.",
- buttonType: "Click here",
- buttonURL: "/academy/js/starter_text_data/text_collections",
- badgeType: "practical",
- learningGoals: [
- "Configure and create a collection then import text data using batch imports."
- ],
- learningOutcomes: [
- "Configure a collection with typical settings and vectorizer set.",
- "Create a collection and work with a collection object.",
- "Import data using batch imports."
- ]
- },
- "text_searches": {
- title: "Perform searches",
- body: "Learn how to use search functions in Weaviate.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_text_data/text_searches",
- badgeType: "practical",
- learningGoals: [
- "Perform semantic, keyword and hybrid searches."
- ],
- learningOutcomes: [
- "Describe differences between semantic, keyword and hybrid searches at a high level.",
- "Perform a semantic search with near text functions.",
- "Perform a keyword search.",
- "Perform a hybrid search.",
- ]
- },
- "text_searches_ts": {
- title: "Perform searches",
- body: "Learn how to use search functions in Weaviate.",
- buttonType: "Click here",
- buttonURL: "/academy/js/starter_text_data/text_searches",
- badgeType: "practical",
- learningGoals: [
- "Perform semantic, keyword and hybrid searches."
- ],
- learningOutcomes: [
- "Describe differences between semantic, keyword and hybrid searches at a high level.",
- "Perform a semantic search with near text functions.",
- "Perform a keyword search.",
- "Perform a hybrid search.",
- ]
- },
- "text_rag": {
- title: "LLMs and Weaviate (RAG)",
- body: "Use large language models to augment and transform retrieved data.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_text_data/text_rag",
- badgeType: "practical",
- learningGoals: [
- "Use AI (large language models) to augment and transform retrieved data."
- ],
- learningOutcomes: [
- "Describe what RAG is and how it works at a high level.",
- "Perform RAG on individual objects.",
- "Perform RAG on the entire set of returned objects.",
- ]
- },
- "text_rag_ts": {
- title: "LLMs and Weaviate (RAG)",
- body: "Use large language models to augment and transform retrieved data.",
- buttonType: "Click here",
- buttonURL: "/academy/js/starter_text_data/text_rag",
- badgeType: "practical",
- learningGoals: [
- "Use AI (large language models) to augment and transform retrieved data."
- ],
- learningOutcomes: [
- "Describe what RAG is and how it works at a high level.",
- "Perform RAG on individual objects.",
- "Perform RAG on the entire set of returned objects.",
- ]
- },
- "byov_setup_weaviate": {
- title: "Set up Weaviate",
- body: "Set up a Weaviate instance and connect to it.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_custom_vectors/setup_weaviate",
- badgeType: "practical",
- learningGoals: [
- "Set up a Weaviate instance and connect to it with the Python client."
- ],
- learningOutcomes: [
- "Install the latest Weaviate Python client.",
- "Create a Weaviate instance.",
- "Connect to the Weaviate instance using the Python client and communicate with it."
- ]
- },
- "byov_collections": {
- title: "Populate the database",
- body: "Create a collection and import data with your own vectors.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_custom_vectors/object_collections",
- badgeType: "practical",
- learningGoals: [
- "Configure and create a collection then import text data with your own vectors using batch imports."
- ],
- learningOutcomes: [
- "Configure a collection with typical settings and vectorizer set as none.",
- "Create a collection and work with a collection object.",
- "Import data using batch imports."
- ]
- },
- "byov_searches": {
- title: "Perform searches",
- body: "Learn how to use search functions in Weaviate.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_custom_vectors/object_searches",
- badgeType: "practical",
- learningGoals: [
- "Perform semantic, keyword and hybrid searches."
- ],
- learningOutcomes: [
- "Describe differences between vector, keyword and hybrid searches at a high level.",
- "Perform a vector search with near vector functions.",
- "Perform a keyword search.",
- "Perform a hybrid search.",
- ]
- },
- "byov_rag": {
- title: "LLMs and Weaviate (RAG)",
- body: "Use large language models to augment and transform retrieved data.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_custom_vectors/object_rag",
- badgeType: "practical",
- learningGoals: [
- "Use AI (large language models) to augment and transform retrieved data."
- ],
- learningOutcomes: [
- "Describe what RAG is and how it works at a high level.",
- "Perform RAG on individual objects.",
- "Perform RAG on the entire set of returned objects.",
- ]
- },
- "docker_mm_basics": {
- title: "Weaviate for multimodal data",
- body: "Create a local Weaviate instance for multimodal data.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_multimodal_data/setup_weaviate",
- badgeType: "practical",
- learningGoals: [
- "Set up a local Weaviate instance with a multimodal module."
- ],
- learningOutcomes: [
- "Create a local Weaviate instance with a multimodal module using Docker.",
- "Install the latest Weaviate Python client.",
- "Connect to the Weaviate instance using the Python client."
- ]
- },
- "docker_mm_basics_ts": {
- title: "Weaviate for multimodal data",
- body: "Create a Weaviate instance for multimodal data.",
- buttonType: "Click here",
- buttonURL: "/academy/js/starter_multimodal_data/setup_weaviate",
- badgeType: "practical",
- learningGoals: [
- "Set up a Weaviate instance with a multimodal module."
- ],
- learningOutcomes: [
- "Create a Weaviate instance with a multimodal module using Docker.",
- "Install the latest Weaviate TypeScript client.",
- "Connect to the Weaviate instance using the Typescript client."
- ]
- },
- "mm_collections": {
- title: "Populate the database",
- body: "Create a collection and import multimodal data, and have Weaviate create vectors for you.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_multimodal_data/mm_collections",
- badgeType: "practical",
- learningGoals: [
- "Configure and create a collection then import multimodal data using batch imports."
- ],
- learningOutcomes: [
- "Configure a collection with typical settings and vectorizer set.",
- "Create a collection and work with a collection object.",
- "Import data using batch imports."
- ]
- },
- "mm_collections_ts": {
- title: "Populate the database",
- body: "Create a collection and import multimodal data, and have Weaviate create vectors for you.",
- buttonType: "Click here",
- buttonURL: "/academy/js/starter_multimodal_data/mm_collections",
- badgeType: "practical",
- learningGoals: [
- "Configure and create a collection then import multimodal data using batch imports."
- ],
- learningOutcomes: [
- "Configure a collection with typical settings and vectorizer set.",
- "Create a collection and work with a collection object.",
- "Import data using batch imports."
- ]
- },
- "mm_searches": {
- title: "Perform searches",
- body: "Learn how to use search functions in Weaviate.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_multimodal_data/mm_searches",
- badgeType: "practical",
- learningGoals: [
- "Perform multimodal searches."
- ],
- learningOutcomes: [
- "Describe how multimodal search works at a high level.",
- "Perform searches through multimodal data in different ways.",
- ]
- },
- "mm_searches_ts": {
- title: "Perform searches",
- body: "Learn how to use search functions in Weaviate.",
- buttonType: "Click here",
- buttonURL: "/academy/js/starter_multimodal_data/mm_searches",
- badgeType: "practical",
- learningGoals: [
- "Perform multimodal searches."
- ],
- learningOutcomes: [
- "Describe how multimodal search works at a high level.",
- "Perform searches through multimodal data in different ways.",
- ]
- },
- "mm_rag": {
- title: "LLMs and Weaviate (RAG)",
- body: "Use large language models to augment and transform retrieved data.",
- buttonType: "Click here",
- buttonURL: "/academy/py/starter_multimodal_data/mm_rag",
- badgeType: "practical",
- learningGoals: [
- "Use AI (large language models) to augment and transform retrieved data."
- ],
- learningOutcomes: [
- "Describe what RAG is and how it works at a high level.",
- "Perform RAG on individual objects.",
- "Perform RAG on the entire set of returned objects.",
- ]
- },
- "mm_rag_ts": {
- title: "LLMs and Weaviate (RAG)",
- body: "Use large language models to augment and transform retrieved data.",
- buttonType: "Click here",
- buttonURL: "/academy/js/starter_multimodal_data/mm_rag",
- badgeType: "practical",
- learningGoals: [
- "Use AI (large language models) to augment and transform retrieved data."
- ],
- learningOutcomes: [
- "Describe what RAG is and how it works at a high level.",
- "Perform RAG on individual objects.",
- "Perform RAG on the entire set of returned objects.",
- ]
- },
- "nv_collections": {
- title: "Populate the database",
- body: "Create a collection with multiple named vectors and import data objects.",
- buttonType: "Click here",
- buttonURL: "/academy/py/named_vectors/nv_collections",
- badgeType: "practical",
- learningGoals: [
- "Configure and create a collection with multiple named vectors then import data using batch imports."
- ],
- learningOutcomes: [
- "Configure a collection with named vectors.",
- "Import data using batch imports to create objects with multiple named vectors."
- ]
- },
- "nv_queries": {
- title: "Perform queries",
- body: "Learn how to use query functions in Weaviate with named vectors.",
- buttonType: "Click here",
- buttonURL: "/academy/py/named_vectors/nv_queries",
- badgeType: "practical",
- learningGoals: [
- "How queries change for collections with named vectors."
- ],
- learningOutcomes: [
- "Describe what query types are affected by named vectors.",
- "Perform vector/similarity and hybrid searches in collections with named vectors.",
- "Perform RAG on collections with named vectors.",
- ]
- },
- "compression_pq": {
- title: "Product quantization",
- body: "What is product quantization (PQ), and how do you use it?",
- buttonType: "Click here",
- buttonURL: "/academy/py/compression/pq",
- badgeType: "practical",
- learningGoals: [
- "Learn what PQ is and how to use it."
- ],
- learningOutcomes: [
- "Describe how PQ works and its key parameters.",
- "Configure a collection with PQ.",
- ]
- },
- "compression_bq": {
- title: "Binary quantization",
- body: "What is binary quantization (BQ), and how do you use it?",
- buttonType: "Click here",
- buttonURL: "/academy/py/compression/bq",
- badgeType: "practical",
- learningGoals: [
- "Learn what BQ is and how to use it."
- ],
- learningOutcomes: [
- "Describe how BQ works and its key parameters.",
- "Configure a collection with BQ.",
- ]
- },
- "compression_strategy": {
- title: "Compression strategy",
- body: "What compression algorithm and settings are right for me?",
- buttonType: "Click here",
- buttonURL: "/academy/py/compression/strategy",
- badgeType: "theory",
- learningGoals: [
- "Learn what compression algorithm might work best for your use case."
- ],
- learningOutcomes: [
- "Describe the pros and cons of using PQ, BQ, or uncompressed vectors?",
- "Make an informed choice on which compression algorithm to use for a given use case.",
- ]
- },
- "vindex_overview": {
- title: "Vector index: Overview",
- body: "What is a vector index, and why is it important?",
- buttonType: "Click here",
- buttonURL: "/academy/py/vector_index/overview",
- badgeType: "practical",
- learningGoals: [
- "Learn what a vector index is and how it affects your vector searches."
- ],
- learningOutcomes: [
- "Describe what a vector index is and what it does.",
- "Recite different types of vector indexes in Weaviate.",
- ]
- },
- "vindex_hnsw": {
- title: "HNSW index in depth",
- body: "Learn about the HNSW index type, and how to tune it for your use case.",
- buttonType: "Click here",
- buttonURL: "/academy/py/vector_index/hnsw",
- badgeType: "practical",
- learningGoals: [
- "How the HNSW index works and how to tune it for your use case."
- ],
- learningOutcomes: [
- "Describe how an HNSW index works and its key attributes.",
- "Configure Weaviate with an HNSW index, and describe its parameters.",
- ]
- },
- "vindex_flat": {
- title: "Flat index in depth",
- body: "Learn about the flat index type, and how to tune it for your use case.",
- buttonType: "Click here",
- buttonURL: "/academy/py/vector_index/flat",
- badgeType: "practical",
- learningGoals: [
- "How the flat index works and how to tune it for your use case."
- ],
- learningOutcomes: [
- "Describe how a flat index works and its key attributes.",
- "Configure Weaviate with a flat index, and describe its parameters.",
- ]
- },
- "vindex_dynamic": {
- title: "Dynamic index in depth",
- body: "Learn about the dynamic index type, and how to tune it for your use case.",
- buttonType: "Click here",
- buttonURL: "/academy/py/vector_index/dynamic",
- badgeType: "practical",
- learningGoals: [
- "How the dynamic index works and how to tune it for your use case."
- ],
- learningOutcomes: [
- "Describe how a dynamic index works and its key attributes.",
- "Configure Weaviate with a dynamic index, and describe its parameters.",
- ]
- },
- "tokenization_basics": {
- title: "Overview of tokenization",
- body: "What is tokenization, and why is it important?",
- buttonType: "Click here",
- buttonURL: "/academy/py/tokenization/basics",
- badgeType: "theory",
- learningGoals: [
- "What tokenization is and its impact."
- ],
- learningOutcomes: [
- "Describe what tokenization is.",
- "Differentiate the tokenizer's role in vectorization and keyword-based operations.",
- ]
- },
- "tokenization_options": {
- title: "Available tokenization options",
- body: "What tokenization options are available in Weaviate?",
- buttonType: "Click here",
- buttonURL: "/academy/py/tokenization/options",
- badgeType: "theory",
- learningGoals: [
- "Review the available tokenization options in Weaviate."
- ],
- learningOutcomes: [
- "Describe different tokenization options available in Weaviate.",
- "Identify the best tokenization option for a given use case.",
- "Identify languages for which specific tokenization options are available.",
- ]
- },
- "tokenization_filters": {
- title: "Tokenization and filters",
- body: "See how tokenization impacts filters.",
- buttonType: "Click here",
- buttonURL: "/academy/py/tokenization/filters",
- badgeType: "practical",
- learningGoals: [
- "The impact of tokenization on filters."
- ],
- learningOutcomes: [
- "Describe the ways in which tokenization can affect filters.",
- "Implement an appropriate tokenization option in Weaviate for a given use case.",
- ]
- },
- "tokenization_searches": {
- title: "Tokenization and searches",
- body: "See how tokenization impacts searches.",
- buttonType: "Click here",
- buttonURL: "/academy/py/tokenization/searches",
- badgeType: "practical",
- learningGoals: [
- "The impact of tokenization on query results."
- ],
- learningOutcomes: [
- "Describe the ways in which tokenization can affect keyword search results.",
- "Implement an appropriate tokenization option in Weaviate for a given use case.",
- ]
- },
- mt_overview: {
- "title": "An overview of multi-tenancy",
- "body": "Learn what multi-tenancy is and how it helps to build scalable production applications.",
- "buttonType": "Click here",
- "buttonURL": "/academy/py/multitenancy/overview",
- "badgeType": "theory",
- "learningGoals": [
- "Understand multi-tenancy in Weaviate and its benefits",
- "Identify use cases for multi-tenancy in production applications"
- ],
- "learningOutcomes": [
- "Describe what multi-tenancy is in the context of Weaviate",
- "Explain the benefits of multi-tenancy for resource management and scalability",
- "Identify scenarios where multi-tenancy is advantageous",
- "Understand the concept of tenant isolation within a shared collection"
- ],
- owner: "jp",
- reviewer: "jp"
- },
- mt_setup: {
- "title": "Multi-tenant setup",
- "body": "Set up a Weaviate instance and configure collections for multi-tenant usage.",
- "buttonType": "Click here",
- "buttonURL": "/academy/py/multitenancy/setup",
- "badgeType": "practical",
- "learningGoals": [
- "Learn how to set up a Weaviate instance and configure collections for multi-tenant usage."
- ],
- "learningOutcomes": [
- "Configure a Weaviate instance for multi-tenancy",
- "Create a multi-tenant collection with appropriate settings",
- "Understand and configure auto-tenant creation and activation",
- "Set up dynamic indexing for efficient resource usage across tenants"
- ],
- owner: "jp",
- reviewer: "jp"
- },
- mt_tenant_data: {
- "title": "Work with tenants and data",
- "body": "Learn how to manage tenants and their data in a multi-tenant collection.",
- "buttonType": "Click here",
- "buttonURL": "/academy/py/multitenancy/tenant_data",
- "badgeType": "practical",
- "learningGoals": [
- "Understand how to create and manage tenants as well their data in a multi-tenant environment"
- ],
- "learningOutcomes": [
- "Create single and multiple tenants",
- "Insert and batch import data for specific tenants",
- "Perform queries on tenant-specific data",
- "Understand and utilize auto-tenant creation"
- ],
- "owner": "jp",
- "reviewer": "jp"
- },
- mt_manage_tenants: {
- "title": "Manage tenant states and resources",
- "body": "Learn how to efficiently manage tenant states and optimize resource usage.",
- "buttonType": "Click here",
- "buttonURL": "/academy/py/multitenancy/manage_tenants",
- "badgeType": "practical",
- "learningGoals": [
- "Understand tenant activity statuses and their implications, and learn how to manage tenant states for optimal resource usage"
- ],
- "learningOutcomes": [
- "Describe the different tenant activity statuses (ACTIVE, INACTIVE, OFFLOADED)",
- "Update tenant states individually and in bulk",
- "Implement tenant offloading to cold storage",
- "Utilize auto-activation for efficient resource management",
- "Remove tenants and their associated data"
- ],
- "owner": "jp",
- "reviewer": "jp"
- },
- "ai_models_deep_dive": {
- title: "AI models: A gentle deep dive",
- body: "A practical introduction to AI models for software engineers or AI builders.",
- buttonType: "Click here",
- buttonURL: "/academy/theory/ai_models_deep_dive",
- badgeType: "theory",
- learningGoals: [
- "What AI models are and what they do",
- "Overview of model architecture & training",
- "Common types of models (generative and embedding)",
- "Examples of models and use cases",
- "How to access AI models (with code example)",
- ],
- learningOutcomes: [
- "Explain the core concepts of AI model architecture and training",
- "Describe multiple model types and their appropriate use cases",
- "Implement basic code for accessing models",
- "Interpret key details of model cards or specifications",
- ],
- owner: "jp",
- reviewer: "jp"
- },
- "embedding_model_selection": {
- title: "Embedding model evaluation & selection",
- body: "A practical guide to choosing embedding models for your use case.",
- buttonType: "Click here",
- buttonURL: "/academy/theory/embedding_model_selection",
- badgeType: "theory",
- learningGoals: [
- "The practical impact of embedding model selection on AI application performance",
- "A systematic, evidence-based approach to embedding model selection",
- "Skills to evaluate, implement, and maintain embedding models in production systems"
- ],
- learningOutcomes: [
- "Evaluate embedding models based on quality, performance, cost, and resource requirements",
- "Apply selection framework to identify, screen, and select embedding models",
- "Design and implement effective evaluation strategies",
- "Articulate key monitoring and maintenance needs for embedding models",
- "Optimize embedding model selection for domain-specific applications"
- ],
- owner: "jp",
- reviewer: "jp"
- },
- hello_weaviate: {
- title: "Hello, Weaviate",
- body: "Start here: Learn what Weaviate is, and about its key capabilities and features, as well as about vectors that power Weaviate.",
- buttonType: "Click here",
- buttonURL: "/academy/py/zero_to_mvp/hello_weaviate",
- badgeType: "mixed",
- learningGoals: [
- "What Weaviate is, and what it does.",
- "How to create your own Weaviate instance on WCD.",
- "Weaviate clients and how to install them.",
- "Hands-on experience with Weaviate.",
- ],
- learningOutcomes: [
- "Broadly describe what Weaviate is.",
- "Outline what vector search is.",
- "Create a Weaviate instance on WCD.",
- "Install your preferred Weaviate client.",
- "Describe some of Weaviate's capabilities.",
- ],
- owner: "jp",
- reviewer: "jp"
- },
- queries_1: {
- title: "Queries 1",
- body: "Learn how queries work in Weaviate, how to use similarity searches and use filters, as well as how search works under the hood.",
- buttonType: "Click here",
- badgeType: "practical",
- buttonURL: "/academy/py/zero_to_mvp/queries_1",
- learningGoals: [
- "How to retrieve objects and properties.",
- "The structure of returned responses from Weaviate.",
- "The difference between `nearVector`, `nearObject` and `nearText`.",
- "How to aggregate meta information about objects.",
- "How to add filters to vector searches.",
- "Weaviate's internal vector search process.",
- ],
- learningOutcomes: [
- "Construct 'Get' queries to retrieve relevant objects and desired properties.",
- "Parse a response from Weaviate.",
- "Explain the differences between `nearVector`, `nearObject` and `nearText`.",
- "Construct 'Aggregate' queries to retrieve aggregated properties about relevant objects.",
- "Add filters to queries to exclude certain results.",
- "Describe how `nearObject` and `nearText` queries are converted to vector searches, and what pre-filtering is.",
- ],
- owner: "jp",
- reviewer: "jp"
- },
- schema_and_import: { // Separate the more difficult topics into their own units (schema 2 / import 2)
- title: "Schema and imports",
- body: "Learn what role the schema plays in Weaviate, and how to define it, before learning how to effectively populate Weaviate with data.",
- buttonType: "Click here",
- badgeType: "mixed",
- buttonURL: "/academy/py/zero_to_mvp/schema_and_imports",
- learningGoals: [
- "How Weaviate organizes and stores data.",
- "An overview of indexes used in Weaviate.",
- "What a schema is, and how to define it.",
- "How to define classes and properties, including appropriate data types.",
- "How to populate Weaviate with data.",
- "Some best practices such as batch imports and additional properties.",
- ],
- learningOutcomes: [
- "Describe how the schema relates to organization and storage of data in Weaviate.",
- "Broadly describe the role of indexes in Weaviate.",
- "Understand how classes and properties represent your data.",
- "Create a schema to suit your data.",
- "Populate Weaviate with data, using batch imports.",
- ],
- owner: "jp",
- reviewer: "jp"
- },
- queries_2: {
- title: "Queries 2",
- body: "Learn about even more query types, from hybrid searches that combine keyword and vector searches to generative searches that transform your data at retrieval.",
- buttonType: "Click here",
- badgeType: "practical",
- buttonURL: "/academy/py/zero_to_mvp/queries_2",
- learningGoals: [
- "How to formulate and perform keyword or BM25 searches.",
- "What Hybrid searches are, how they are ranked and how to use them.",
- "How Generative searches utilize language models to transform data before delivery.",
- "How you can extract the exact answers from data with the Question & Answer (QnA) module.",
- ],
- learningOutcomes: [
- "Perform BM25 and hybrid searches.",
- "Differentiate between vector, BM25 and hybrid searches.",
- "Transform data before delivery with generative searches.",
- "Extract answers from data with QnA searches.",
- ],
- owner: "jp",
- reviewer: "jp"
- },
-
- which_search: {
- title: "Which search is right for me?",
- body: "Weaviate offers many search types (vector, keyword and hybrid), and options. Let's discuss some good, baseline search strategies.",
- buttonType: "Click here",
- badgeType: "mixed",
- buttonURL: "/academy/py/standalone/which_search",
- learningGoals: [
- "Impact of search type on search quality.",
- "Impact of search type on search performance.",
- "How the dataset and chunking affect search",
- "Key considerations for selecting a search type.",
- "Strategies to apply to improve search quality.",
- ],
- learningOutcomes: [
- "Broadly recite pros and cons of each search type (vector, keyword and hybrid).",
- "Suggest a suitable search type given a description of the dataset and aim.",
- "Suggest alternative or additional search strategies to improve search quality.",
- "Outline broad methods to evaluate search quality."
- ],
- owner: "jp",
- reviewer: "jp"
- },
- schema_design: {
- title: "Collection schema design",
- body: "How to design your collection data structure. Whether to use classes or multi-tenancy, or cross-references.",
- buttonType: "TBD",
- badgeType: "theory",
- // buttonURL: "/academy/py/building_with_weaviate/schema_design",
- learningGoals: [
- "Starting suggestions for selecting appropriate data structures to have Weaviate work for your needs.",
- ],
- learningOutcomes: [
- "Outline what collection schema is for.",
- "Describe when to use single or multi-tenancy collections.",
- "Explain the impact of cross-references on data, and queries.",
- "Make informed choices on whether to use cross-references.",
- ],
- owner: "jp",
- reviewer: "jp"
- },
- vectorizer_selection: {
- title: "Vectorizer selection",
- body: "The basics on how to select a good baseline vectorizer for given data and task types.",
- buttonType: "TBD",
- badgeType: "theory",
- // buttonURL: "/academy/py/building_with_weaviate/vectorizer_selection",
- learningGoals: [
- "Theory and heuristics for selecting appropriate, robust vectorizers for the data type and task at hand and how to set the vectorizer appropriately in Weaviate.",
- ],
- learningOutcomes: [
- "Describe key considerations in vectorizer selection.",
- "List types of vectorizer modules available with Weaviate.",
- "Identify key differences between using an inference service and a local model.",
- "Select an appropriate vectorizer model for a given data and task type.",
- "Set the vectorizer for the data collection.",
- ],
- owner: "jp",
- reviewer: "jp"
- },
- indexing: {
- title: "Indexing (Advanced)",
- buttonType: "TBD",
- body: "Learn how Weaviate indexes data, and how to balance search quality with speed.",
- badgeType: "theory",
- learningGoals: [
- "How Weaviate indexes data.",
- "Weaviate's inverted and vector indexes.",
- "The available vector index algorithms.",
- "Tunable vector index parameters to balance search quality and performance."
- ],
- learningOutcomes: [
- "Describe how data indexing works within Weaviate.",
- "Recognize and describe the different indexes.",
- "Understand available .",
- "Understand and use vector indexing parameters to balance search quality and performance.",
- "Strategies for troubleshooting low search quality or performance.",
- ],
- owner: "jp",
- reviewer: "jp"
- },
-
-
- // vectorization_essentials: {
- // title: "Vectorization Essentials",
- // body: "Learn about vectors including how to create and use them.",
- // buttonType: "Click here",
- // badgeType: "mixed",
- // buttonURL: "/academy/py/units/vectorization_essentials",
- // learningGoals: [
- // "Various options for providing vector embeddings.",
- // "Important class and property parameters, including cross-references.",
- // "The relationship between the data structure and searches.",
- // "How vectors and deep learning models enable semantic search.",
- // "How Weaviate converts data into vectors.",
- // "Weaviate's text vectorization process.",
- // "Basics of vectorizer selection for Weaviate.",
- // ],
- // learningOutcomes: [
- // "Different options for providing vector embeddings in Weaviate",
- // "Understand how the data structure affects search capabilities and results.",
- // "Describe the use of vectors and deep learning models in semantic search.",
- // "Broadly explain the process by which Weaviate converts data into vectors.",
- // "Understand the fundamental concepts of text vectorization.",
- // "Outline why vectorizer selection is important for effective search.",
- // "Broadly outline available vectorizer types in Weaviate.",
- // ],
- // owner: "jp",
- // reviewer: "jp"
- // },
- // imports_in_detail: {
- // title: "Data import",
- // body: "How to efficiently import data into Weaviate.",
- // buttonType: "Click here",
- // badgeType: "practical",
- // buttonURL: "/academy/py/units/data_import",
- // learningGoals: [
- // "The ability to populate Weaviate with data, including an understanding of suggested best practices such as batching, error diagnoses and handling.",
- // ],
- // learningOutcomes: [
- // "Use upload, retrieve and change operations to a data schema in Weaviate.",
- // "Describe available batching parameters and their purpose.",
- // "Distinguish between object-level and batch-level errors occurring during import.",
- // "Identify object-level import errors and implement error handling through the Weaviate client.",
- // ],
- // owner: "jp",
- // reviewer: "jp"
- // },
- // modules: {
- // title: "Modules",
- // body: "What roles modules play, and how to enable and use them, including examples.",
- // buttonType: "TBD",
- // buttonURL: "/academy/py/units/modules",
- // badgeType: "mixed",
- // learningGoals: [
- // "Learn that Weaviate is fully modularized",
- // "Learn what modules are, how to choose, enable and use them.",
- // "Learn the difference between vectorization modules and reader/generator modules"
- // ],
- // learningOutcomes: [
- // "Explain the different types of modules that you can use with Weaviate.",
- // "Select the right modules for your use case, and enable them for your Weaviate instance."
- // ],
- // owner: "jp",
- // reviewer: "jp"
- // },
-
-
-
-
-
- // schema_2: {
- // title: "Schema 2 (Advanced)",
- // body: "Implement cross-references, and modify indexing options through the schema.",
- // buttonType: "TBD",
- // badgeType: "practical",
- // learningGoals: [
- // "Learn how to implement cross-references between data objects.",
- // "Learn about indexing options and how to set this in a schema.",
- // "Learn about module-specific schema settings and how to modify this."
- // ],
- // learningOutcomes: [
- // "Create cross-references between data classes in the schema.",
- // "Use different indexing settings for data objects in the schema.",
- // "Use module-specific settings for data objects in the schema."
- // ],
- // owner: "jp",
- // reviewer: "jp"
- // },
- // t2v_under_hood: {
- // title: "Text2vec under the hood",
- // body: "Find out exactly how Weaviate vectorizes text, and how to modify its behavior.",
- // buttonType: "TBD",
- // badgeType: "theory",
- // learningGoals: [
- // "Understand Weaviate's default object vectorization behavior in terms of data types and order of text concatenation, and how to modify this"
- // ],
- // learningOutcomes: [
- // "Explain the processes Weaviate employs to pre-process text before vectorization.",
- // "Demonstrate where to locate the exact references for data types and concatenation order in the documentation.",
- // "Implement manual vectorization (e.g. via OpenAI API) to reproduce a vector produced by Weaviate.",
- // "Formulate a vectorization strategy and defend its reasoning given a data object and a goal."
- // ],
- // owner: "jp",
- // reviewer: "jp"
- // },
- // vectorizer_selection_2: {
- // title: "Vectorizer selection 2",
- // body: "Get into the weeds in model selection: how to look for a model that best suits your use case.",
- // buttonType: "TBD",
- // badgeType: "theory",
- // learningGoals: [
- // ""
- // ],
- // learningOutcomes: [
- // ""
- // ]
- // },
- // custom_models: {
- // title: "Custom models with Weaviate",
- // body: "How you can combine your custom vectorizer model with Weaviate.",
- // buttonType: "TBD",
- // badgeType: "mixed",
- // learningGoals: [
- // ""
- // ],
- // learningOutcomes: [
- // ""
- // ]
- // },
- // module_building: {
- // title: "Module building",
- // body: "You can extend Weaviate's capabilities with custom modules. Learn how to build one to fit your needs.",
- // buttonType: "TBD",
- // badgeType: "practical",
- // learningGoals: [
- // ""
- // ],
- // learningOutcomes: [
- // ""
- // ]
- // },
- // backups: {
- // title: "Backups",
- // body: "How to back up and restore data in Weaviate: try partial and full backups to local or cloud storage.",
- // buttonType: "TBD",
- // badgeType: "practical",
- // learningGoals: [
- // "Learn how to back up and restore data in Weaviate, both partial and full backups to local and cloud storage. "
- // ],
- // learningOutcomes: [
- // "Create partial and full backups of a Weaviate instance.",
- // "Restore backups of a Weaviate instance into another instance."
- // ],
- // owner: "jp",
- // reviewer: "jp"
- // },
- // auth: {
- // title: "Authentication & Authorization",
- // body: "Identify users and control access with OpenID Connect (OIDC).",
- // buttonType: "TBD",
- // badgeType: "practical",
- // learningGoals: [
- // "An overview of authentication and authorization, as well as how to implement token-based authentication and authorization in Weaviate."
- // ],
- // learningOutcomes: [
- // "Describe the principles behind token-based security such as OIDC / OAuth.",
- // "Differentiate between an ID token and access token",
- // "Implement OIDC-based authentication with Weaviate using Weaviate Cloud as the identity provider.",
- // "Implement authorization with Weaviate based on OIDC authentication."
- // ],
- // owner: "jp",
- // reviewer: "jp"
- // },
- // scaling: {
- // title: "Scaling",
- // body: "What to expect and consider when scaling Weaviate to production.",
- // buttonType: "TBD",
- // badgeType: "theory",
- // learningGoals: [
- // ""
- // ],
- // learningOutcomes: [
- // ""
- // ]
- // },
- // replication: {
- // title: "Replication",
- // body: "What to consider when adding replication, and how to implement it.",
- // buttonType: "TBD",
- // badgeType: "mixed",
- // learningGoals: [
- // "Learn what replication is, what to consider and how to implement it for your Weaviate instance."
- // ],
- // learningOutcomes: [
- // "Describe what replication is, when to use it and how replication is designed in Weaviate.",
- // "Select the correct replication factor and consistency levels for read and write operations for various use cases.",
- // "Create a replicated Weaviate setup through settings in the schema.",
- // "Create queries to retrieve data with various consistency levels."
- // ],
- // owner: "jp",
- // reviewer: "jp"
- // },
- // clients: {
- // title: "Weaviate Clients",
- // body: "An overview: what's available, where to find them, and their capabilities.",
- // buttonType: "TBD",
- // badgeType: "mixed",
- // learningGoals: [
- // "Learn what Weaviate client libraries offer, which client languages are available and how to use them."
- // ],
- // learningOutcomes: [
- // "Understand where to find which Weaviate clients are available.",
- // "Understand the capabilities of the Weaviate clients.",
- // "Use a client to interact with Weaviate's API endpoints."
- // ],
- // owner: "jp",
- // reviewer: "jp"
- // },
- // docker: {
- // title: "Weaviate with Docker",
- // body: "How to run Weaviate on Docker, and best practice tips.",
- // buttonType: "TBD",
- // badgeType: "practical",
- // learningGoals: [
- // ""
- // ],
- // learningOutcomes: [
- // ""
- // ]
- // },
- // reader_generator: {
- // title: "Reader and Generator modules",
- // body: "Overview of question-answering, summarization and named entity recognition modules.",
- // buttonType: "TBD",
- // badgeType: "mixed",
- // learningGoals: [
- // "Learn about reader and generator modules like question answering, summerization and NER in a Weaviate pipeline, and how to use them."
- // ],
- // learningOutcomes: [
- // "Describe what reader and generator modules are.",
- // "Choose fitting reader and/or generator modules for various use cases.",
- // "Use third party (HuggingFace, OpenAI) reader and/or generator models in a Weaviate setup.",
- // "Use reader and generator modules in GraphQL queries."
- // ],
- // owner: "jp",
- // reviewer: "jp"
- // },
-
- chunking: {
- title: "Document chunking - why and how?",
- body: "Chunking is essential for working with longer texts in vector databases. This unit covers how to use it as well as tips and best practices.",
- buttonType: "Click here",
- buttonURL: "/academy/py/standalone/chunking",
- badgeType: "practical",
- learningGoals: [
- "What chunking is",
- "Its role in vector search and generative search",
- "Various chunking methods",
- "Key considerations and suggested starting points",
- ],
- learningOutcomes: [
- "Describe chunking at a high level",
- "Explain the impact of chunking in vector search and retrieval augmented generation",
- "Implement various chunking methods and know where to explore others",
- "Evaluate chunking strategies based on your needs",
- ],
- owner: "jp",
- reviewer: "jp"
- },
- kubernetes_intro: {
- title: "Weaviate with Kubernetes - An introduction",
- body: "What is Kubernetes, and how do you configure and run Weaviate on it?.",
- buttonType: "Click here",
- badgeType: "practical",
- learningGoals: [
- "What Kubernetes is, and why it is used.",
- "How to set up a local Kubernetes cluster.",
- "Run a Weaviate instance on the local Kubernetes cluster.",
- "How to configure Weaviate running on Kubernetes."
- ],
- learningOutcomes: [
- "Describe what Kubernetes is at a high level",
- "Set up minikube on your local device and run a multi-node cluster.",
- "Deploy Weaviate with replication on the multi-node cluster.",
- "Modify configurations and perform a rolling update on the Weaviate instance.",
- ]
- },
- // vectorizer_text_overview: {
- // title: "Text vectorizers: An overview",
- // body: "An overview of models - from bag-of-words to word2vec and all the way to transformers.",
- // buttonType: "TBD",
- // badgeType: "mixed",
- // learningGoals: [
- // "A brief history of text vectorization in modern natural language processing to provide context for their development including pros and cons."
- // ],
- // learningOutcomes: [
- // "Describe each of bag-of-words, word-based, and transformer models as well as RNN and LSTM models at a high level.",
- // "Categorize well-known models and methods such as TF-IDF, BM25, text2vec, GloVe, FastText, BERT, GPT and CLIP to a model type.",
- // "Distinguish key differences between each model types as well as key limitations or challenges for each model type.",
- // ],
- // owner: "jp",
- // reviewer: "jp"
- // },
- // {
- // title: "Placeholder",
- // body: "Something something dark side",
- // buttonType: "TBD",
- // badgeType: "mixed",
- // learningGoals: [
- // "TBC"
- // ],
- // learningOutcomes: [
- // "TBC"
- // ]
- // },
-
- intro_weaviate_typescript: {
- title: "Introduction to Weaviate with TS (or JS)",
- body: "A practical course where you can learn how to add Weaviate to a TypeScript (or JavaScript) app.",
- buttonType: "Click here",
- buttonURL: "/academy/js/intro_weaviate_typescript",
- badgeType: "practical",
- learningGoals: [
- "The basics of Weaviate, and how to integrate it to a TypeScript (or JavaScript) app."
- ],
- learningOutcomes: [
- "Create a cloud (WCD) instance fo Weaviate.",
- "Gain an understanding of what a vector database is.",
- "Define a schema (collection definition) and import data.",
- "Perform queries on your data.",
- "Integrate Weaviate into your TypeScript/JavaScript app.",
- ],
- owner: "jp",
- reviewer: "jp"
- },
- client_server: {
- title: "Building client-server applications",
- body: "Learn how to build fullstack applications with the Weaviate Typescript client that use the client-server approach with modern web tools.",
- buttonType: "Click here",
- buttonURL: "/academy/js/standalone/client-server",
- badgeType: "practical",
- learningGoals: [
- "What the client-server approach is and why we use it.",
- "Implementing Semantic search in Fullstack Web frameworks with Weaviate using the client-server approach.",
- "Implementing Semantic search in Backend Web frameworks with Weaviate using the client-server approach.",
- ],
- learningOutcomes: [
- "Explain where in the client-server architecture of your Web Applications Weaviate can be used.",
- "Integrate semantic search functionality into new or existing Backend and Fullstack Web Applications.",
- "Initialize and use Weaviate in your Fullstack or Backend Web Frameworks."
- ],
- owner: "daniel",
- reviewer: "daniel"
- },
- using_ml_models: {
- title: "Using Machine Learning Models",
- body: "Understand the fundamental concepts of generative and embedding models, their configuration and application in Weaviate.",
- buttonType: "Click here",
- buttonURL: "/academy/js/standalone/using-ml-models",
- badgeType: "mixed",
- learningGoals: [
- "A high level understanding of embedding and generative models.",
- "Distinguish between text and multimodal embedding types.",
- "Configuring Weaviate to use embedding and generative models.",
- "Making semantic and generative searches in Weaviate using JavaScript.",
- ],
- learningOutcomes: [
- "Differentiate between embedding and generative machine learning models.",
- "Configure Weaviate to use text and multimodal embedding models for semantic search.",
- "Configure Weaviate to use supported generative models for generative search."
- ],
- owner: "daniel",
- reviewer: "daniel"
- },
- want_stack: {
- title: "Understanding the WANT stack?",
- body: "Chunking is essential for working with longer texts in vector databases. This unit covers how to use it as well as tips and best practices.",
- buttonType: "Click here",
- buttonURL: "/academy/js/standalone/want-stack",
- badgeType: "practical",
- learningGoals: [
- "What chunking is",
- ],
- learningOutcomes: [
- "Describe chunking at a high level",
- "Explain the impact of chunking in vector search and retrieval augmented generation",
- ],
- owner: "daniel",
- reviewer: "daniel"
- },
- which_search_js: {
- title: "Which search is right for me?",
- body: "Weaviate offers many search types (vector, keyword and hybrid), and options. Let's discuss some good, baseline search strategies.",
- buttonType: "Click here",
- badgeType: "mixed",
- buttonURL: "/academy/js/standalone/which-search",
- learningGoals: [
- "Impact of search type on search quality.",
- "Impact of search type on search performance.",
- "How the dataset and chunking affect search",
- "Key considerations for selecting a search type.",
- "Strategies to apply to improve search quality.",
- ],
- learningOutcomes: [
- "Broadly recite pros and cons of each search type (vector, keyword and hybrid).",
- "Suggest a suitable search type given a description of the dataset and aim.",
- "Suggest alternative or additional search strategies to improve search quality.",
- "Outline broad methods to evaluate search quality."
- ],
- owner: "daniel",
- reviewer: "daniel"
- }
-};
diff --git a/src/components/Academy/unitcards.jsx b/src/components/Academy/unitcards.jsx
deleted file mode 100644
index 2e88079b6..000000000
--- a/src/components/Academy/unitcards.jsx
+++ /dev/null
@@ -1,27 +0,0 @@
-import React from 'react';
-import AcademyCard from './card';
-import './academy.css'
-
-function CardSet(props) {
-
- let cardData = props.cardData;
- let cardItems = [];
-
- for (let i = 0; i < cardData.length; i++) {
- cardItems.push(
-
-
- ) : (
- <>
-
-
- Feedback
-{feedback}
-
-
-
-
-
-
- Question
-
- {questions[currentQuestion].questionText}
-
- {questions[currentQuestion].answerOptions.map((answerOption) => (
-
-
- )}
-
-
-
- ))}
-
-
- )
- };
-
- return ({cardItems}
)
-}
-
-export default CardSet;
diff --git a/src/components/Academy/units.jsx b/src/components/Academy/units.jsx
deleted file mode 100644
index 69b0246b2..000000000
--- a/src/components/Academy/units.jsx
+++ /dev/null
@@ -1,31 +0,0 @@
-import React from "react";
-import UnitCardSet from "./unitcards";
-import { unitData } from '/src/components/Academy/unitData.js'
-import './academy.css'
-
-function Units(props) {
-
- let cardData = props.courseData;
- let courseName = props.courseName;
- let cardItems = [];
-
- for (let k in cardData) {
- let units = cardData[k].units.map(d => unitData[d])
-
- cardItems.push(
-
-
-
-
- )
- };
-
- return (-
{cardData[k].courseId}. {cardData[k].title}
-{cardData[k].body}
--
-
-
- {cardItems}
)
-}
-
-export default Units;
\ No newline at end of file
diff --git a/src/components/AcademyAdmonition/courses.json b/src/components/AcademyAdmonition/courses.json
new file mode 100644
index 000000000..f9d7aabaa
--- /dev/null
+++ b/src/components/AcademyAdmonition/courses.json
@@ -0,0 +1,44 @@
+[
+ {
+ "id": "quick-tour-of-weaviate",
+ "title": "A Quick Tour of Weaviate",
+ "description": "Become familiar with Weaviate's architecture, core concepts, and key capabilities. Understand how its features and integrations map to AI builders' needs.",
+ "url": "https://academy.weaviate.io/courses/wa050-py"
+ },
+ {
+ "id": "weaviate-with-python",
+ "title": "Hands-on Weaviate with Python",
+ "description": "Build your first Weaviate project to store and search data, and perform retrieval-augmented generation (RAG) with generative AI models.",
+ "url": "https://academy.weaviate.io/courses/wa101t-py"
+ },
+ {
+ "id": "ai-models-deep-dive",
+ "title": "A Gentle Deep Dive into AI Models",
+ "description": "A detailed overview of AI models from the perspective of application builders and developers. Learn how to distinguish between models and why model selection matters.",
+ "url": "https://academy.weaviate.io/courses/wa150-py"
+ },
+ {
+ "id": "fast-api-weaviate-rag",
+ "title": "Your First AI App (Search and RAG)",
+ "description": "A hands-on course where you will build a movie recommendation API with Weaviate and FastAPI.",
+ "url": "https://academy.weaviate.io/courses/wa180-py"
+ },
+ {
+ "id": "architectural-decisions-guide",
+ "title": "Architectural Decisions Guide",
+ "description": "Learn what the most important architectural decisions are when building applications with Weaviate, and how to make the right choices for your use case.",
+ "url": "https://academy.weaviate.io/courses/wa210-py"
+ },
+ {
+ "id": "embedding-model-evaluation",
+ "title": "Embedding Model Evaluation & Selection",
+ "description": "Embedding models are the heart of vector search. Learn how to evaluate and select appropriate embedding models for your use case.",
+ "url": "https://academy.weaviate.io/courses/wa150-py"
+ },
+ {
+ "id": "async-python-client-usage",
+ "title": "Async Python Client Usage",
+ "description": "Learn how to use Weaviate's async capabilities to handle concurrent operations and large-scale data processing efficiently.",
+ "url": "https://academy.weaviate.io/courses/wa280-py"
+ }
+]
diff --git a/src/components/AcademyAdmonition/index.jsx b/src/components/AcademyAdmonition/index.jsx
new file mode 100644
index 000000000..fdb350a7a
--- /dev/null
+++ b/src/components/AcademyAdmonition/index.jsx
@@ -0,0 +1,133 @@
+// src/components/AcademyAdmonition/index.jsx
+import React from "react";
+import styles from "./styles.module.scss";
+import useBaseUrl from "@docusaurus/useBaseUrl";
+import { useColorMode } from "@docusaurus/theme-common";
+import courses from "./courses.json";
+
+const AcademyAdmonition = ({
+ courseId,
+ buttonText = "Open Academy Course",
+ // Allow optional overrides
+ customTitle,
+ customDescription,
+ customUrl,
+}) => {
+ const { colorMode } = useColorMode();
+ const isDarkMode = colorMode === "dark";
+
+ // Find course by ID
+ const course = courses.find((c) => c.id === courseId);
+
+ // If course not found, show error in development
+ if (!course && !customUrl) {
+ if (process.env.NODE_ENV === "development") {
+ return (
+
+
+ );
+ }
+ // In production, return null to not render anything
+ return null;
+ }
+
+ // Use course data or custom overrides
+ const title = customTitle || course?.title;
+ const description = customDescription || course?.description;
+ const url = customUrl || course?.url;
+
+ // If still no URL, don't render
+ if (!url) return null;
+
+ // Switch logo based on theme
+ const logoPath = isDarkMode
+ ? "/img/docs/weaviate-academy-white.png"
+ : "/img/docs/weaviate-academy-purple.png";
+
+ const academyLogoUrl = useBaseUrl(logoPath);
+
+ return (
+
+ ⚠️ Academy course not found: {courseId}
+
+ Available IDs: {courses.map((c) => c.id).join(", ")} +
+
+
+ );
+};
+
+export default AcademyAdmonition;
+
+// ============================================
+// Example usage in MDX files:
+/*
+import AcademyAdmonition from '@site/src/components/AcademyAdmonition';
+
+// Simple usage with courseId from courses.json
+
+ 
+
+
+ Course: {title}
+
+
+ {description}
+ + + {buttonText} + + +