Index to search #391

joehybird · 2025-11-06T09:54:10Z

Purpose

We want to add fulltext (and semantic in a second phase) search to Drive.

The goal is to enable efficient and scalable search across document content by pushing relevant data to a dedicated search backend, such as OpenSearch. The backend should be pluggable.

Proposal

Add indexing logic in a search indexer that can be declared as a backend
Implement indexing for the Find backend. See corresponding PR in Find
Implement search views as a proxy
Implement triggers to update search index when a document or its accesses change.

This is a backport of this PR on docs

lunika

Just for your information, the library used in Drive to manage the item tree is not the same from docs. We use django-ltree and use the postgresql ltree extension. This is a significant change between the two projects, changing how to make queries and how to work with the path parameter.

src/backend/demo/management/commands/create_demo.py

env.d/development/common

src/backend/core/management/commands/index.py

src/backend/core/services/search_indexers.py

src/backend/core/tasks/search.py

Search in Drive relies on an external project like "La Suite Find". We need to declare a common external network in order to connect to the search app and index our documents. Signed-off-by: Fabre Florian <[email protected]>

Add a new Django app 'demo' that contains the command 'create_demo' Generate dummy users and files with existing factories. Signed-off-by: Fabre Florian <[email protected]>

Add SearchIndexer service that handles indexation & search API calls to Find Add SEARCH_INDEXER_* settings to configure it. Signed-off-by: Fabre Florian <[email protected]>

Add a celery task that send an item changes to the Find API A simple flag is set in cache for an amount of time that block any other task creation and do the throttle. The SEARCH_INDEXER_COUNTDOWN setting gives the number of seconds between tasks Signed-off-by: Fabre Florian <[email protected]>

When the file indexer is enabled (SEARCH_INDEXER_* settings are set) use it in place of the title filtering. Signed-off-by: Fabre Florian <[email protected]>

Reduce the number of Find API calls by grouping all the latest changes for indexation : send all the items updated or deleted since the triggering of the task. Signed-off-by: Fabre Florian <[email protected]>

Set SEARCH_INDEXER_CLASS=None as default configuration for dev. Add documentation for Find service setup. Signed-off-by: Fabre Florian <[email protected]>

Keep ordering by score from Find API on search/ results when the fulltext search is enabled. Refactor pagination to work with a list instead of a queryset Fix Changelog Signed-off-by: Fabre Florian <[email protected]>

Use nb_results instead of page/page_size argument for /search API. Add --batch-size argument to the index command. Fix an issue in SearchIndexer.has_text when item.mimetype is empty. Signed-off-by: Fabre Florian <[email protected]>

In Drive the search should find files in the trashbin for a limited amount of time. So the soft-delete cannot disable a indexed entry Be more strict on mimetype patterns. Signed-off-by: Fabre Florian <[email protected]>

Use SEARCH_INDEXER_CONTENT_MAX_SIZE as limit (in bytes) for the file content. Fix default configuration of OIDC_STORE_ACCESS_TOKEN Signed-off-by: Fabre Florian <[email protected]>

lunika · 2025-11-20T13:35:48Z

src/backend/core/api/viewsets.py

        return drf.response.Response(serializer.data, status=drf.status.HTTP_200_OK)

+    # pylint: disable-next=too-many-arguments,too-many-positional-arguments
+    def _fulltext_search(self, queryset, indexer, request, text):


something more relevant ? _external_search ? This is not really related to full text search but using an external service managing search

lunika · 2025-11-20T13:38:42Z

src/backend/core/services/search_indexers.py

+            data={
+                "q": text,
+                "visited": visited,
+                "services": ["drive"],


drive should be a constant

lunika · 2025-11-20T13:42:07Z

src/backend/core/services/search_indexers.py

+            token=token,
+        )
+
+        return [d["_id"] for d in response]


IMO this should not be part of this method. It is the class implementing this BaseItemIndexer class that know what the reponse contains. If we create an other backend and the response in the query does not have this _id property, it will not works. Probably the search method should be abstract and every backend should implement its own logic.

When indexer service is not configured, the search view should work event with a disabled OIDC_STORE_ACCESS_TOKEN. Disable token storage for the unit tests. Add bin/fernetkey that generates a key for the OIDC_STORE_REFRESH_TOKEN_KEY setting. Signed-off-by: Fabre Florian <[email protected]>

Replace "drive" by a SERVICE_NAME constant. Merge search() & search_query() methods. Signed-off-by: Fabre Florian <[email protected]>

lunika · 2025-11-21T10:07:14Z

docs/setup-find.md

+
+## Create an index service for Drive
+
+Configure a **Service** for Docs application with these settings


Suggested change

Configure a **Service** for Docs application with these settings

Configure a **Service** for Drive application with these settings

lunika · 2025-11-21T10:07:30Z

docs/setup-find.md

+
+## Configure settings of Drive
+
+Add those Django settings the Docs application to enable the feature.


Suggested change

Add those Django settings the Docs application to enable the feature.

Add those Django settings the Drive application to enable the feature.

lunika · 2025-11-21T10:08:44Z

src/backend/core/api/viewsets.py

+    @method_decorator(refresh_oidc_access_token)
+    def _indexed_search(self, request, queryset, indexer, text):
+        """
+        Returns a queryset from the results the fulltext search of Find


It doesn't return a queryset but a response.

joehybird requested review from NathanVss and lunika November 6, 2025 09:54

joehybird self-assigned this Nov 6, 2025

joehybird added enhancement New feature or request Backend labels Nov 6, 2025

joehybird force-pushed the index-to-search branch from 584dc5e to 64573c6 Compare November 6, 2025 14:25

joehybird changed the title ~~🔧(compose) configure external network for communication with search~~ Index to search Nov 7, 2025

joehybird force-pushed the index-to-search branch 2 times, most recently from 3aad56e to 8527a84 Compare November 7, 2025 14:36

joehybird requested a review from qbey November 7, 2025 14:36

lunika reviewed Nov 10, 2025

View reviewed changes

joehybird force-pushed the index-to-search branch 2 times, most recently from 8b3b277 to a1a9b14 Compare November 13, 2025 06:18

lunika reviewed Nov 13, 2025

View reviewed changes

src/backend/core/services/search_indexers.py Outdated Show resolved Hide resolved

lunika reviewed Nov 13, 2025

View reviewed changes

src/backend/core/services/search_indexers.py Outdated Show resolved Hide resolved

src/backend/core/services/search_indexers.py Show resolved Hide resolved

src/backend/core/tasks/search.py Show resolved Hide resolved

src/backend/core/tasks/search.py Outdated Show resolved Hide resolved

joehybird force-pushed the index-to-search branch 4 times, most recently from cdaa5d5 to eac0ffb Compare November 17, 2025 14:26

joehybird requested a review from lunika November 17, 2025 14:26

joehybird force-pushed the index-to-search branch from eac0ffb to d97444e Compare November 19, 2025 06:22

joehybird added 9 commits November 20, 2025 09:30

✨(backend) add demo populate command

c05ff32

Add a new Django app 'demo' that contains the command 'create_demo' Generate dummy users and files with existing factories. Signed-off-by: Fabre Florian <[email protected]>

✨(backend) add indexer service

0866cad

Add SearchIndexer service that handles indexation & search API calls to Find Add SEARCH_INDEXER_* settings to configure it. Signed-off-by: Fabre Florian <[email protected]>

✨(backend) implement fulltext search in existing search view

2f295e4

When the file indexer is enabled (SEARCH_INDEXER_* settings are set) use it in place of the title filtering. Signed-off-by: Fabre Florian <[email protected]>

✨(backend) use batches in indexing task

4d20320

Reduce the number of Find API calls by grouping all the latest changes for indexation : send all the items updated or deleted since the triggering of the task. Signed-off-by: Fabre Florian <[email protected]>

🔧(compose) disable indexer in default configuration

32bc7d6

Set SEARCH_INDEXER_CLASS=None as default configuration for dev. Add documentation for Find service setup. Signed-off-by: Fabre Florian <[email protected]>

✨(backend) keep ordering from fulltext search in results

33af2fb

Keep ordering by score from Find API on search/ results when the fulltext search is enabled. Refactor pagination to work with a list instead of a queryset Fix Changelog Signed-off-by: Fabre Florian <[email protected]>

joehybird added 2 commits November 20, 2025 09:32

✨(backend) Fix trashbin & mimetype issues

000da55

In Drive the search should find files in the trashbin for a limited amount of time. So the soft-delete cannot disable a indexed entry Be more strict on mimetype patterns. Signed-off-by: Fabre Florian <[email protected]>

✨(backend) enable limit for the indexed content

e72f76b

Use SEARCH_INDEXER_CONTENT_MAX_SIZE as limit (in bytes) for the file content. Fix default configuration of OIDC_STORE_ACCESS_TOKEN Signed-off-by: Fabre Florian <[email protected]>

joehybird force-pushed the index-to-search branch from d97444e to e72f76b Compare November 20, 2025 08:32

lunika reviewed Nov 20, 2025

View reviewed changes

joehybird force-pushed the index-to-search branch from b8b2b58 to 69e5346 Compare November 20, 2025 15:27

✨(backend) refactor indexer service

4bd9cc6

Replace "drive" by a SERVICE_NAME constant. Merge search() & search_query() methods. Signed-off-by: Fabre Florian <[email protected]>

joehybird force-pushed the index-to-search branch from 66d0331 to 4bd9cc6 Compare November 20, 2025 17:02

joehybird requested a review from lunika November 21, 2025 09:49

lunika reviewed Nov 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Index to search #391

Index to search #391

joehybird commented Nov 6, 2025 •

edited

Loading

Uh oh!

lunika left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lunika Nov 20, 2025

Uh oh!

lunika Nov 20, 2025

Uh oh!

lunika Nov 20, 2025

Uh oh!

lunika Nov 21, 2025

Uh oh!

lunika Nov 21, 2025

Uh oh!

lunika Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		## Create an index service for Drive

		Configure a Service for Docs application with these settings

	Configure a Service for Docs application with these settings
	Configure a Service for Drive application with these settings


		## Configure settings of Drive

		Add those Django settings the Docs application to enable the feature.

	Add those Django settings the Docs application to enable the feature.
	Add those Django settings the Drive application to enable the feature.

Index to search #391

Are you sure you want to change the base?

Index to search #391

Conversation

joehybird commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Proposal

Uh oh!

lunika left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lunika Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

lunika Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

lunika Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

lunika Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

lunika Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

lunika Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joehybird commented Nov 6, 2025 •

edited

Loading