Skip to content

WIP: 1080 enable search metadata #1126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

WIP: 1080 enable search metadata #1126

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
96271f0
update metadata index and add it to search
longshuicy Jun 26, 2024
34d7227
alternative ways to allow metadata searchable
longshuicy Jun 26, 2024
7b35c2f
mising one spot
longshuicy Jun 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions backend/app/models/search.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,5 @@ class ElasticsearchEntry(BaseModel):
bytes: Optional[int]
# metadata fields
metadata: Optional[List[dict]] = []
metadata_stringify: Optional[str]
status: Optional[str]
3 changes: 3 additions & 0 deletions backend/app/search/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,10 @@ class IndexSettings:
# metadata fields
"metadata": {
"type": "object",
"dynamic": True,
},
# metadata fields cast to plain string to enable search
"metadata_stringify": {"type": "text"},
}
}

Expand Down
4 changes: 4 additions & 0 deletions backend/app/search/index.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import json
from typing import List, Optional, Union

from app.config import settings
Expand Down Expand Up @@ -51,6 +52,7 @@ async def index_dataset(
downloads=dataset.downloads,
user_ids=authorized_user_ids,
metadata=metadata,
metadata_stringify=json.dumps(metadata),
status=dataset_status,
).dict()

Expand Down Expand Up @@ -103,6 +105,7 @@ async def index_file(
folder_id=str(file.folder_id),
bytes=file.bytes,
metadata=metadata,
metadata_stringify=json.dumps(metadata),
status=file.status,
).dict()
if update:
Expand Down Expand Up @@ -198,6 +201,7 @@ async def index_thumbnail(
folder_id=str(file.folder_id),
bytes=thumbnail.bytes,
metadata=metadata,
metadata_stringify=json.dumps(metadata),
downloads=thumbnail.downloads,
).dict()
if update:
Expand Down
8 changes: 7 additions & 1 deletion frontend/src/components/search/EmbeddedPublicSearch.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,13 @@ export function EmbeddedPublicSearch() {
showFilter={true}
showClear
renderNoSuggestion="No suggestions found."
dataField={["name", "description"]}
dataField={[
"name",
"description",
"metadata_stringify",
"creator.keyword",
]}
fieldWeights={[3, 2, 2, 1]}
// placeholder="Search for Dataset"
innerClass={{
title: "search-title",
Expand Down
8 changes: 7 additions & 1 deletion frontend/src/components/search/EmbeddedSearch.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,13 @@ export function EmbeddedSearch() {
showFilter={true}
showClear
renderNoSuggestion="No suggestions found."
dataField={["name", "description", "creator.keyword"]}
dataField={[
"name",
"description",
"metadata_stringify",
"creator.keyword",
]}
fieldWeights={[3, 2, 2, 1]}
// placeholder="Search for Dataset"
innerClass={{
title: "search-title",
Expand Down
9 changes: 7 additions & 2 deletions frontend/src/components/search/PublicSearch.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,8 +95,13 @@ export function PublicSearch() {
showFilter={true}
showClear={true}
renderNoSuggestion="No suggestions found."
dataField={["name", "description"]}
fieldWeights={[3, 2, 1]}
dataField={[
"name",
"description",
"metadata_stringify",
"creator.keyword",
]}
fieldWeights={[3, 2, 2, 1]}
innerClass={{
title: "search-title",
input: "search-input",
Expand Down
9 changes: 7 additions & 2 deletions frontend/src/components/search/Search.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,8 +122,13 @@ export function Search() {
showFilter={true}
showClear={true}
renderNoSuggestion="No suggestions found."
dataField={["name", "description", "creator.keyword"]}
fieldWeights={[3, 2, 1]}
dataField={[
"name",
"description",
"metadata_stringify",
"creator.keyword",
]}
fieldWeights={[3, 2, 2, 1]}
innerClass={{
title: "search-title",
input: "search-input",
Expand Down