Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Couchbase support [VectorDB] #2179

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

Conversation

shyam-cb
Copy link

@shyam-cb shyam-cb commented Feb 19, 2025

Description

This PR integrates the Agno framework with Couchbase for vector search

Type of change

Please check the options that are relevant:

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Model update (Addition or modification of models)
  • Other (please describe):

Checklist

  • Adherence to standards: Code complies with Agno’s style guidelines and best practices.
  • Formatting and validation: You have run ./scripts/format.sh and ./scripts/validate.sh to ensure code is formatted and linted.
  • Self-review completed: A thorough review has been performed by the contributor(s).
  • Documentation: Docstrings and comments have been added or updated for any complex logic.
  • Examples and guides: Relevant cookbook examples have been included or updated (if applicable).
  • Tested in a clean environment: Changes have been tested in a clean environment to confirm expected behavior.
  • Tests (optional): Tests have been added or updated to cover any new or changed functionality.

Additional Notes

You need to have a running Couchbase cluster with a created bucket.

@shyam-cb shyam-cb marked this pull request as ready for review February 20, 2025 05:54
@shyam-cb shyam-cb requested a review from a team as a code owner February 20, 2025 05:54
Copy link
Contributor

@manthanguptaa manthanguptaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add steps to set up couchbase at the top of the file?

raise ImportError("`couchbase` not installed. Please install using `pip install couchbase`")


class CouchbaseFTS(VectorDb):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
class CouchbaseFTS(VectorDb):
class Couchbase(VectorDb):

What do you think about this change?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While the Couchbase name is more concise, I believe retaining CouchbaseSearch (changed from CouchbaseFTS in the latest commit) is preferable as we're planning to support vector search through multiple Couchbase services (both FTS and the indexing service). The more specific naming clearly indicates which implementation is being used and will help avoid confusion when we add additional Couchbase vector search capabilities in the future.

Comment on lines 246 to 273
def insert(self, documents: List[Document], filters: Optional[Dict[str, Any]] = None) -> None:
"""
Insert documents into the Couchbase bucket. Fails if any document already exists.

Args:
documents: List of documents to insert
filters: Optional filters (not used)
"""
logger.info(f"Inserting {len(documents)} documents")

docs_to_insert: Dict[str, Any] = {}
for document in documents:
try:
doc_data = self.prepare_doc(document)
docs_to_insert[doc_data["_id"]] = doc_data
del doc_data["_id"]
except Exception as e:
logger.error(f"Error preparing document '{document.name}': {e}")

if docs_to_insert:
try:
result = self._collection.insert_multi(docs_to_insert)
if result.all_ok:
logger.info(f"Inserted {len(docs_to_insert)} documents successfully")
else:
logger.warning(f"Bulk write error while inserting documents: {result.exceptions}")
except Exception as e:
logger.error(f"Error during bulk insert: {e}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here. Why not use filters and insert it with documents?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@manthanguptaa
Copy link
Contributor

Thanks for adding the tests! There are conflicts with the main branch that need resolution.

@shyam-cb
Copy link
Author

shyam-cb commented Mar 6, 2025

Thanks for adding the tests! There are conflicts with the main branch that need resolution.

@manthanguptaa Resolved the conflicts.

@shyam-cb shyam-cb requested a review from manthanguptaa March 6, 2025 06:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants