Milvus+PyAirbyte GenAI Solution - fully in Python local mode

[aaronsteers]

Summary

This goal of this task is to show how to use PyAirbyte with the new "Milvus Lite" product in order to build a fully local and 100% python-baesd GenAI application. This should include a RAG demo based on the loaded data. (Note: Milvus is a vector store built by Zilliz, so you might see either name in different places).

Related to:

• End-to-end RAG tutorial using Amazon SQS, PyAirbyte and Milvus (using Langchain) #21
• End-to-end RAG with Airbyte Cloud, Microsoft Sharepoint and Zilliz(Milvus) #12

This item is different from each of those because (1) we are more flexible on which data source used, and (2) the emphasis is on using the local python versions of the products, rather than the hosted or self-managed Mivus service.

Definition of Done

• Deliver documentation (Blog post, Python notebook, etc., )covering end to end steps including code snippets when applicable. When providing a python notebook, please add a "What / Why / How" blurb at the top to explain what the code is doing.

Resources to Assist

• Example tutorial: https://github.com/airbytehq/quickstarts/blob/main/vector_store_integration/RAG_using_Snowflake_Cortex.ipynb
• s3 source documentation: https://docs.airbyte.com/integrations/sources/s3
• Milvus documentation: https://docs.airbyte.com/integrations/destinations/milvus
• LangChain Zilliz(Milvus) support: https://python.langchain.com/v0.1/docs/integrations/vectorstores/zilliz/
• Sample blog post: https://airbyte.com/tutorials/chat-with-your-data-using-openai-pinecone-airbyte-and-langchain

[pangeran-bottor]

Hi @aaronsteers , I would like to work on this. I would appreciate it if you could update the summary here so I can get the tutorial's scope. Thank you!

[marcosmarxm]

@pangeran-bottor I assigned the issue to you.
@aaronsteers let's work to update the description to provide more information? :D

[aaronsteers]

@pangeran-bottor - Thanks very much for your interest! I've just added a full description above. The key attribute that makes this different from other related demos, is that using "Milvus Lite" and PyAirbyte, you can build a fully local pipeline that only requires your local Python runtime.

We think this kind of a tutorial would be super helpful for people who want to build a fast POC or a very specific solution to a problem where scale+redundancy are less of a priority.

We've already assigned you - but let us know if this still sounds like a good fit for you! We're excited to work with you on it if so. 🚀

[pangeran-bottor]

@aaronsteers Thanks for the detailed explanation! This sounds good; let me work on it.

[aaronsteers]

@pangeran-bottor - It's all yours! 👍

[marcosmarxm]

@pangeran-bottor do you need any help here?

[pangeran-bottor]

@marcosmarxm sorry for the delay; aiming to send the PR within the next two days

[marcosmarxm]

Don't worry just doing a check-in to see if you need any help!

[pangeran-bottor]

Hi guys, putting the PR here: airbytehq/quickstarts#120

[bindipankhudi]

Thank you @pangeran-bottor. We will review it shortly.

[bindipankhudi]

The PR looks great @pangeran-bottor. Reviewed and merged!

[pangeran-bottor]

Thanks @bindipankhudi !. Can you or @marcosmarxm / @aaronsteers perhaps change the status to DONE? 😄
Also, I would appreciate it if you could assign me to another quickstart/tutorial issue.
I was thinking either #19 or #21 , both seems inactive for a week. Thank you.

[aaronsteers]

@pangeran-bottor - Thanks for contributing! Marking as "Done". ✅ 🙌