Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Litellm proxy related error handling #491

Merged
merged 193 commits into from
Mar 7, 2025
Merged
Changes from 1 commit
Commits
Show all changes
193 commits
Select commit Hold shift + click to select a range
c12c919
Adding termcolor to requirements.
Nov 15, 2024
a9eb059
Adding new litellm model for chat.
Nov 15, 2024
64b0d39
Fixing import path for add dummy data script.
Nov 15, 2024
d1b0750
Adding openai/chat litellm config.
Nov 15, 2024
81569bc
Add utils to generate random int32.
Nov 15, 2024
7a53deb
Adding chat management functionalities.
Nov 15, 2024
cb3bc81
Adding prompts for ChatHistory.
Nov 15, 2024
7a11b0e
Removed zero-shot cot and updated system message passing.
Nov 15, 2024
bd9dfde
Updated pre-commit to include types-requests.
Nov 15, 2024
5a850b3
Added chat fallback model.
Nov 18, 2024
991a9e9
Added REDIS timeout variable to config.
Nov 18, 2024
0a590bf
Updated prompts for ChatHistory.
Nov 18, 2024
7ca49d7
Passing in paraphrase argument so that paraphrasing can be skipped fo…
Nov 19, 2024
6656e9f
Updat chat management utilities.
Nov 19, 2024
e8c0ca8
Updated prompts for ChatHistory.
Nov 19, 2024
197edc5
Added response generation with RAG and chat history.
Nov 19, 2024
13c1b2f
Updated decorators and llm query generation to include chat history. …
Nov 19, 2024
57bbf15
Updated chat manager functions. Updated parameter name for _ask_llm_a…
Nov 19, 2024
72ea6b1
Updated prompts for spacing.
Nov 19, 2024
9cd02dc
Updated prompt and chat management.
Nov 21, 2024
0adec3e
Added utils for generating random int32.
Jan 10, 2025
ff9da93
Refactored chat and search endpoints.
Jan 10, 2025
ba5e21a
Consolidated chat and search endpoints.
Jan 10, 2025
6828aa9
CCs.
Jan 10, 2025
8ee16eb
Removed termcolor package.
Jan 10, 2025
14d2481
Adding types-requests to requirements-dev.txt for github workflow.
Jan 10, 2025
5873478
Passing along session ID for QueryResponse.
Jan 11, 2025
eed61dd
Logic shift to query refined template.
Jan 13, 2025
7b15749
CCs.
Jan 13, 2025
d1a205e
Removing paraphrase argument.
Jan 13, 2025
161c1dc
Removing paraphrase argument.
Jan 13, 2025
959808c
CCs.
Jan 13, 2025
35713e4
No need to return session ID.
Jan 13, 2025
156243f
Added tests for chat.
Jan 13, 2025
a3efcc1
Fixing os env issue with github workflow.
Jan 13, 2025
9f2fe73
Fixing os env issue with github workflow.
Jan 13, 2025
6a2d09e
Fixing os env issue with github workflow.
Jan 13, 2025
da2135a
Fixing os env issue with github workflow.
Jan 13, 2025
96d989a
Fixing os env issue with github workflow.
Jan 13, 2025
e93f71a
Test.
Jan 13, 2025
1b95c3e
Reverting tests.
Jan 13, 2025
43875cd
CCs.
Jan 14, 2025
c288b2a
Checking mocked tests for github actions.
Jan 14, 2025
22f3820
Updated tests.
Jan 14, 2025
fa3277c
Updated tests and fixed issue with truncation.
Jan 15, 2025
f552e40
CCs.
Jan 17, 2025
e62e8c2
CCs.
Jan 18, 2025
7b7b80d
CCs.
Jan 20, 2025
ea59f0f
CCs.
Jan 22, 2025
a639ce4
Removed WorkspaceRetrieve pydantic model.
Jan 22, 2025
562eabe
CCs.
Jan 22, 2025
f4d14e7
CCs.
Jan 22, 2025
9f66f06
CCs.
Jan 22, 2025
5fdb9ac
CCs.
Jan 22, 2025
9b08f3e
Updated contents and tags packages for workspaces
Jan 23, 2025
a87e473
Updated question_answer package for workspaces. Modified parts of dat…
Jan 23, 2025
c32ec42
Fixed function signatures.
Jan 23, 2025
231c320
Finished data_api package.
Jan 23, 2025
ac840a1
Finished admin package.
Jan 24, 2025
10e7334
Updated urgency_detection and urgency_rules packages.
Jan 24, 2025
74b6e7f
CCs.
Jan 24, 2025
fa1b607
Updated user_tools package. CCs.
Jan 24, 2025
2fe56b8
CCs to utils and tags packages.
Jan 24, 2025
64fda6d
Updated question_answer and contents packages.
Jan 24, 2025
3ae784a
CCs to urgency_detection and uregncy_rules packages.
Jan 24, 2025
ff874e6
Updated data_api package.
Jan 24, 2025
4769777
CCs to llm_call/dashboard.py.
Jan 24, 2025
b2a103c
CCs to llm_call/llm_prompts.py.
Jan 24, 2025
0ea2847
Updated add_dummy_data_to_db to use workspace_id.
Jan 24, 2025
f844eb3
Updated add_new_data_to_db to use workspace_id.
Jan 24, 2025
8161e99
Removed unused import.
Jan 25, 2025
a6f50cd
Separated workspace logic into its own package with its own routers, …
Jan 25, 2025
0239ff7
Linting.
Jan 25, 2025
198adff
Changing default workspace to be Workspace_{user.username}.
Jan 25, 2025
e474c3b
Added delete workspace and get workspace by user ID endpoints.
Jan 27, 2025
fa488b9
Updated table names. Added default_workspace column. Updated auth to …
Jan 27, 2025
4c63878
CCs.
Jan 27, 2025
923b427
Updated workspace endpoints and schemas. Included better checks for q…
Jan 27, 2025
0185feb
Checking for unique workspace name when updating workspace. Added abi…
Jan 28, 2025
3a86ea6
Added user removal functionality.
Jan 28, 2025
da3a5a1
CCs to remaining modules. Fixed circular import issue and removed use…
Jan 28, 2025
a4f1a91
CCs.
Jan 28, 2025
c249dff
Updated tests/rails package.
Jan 28, 2025
582ade1
CCs. Going through and updating tests/api/conftest.py.
Jan 28, 2025
e10c6c3
Updated test_admin.py.
Jan 29, 2025
05a3d77
Fixed alembic migration naming issue. Verified alembic tests pass.
Jan 29, 2025
b6043ae
Verified test_archive_content.py.
Jan 29, 2025
b6f9f96
Verified test_chat.py
Jan 29, 2025
df67c67
Verified test_data_api.py.
Jan 29, 2025
aea1553
Verified test_import_content.py.
Jan 29, 2025
166203d
Verified test_import_content.py test_manage_content.py test_manage_ta…
Jan 30, 2025
259b441
Verified test_manage_ud_rules.py.
Jan 30, 2025
c1bba99
Finished verifying existing tests except for dashboard tests. Added m…
Jan 30, 2025
8094da3
Finished verifying existing tests with pytest-randomly. Fixed lagging…
Jan 30, 2025
23eb0f3
Added ability for any user to create a workspace.
Jan 30, 2025
acc9bb4
commit message
lickem22 Jan 31, 2025
bc59631
commit message
lickem22 Jan 31, 2025
99a08f7
Adding BDD tests.
Jan 31, 2025
02fad28
Updating workspace BDD tests.
Feb 3, 2025
ca69ccb
Updating workspace BDD tests.
Feb 3, 2025
5d90c93
Merging in frontend changes only for multi-turn conv.
Feb 3, 2025
acb3a93
Updating with multi-turn conv frontend PR and pylint fixes.
Feb 3, 2025
68bfeff
Adding linting make command.
Feb 3, 2025
5706a1e
Merged with topic modeling PR.
Feb 4, 2025
53cd86e
Folding in hotfixes to admin_app.
Feb 4, 2025
f195e76
CCs.
Feb 4, 2025
601009b
CCs.
Feb 4, 2025
2bfa8d3
CCs.
Feb 4, 2025
56a3faf
CCs.
Feb 4, 2025
390b592
Folding in hotfixes to admin_app.
Feb 4, 2025
601864b
Updated dashboard package for workspace.
Feb 4, 2025
2591f5e
Verified remaining tests.
Feb 4, 2025
3e6d30b
Add workspace bar
lickem22 Feb 5, 2025
21c187d
Merge branch 'tonyzhao6/workspace_feature' into lickem22/workspace_fr…
lickem22 Feb 5, 2025
8fc058c
Updated github workflow for tests. Updated test_urgency_detect.py to …
Feb 5, 2025
a4ab171
Updated optional_components for linting and updated httpx dependency …
Feb 5, 2025
d6f49c3
Testing reverting back to using type.
Feb 5, 2025
b298c4e
Testing reverting back to using isinstance.
Feb 5, 2025
dad1043
CCs.
Feb 5, 2025
92dc85b
CCs.
Feb 5, 2025
e7c8aff
CCs.
Feb 5, 2025
ec52e2b
CCs.
Feb 5, 2025
37aeed7
Added accidentally deleted pytest fixture.
Feb 5, 2025
8179bd5
Moved archive content test to its own workspace.
Feb 5, 2025
dc8481a
login endpoints now return workspace_name in AuthenticatedDetails. lo…
Feb 6, 2025
321c495
Create new workspace component
lickem22 Feb 7, 2025
f6984a9
Returning WorkspaceRetrieve after creating workspaces instead of Work…
Feb 7, 2025
47c9b85
Edit workspace button
lickem22 Feb 7, 2025
0928c15
Merge branch 'tonyzhao6/workspace_feature' into lickem22/workspace_fr…
lickem22 Feb 7, 2025
1a1021a
Moved login-workspace endpoint to workspace/routers.py and changed to…
Feb 7, 2025
3fedf03
Update edit users
lickem22 Feb 7, 2025
6bcdb1c
Merge branch 'tonyzhao6/workspace_feature' into lickem22/workspace_fr…
lickem22 Feb 7, 2025
4fcd1de
Added is_default_workspace in return object when adding existing user…
Feb 7, 2025
cb19287
Switch to diferent workspaces feature
lickem22 Feb 7, 2025
7b2dcbf
Folding in changes for frontend and backend from update dashboard pag…
Feb 7, 2025
a264f2d
Modularizing BDD test.
Feb 7, 2025
0a30d6a
CCs.
Feb 7, 2025
dc9dd1f
Added user resetting passwords BDD tests.
Feb 7, 2025
457736a
Changed endpoint from /workspace/current to /workspace/current-worksp…
Feb 7, 2025
c5160fe
Added retrieving user information BDD tests.
Feb 8, 2025
af3559d
Added removing user BDD tests.
Feb 8, 2025
018c7bf
Fixed error in removing users from workspaces BDD tests.
Feb 8, 2025
a9ac92f
Added updating user information BDD tests. Other CCs.
Feb 9, 2025
f195907
Added creating workspaces BDD tests.
Feb 9, 2025
bcf6acd
Updating tests to pass in GHA.
Feb 9, 2025
cc9f140
Adding user role to access token and authentication. Removed is_admin…
Feb 10, 2025
98d7691
Added users endpoint to check if a username exists.
Feb 10, 2025
6696722
Added user routers to differentiate between creating new users and ad…
Feb 10, 2025
88ed9b4
Added adding users BDD tests. Put endpoint for checking if username e…
Feb 10, 2025
4bf6a96
Added updating workspaces BDD tests.
Feb 10, 2025
54f34cc
Added type check.
Feb 10, 2025
ef4ac3e
Added retrieving workspaces BDD tests. Updated pyproject.toml and req…
Feb 10, 2025
d108d14
Router name change to add-existing-user-to-workspace.
Feb 11, 2025
9c74046
Updated tests.
Feb 11, 2025
4113bcf
Add new changes
lickem22 Feb 11, 2025
157a14c
Add new changes
lickem22 Feb 11, 2025
8103915
Merge branch 'tonyzhao6/workspace_feature' into lickem22/workspace_fr…
lickem22 Feb 11, 2025
a972cb7
Updated user head endpoint to check if username exists to return a st…
Feb 11, 2025
a78d02a
CCs.
Feb 11, 2025
af575b4
Add user to workspace
lickem22 Feb 12, 2025
46800c3
Merge branch 'tonyzhao6/workspace_feature' into lickem22/workspace_fr…
lickem22 Feb 12, 2025
88920aa
Removed access token requirement when resetting user password. Update…
Feb 14, 2025
3978059
Default workspace implementation
lickem22 Feb 16, 2025
47961f6
Merge branch 'tonyzhao6/workspace_feature' into lickem22/workspace_fr…
lickem22 Feb 16, 2025
4bb87bc
Merging frontend changes from main.
Feb 17, 2025
71f1bcb
Added official docs for multi-turn chat and workspaces. Removed HACK …
Feb 17, 2025
ad21aeb
Add reset user logic
lickem22 Feb 17, 2025
9a5e762
Merge branch 'tonyzhao6/workspace_feature' into lickem22/workspace_fr…
lickem22 Feb 17, 2025
d16c3f5
Remove quotas from form
lickem22 Feb 17, 2025
df57af2
clean up
lickem22 Feb 17, 2025
5628cdb
Fix read only issue
lickem22 Feb 18, 2025
1437c5f
Remove section from integration page for read only users
lickem22 Feb 19, 2025
e6ff5ad
Few bug fixes
lickem22 Feb 19, 2025
80d6995
Handle long workspace names
lickem22 Feb 19, 2025
e6bde96
Changing default workspace name to {username}'s Workspace.
Feb 20, 2025
68fd592
Fix user role bug
lickem22 Feb 21, 2025
dea7fbe
Consolidated workspace migration files to a single migration file tha…
Feb 22, 2025
3f1934c
Fixing github workflow for tests.
Feb 22, 2025
f19a38a
Separated single workspace migration file into 3 stages for production.
Feb 23, 2025
843c529
Merge branch 'tonyzhao6/workspace_feature' into lickem22/workspace_fr…
lickem22 Feb 24, 2025
29fbe58
Final fixes
lickem22 Feb 24, 2025
f0e03e5
Final final changes
lickem22 Feb 24, 2025
a8386ac
Fixing migration script errors.
Feb 24, 2025
7c685b1
Fixing migration script errors.
Feb 24, 2025
61b40ad
Removed typo in front of Make command.
Feb 25, 2025
b4b35a1
Fixing migration script errors.
Feb 24, 2025
72d6e58
Fixing migration script errors.
Feb 24, 2025
5210472
Updating with main.
Feb 25, 2025
5168e2e
Merge branch 'tonyzhao6/workspace_feature' into lickem22/workspace_fr…
lickem22 Feb 25, 2025
529c844
Fix recovery_code being null issue
lickem22 Feb 25, 2025
c702b43
Improve error handling for LLM endpoint
lickem22 Feb 27, 2025
94661b2
Merge branch 'main' into improve-llm-error-handling
lickem22 Feb 28, 2025
f8964e2
Catch exception when adding/edting rules
lickem22 Feb 28, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updated prompts for spacing.
  • Loading branch information
tonyzhao6 committed Jan 10, 2025
commit 72ea6b1253ec0ab0b757b80fff62da06cdddfb2b
179 changes: 93 additions & 86 deletions core_backend/app/llm_call/llm_prompts.py
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@

from pydantic import BaseModel, ConfigDict, Field

from .utils import remove_json_markdown
from .utils import format_prompt, remove_json_markdown


# ---- Language identification bot
@@ -466,88 +466,95 @@ def parse_json(self, json_str: str) -> dict[str, str]:


class ChatHistory:
system_message_construct_search_query = textwrap.dedent(
"""You are an AI assistant designed to help expecting and new mothers with
their questions/concerns related to prenatal and newborn care. You interact
with mothers via a chat interface.

Your task is to analyze the mother's LATEST MESSAGE by following these steps:

1. Determine the Type of the Mother's LATEST MESSAGE:
- Follow-up Message: These are messages that build upon the conversation so
far and/or seeks more information on a previously discussed
question/concern.
- Clarification Message: These are messages that seek to clarify something
that was previously mentioned in the conversation.
- New Message: These are messages that introduce a new topic that was not
previously discussed in the conversation.

2. Obtain More Information to Help Address the Mother's LATEST MESSAGE:
- Keep in mind the context given by the conversation history thus far.
- Use the conversation history and the Type of the Mother's LATEST MESSAGE
to formulate a precise query to execute against a vector database in order
to retrieve the most relevant information that can address the mother's
latest message given the context of the conversation history.
- Ensure the vector database query is specific and accurately reflects the
mother's information needs.
- Use specific keywords that captures the semantic meaning of the mother's
information needs.

Do NOT attempt to answer the mother's question/concern. Only output the vector
database query between the tags <Query> and </Query>, without any additional
text.
"""
).strip()
system_message_generate_response = textwrap.dedent(
"""You are an AI assistant designed to help expecting and new mothers with
their questions/concerns related to prenatal and newborn care. You interact
with mothers via a chat interface. You will be provided with ADDITIONAL
RELEVANT INFORMATION that can address the mother's questions/concerns.

BEFORE answering the mother's LATEST MESSAGE, follow these steps:

1. Review the conversation history to ensure that you understand the context in
which the mother's LATEST MESSAGE is being asked.
2. Review the provided ADDITIONAL RELEVANT INFORMATION to ensure that you
understand the most useful information related to the mother's LATEST MESSAGE.

When you have completed the above steps, you will then write a JSON, whose
TypeScript Interface is given below:

interface Response {{
extracted_info: string[];
answer: string;
}}

For "extracted_info", extract from the provided ADDITIONAL RELEVANT INFORMATION
the most useful information related to the LATEST MESSAGE asked by the mother,
and list them one by one. If no useful information is found, return an empty
list.

For "answer", understand the conversation history, ADDITIONAL RELEVANT
INFORMATION, and the mother's LATEST MESSAGE, and then provide an answer to the
mother's LATEST MESSAGE. If no useful information was found in the either the
conversation history or the ADDITIONAL RELEVANT INFORMATION, respond with
{failure_message}.

EXAMPLE RESPONSES:
{{"extracted_info": [
"Pineapples are a blend of pinecones and apples.",
"Pineapples have the shape of a pinecone."
],
"answer": "The 'pine-' from pineapples likely come from the fact that \
pineapples are a hybrid of pinecones and apples and its pinecone-like \
shape."
}}
{{"extracted_info": [], "answer": "{failure_message}"}}

IMPORTANT NOTES ON THE "answer" FIELD:
- Answer in the language of the question ({original_language}).
- Answer should be concise and to the point.
- Do not include any information that is not present in the ADDITIONAL RELEVANT
INFORMATION.

Output the JSON response between tags <JSON> and </JSON>, without any
additional text.
"""
).strip()
system_message_construct_search_query = format_prompt(
prompt=textwrap.dedent(
"""You are an AI assistant designed to help expecting and new mothers with
their questions/concerns related to prenatal and newborn care. You interact
with mothers via a chat interface.

Your task is to analyze the mother's LATEST MESSAGE by following these
steps:

1. Determine the Type of the Mother's LATEST MESSAGE:
- Follow-up Message: These are messages that build upon the
conversation so far and/or seeks more information on a previously
discussed question/concern.
- Clarification Message: These are messages that seek to clarify
something that was previously mentioned in the conversation.
- New Message: These are messages that introduce a new topic that was
not previously discussed in the conversation.

2. Obtain More Information to Help Address the Mother's LATEST MESSAGE:
- Keep in mind the context given by the conversation history thus far.
- Use the conversation history and the Type of the Mother's LATEST
MESSAGE to formulate a precise query to execute against a vector
database in order to retrieve the most relevant information that can
address the mother's LATEST MESSAGE given the context of the
conversation history.
- Ensure the vector database query is specific and accurately reflects
the mother's information needs.
- Use specific keywords that captures the semantic meaning of the
mother's information needs.

Do NOT attempt to answer the mother's question/concern. Only output the
vector database query between the tags <Query> and </Query>, without any
additional text.
"""
)
)
system_message_generate_response = format_prompt(
prompt=textwrap.dedent(
"""You are an AI assistant designed to help expecting and new mothers with
their questions/concerns related to prenatal and newborn care. You interact
with mothers via a chat interface. You will be provided with ADDITIONAL
RELEVANT INFORMATION that can address the mother's questions/concerns.

BEFORE answering the mother's LATEST MESSAGE, follow these steps:

1. Review the conversation history to ensure that you understand the
context in which the mother's LATEST MESSAGE is being asked.
2. Review the provided ADDITIONAL RELEVANT INFORMATION to ensure that you
understand the most useful information related to the mother's LATEST
MESSAGE.

When you have completed the above steps, you will then write a JSON, whose
TypeScript Interface is given below:

interface Response {{
extracted_info: string[];
answer: string;
}}

For "extracted_info", extract from the provided ADDITIONAL RELEVANT
INFORMATION the most useful information related to the LATEST MESSAGE asked
by the mother, and list them one by one. If no useful information is found,
return an empty list.

For "answer", understand the conversation history, ADDITIONAL RELEVANT
INFORMATION, and the mother's LATEST MESSAGE, and then provide an answer to
the mother's LATEST MESSAGE. If no useful information was found in the
either the conversation history or the ADDITIONAL RELEVANT INFORMATION,
respond with {failure_message}.

EXAMPLE RESPONSES:
{{"extracted_info": [
"Pineapples are a blend of pinecones and apples.",
"Pineapples have the shape of a pinecone."
],
"answer": "The 'pine-' from pineapples likely come from the fact that \
pineapples are a hybrid of pinecones and apples and its pinecone-like \
shape."
}}
{{"extracted_info": [], "answer": "{failure_message}"}}

IMPORTANT NOTES ON THE "answer" FIELD:
- Answer in the language of the question ({original_language}).
- Answer should be concise and to the point.
- Do not include any information that is not present in the ADDITIONAL
RELEVANT INFORMATION.

Output the JSON response between tags <JSON> and </JSON>, without any
additional text.
"""
)
)
2 changes: 1 addition & 1 deletion core_backend/app/question_answer/routers.py
Original file line number Diff line number Diff line change
@@ -135,7 +135,7 @@ async def chat(
returned that includes the search results as well as the details of the failure.
"""

reset_user_assistant_chat_history = False # For testing purposes only
reset_user_assistant_chat_history = True # For testing purposes only
user_query.session_id = 666 # For testing purposes only

# 1.