Skip to content

Commit 5726038

Browse files
tonyzhao6lickem22
tonyzhao6
and
lickem22
authored
BREAKING CHANGES: Workspace feature (#484)
* Adding termcolor to requirements. * Adding new litellm model for chat. * Fixing import path for add dummy data script. * Adding openai/chat litellm config. * Add utils to generate random int32. * Adding chat management functionalities. * Adding prompts for ChatHistory. * Removed zero-shot cot and updated system message passing. * Updated pre-commit to include types-requests. * Added chat fallback model. * Added REDIS timeout variable to config. * Updated prompts for ChatHistory. * Passing in paraphrase argument so that paraphrasing can be skipped for chat histories. * Updat chat management utilities. * Updated prompts for ChatHistory. * Added response generation with RAG and chat history. * Updated decorators and llm query generation to include chat history. Added /chat endpoint. Temporariliy commented out /search endpoint for quick testing. * Updated chat manager functions. Updated parameter name for _ask_llm_async from json to _json. * Updated prompts for spacing. * Updated prompt and chat management. * Added utils for generating random int32. * Refactored chat and search endpoints. * Consolidated chat and search endpoints. * CCs. * Removed termcolor package. * Adding types-requests to requirements-dev.txt for github workflow. * Passing along session ID for QueryResponse. * Logic shift to query refined template. * CCs. * Removing paraphrase argument. * Removing paraphrase argument. * CCs. * No need to return session ID. * Added tests for chat. * Fixing os env issue with github workflow. * Fixing os env issue with github workflow. * Fixing os env issue with github workflow. * Fixing os env issue with github workflow. * Fixing os env issue with github workflow. * Test. * Reverting tests. * CCs. * Checking mocked tests for github actions. * Updated tests. * Updated tests and fixed issue with truncation. * CCs. * CCs. * CCs. * CCs. * Removed WorkspaceRetrieve pydantic model. * CCs. * CCs. * CCs. * CCs. * Updated contents and tags packages for workspaces * Updated question_answer package for workspaces. Modified parts of data_api and llm_call packages. Finished up lagging function calls that were missing workspaces in previous commits. * Fixed function signatures. * Finished data_api package. * Finished admin package. * Updated urgency_detection and urgency_rules packages. * CCs. * Updated user_tools package. CCs. * CCs to utils and tags packages. * Updated question_answer and contents packages. * CCs to urgency_detection and uregncy_rules packages. * Updated data_api package. * CCs to llm_call/dashboard.py. * CCs to llm_call/llm_prompts.py. * Updated add_dummy_data_to_db to use workspace_id. * Updated add_new_data_to_db to use workspace_id. * Removed unused import. * Separated workspace logic into its own package with its own routers, utils, and schemas. Updated auth dependencies and routers to resolve circular import issues. * Linting. * Changing default workspace to be Workspace_{user.username}. * Added delete workspace and get workspace by user ID endpoints. * Updated table names. Added default_workspace column. Updated auth to pull default workspace. Added login-workspace endpoint. Updating tests... * CCs. * Updated workspace endpoints and schemas. Included better checks for quotas. * Checking for unique workspace name when updating workspace. Added ability to remove users from workspaces. * Added user removal functionality. * CCs to remaining modules. Fixed circular import issue and removed user_tools package---consolidated with users package now. Additional updates to users routers. * CCs. * Updated tests/rails package. * CCs. Going through and updating tests/api/conftest.py. * Updated test_admin.py. * Fixed alembic migration naming issue. Verified alembic tests pass. * Verified test_archive_content.py. * Verified test_chat.py * Verified test_data_api.py. * Verified test_import_content.py. * Verified test_import_content.py test_manage_content.py test_manage_tags.py * Verified test_manage_ud_rules.py. * Finished verifying existing tests except for dashboard tests. Added migration for on cascade deletion. * Finished verifying existing tests with pytest-randomly. Fixed lagging issues. * Added ability for any user to create a workspace. * Adding BDD tests. * Updating workspace BDD tests. * Updating workspace BDD tests. * Merging in frontend changes only for multi-turn conv. * Updating with multi-turn conv frontend PR and pylint fixes. * Adding linting make command. * Merged with topic modeling PR. * Folding in hotfixes to admin_app. * CCs. * CCs. * CCs. * CCs. * Folding in hotfixes to admin_app. * Updated dashboard package for workspace. * Verified remaining tests. * Updated github workflow for tests. Updated test_urgency_detect.py to include proper teardown. Updated dashboard filtering logic to point to UrgencyResponseDB instead of ResponseFeedbackDB. CCs. * Updated optional_components for linting and updated httpx dependency in order to pass github workflow. * Testing reverting back to using type. * Testing reverting back to using isinstance. * CCs. * CCs. * CCs. * CCs. * Added accidentally deleted pytest fixture. * Moved archive content test to its own workspace. * login endpoints now return workspace_name in AuthenticatedDetails. login-workspace now has dependency injection on get_current_user so that access token is required. workspace default quotas changed to env defaults. UserRetrieve now returns list of dicts instead of two separate lists. retrieve_all_users is now retrieve_all_users_in_current_workspace. added get_current_workspace endpoint. * Returning WorkspaceRetrieve after creating workspaces instead of WorkspaceCreate so that workspace_id is available. * Moved login-workspace endpoint to workspace/routers.py and changed to switch-workspace endpoint due to authentication requirement. Disabled updating workspace quotas on backend. * Added is_default_workspace in return object when adding existing users to a workspace. * Folding in changes for frontend and backend from update dashboard page 2 PR. * Modularizing BDD test. * CCs. * Added user resetting passwords BDD tests. * Changed endpoint from /workspace/current to /workspace/current-workspace. * Added retrieving user information BDD tests. * Added removing user BDD tests. * Fixed error in removing users from workspaces BDD tests. * Added updating user information BDD tests. Other CCs. * Added creating workspaces BDD tests. * Updating tests to pass in GHA. * Adding user role to access token and authentication. Removed is_admin attribute. * Added users endpoint to check if a username exists. * Added user routers to differentiate between creating new users and adding existing users. * Added adding users BDD tests. Put endpoint for checking if username exists back in. Separated out logic for creating new users vs. adding existing users to workspaces. * Added updating workspaces BDD tests. * Added type check. * Added retrieving workspaces BDD tests. Updated pyproject.toml and requirements-dev for coverage. * Router name change to add-existing-user-to-workspace. * Updated tests. * Updated user head endpoint to check if username exists to return a status code instead of a boolean. * CCs. * Removed access token requirement when resetting user password. Updated tests. * Merging frontend changes from main. * Added official docs for multi-turn chat and workspaces. Removed HACK FIX comments. * Changing default workspace name to {username}'s Workspace. * Consolidated workspace migration files to a single migration file that also takes care of data migration. Updated tests. * Fixing github workflow for tests. * Separated single workspace migration file into 3 stages for production. * Fixing migration script errors. * Fixing migration script errors. * Removed typo in front of Make command. * Updating with main. * Updated google authentication to check for existing user. Removed underscore character in default workspace name in migration file. * CCs. * Added BDD tests for google sign in and updated google auth logic. * Workspace feature frontend (#488) * commit message * commit message * Add workspace bar * Create new workspace component * Edit workspace button * Update edit users * Switch to diferent workspaces feature * Add new changes * Add new changes * Add user to workspace * Default workspace implementation * Add reset user logic * Remove quotas from form * clean up * Fix read only issue * Remove section from integration page for read only users * Few bug fixes * Handle long workspace names * Fix user role bug * Final fixes * Final final changes * Fixing migration script errors. * Fixing migration script errors. * Fix recovery_code being null issue * Fix comments --------- Co-authored-by: tonyzhao6 <> --------- Co-authored-by: tonyzhao6 <> Co-authored-by: lickem22 <[email protected]>
1 parent e606d0a commit 5726038

File tree

179 files changed

+24382
-8206
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

179 files changed

+24382
-8206
lines changed

‎.github/workflows/tests.yaml

+54-5
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ env:
1313
POSTGRES_DB: postgres-test-db
1414
REDIS_HOST: redis://redis:6379
1515
jobs:
16-
container-job:
16+
alembic-tests:
1717
runs-on: ubuntu-20.04
1818
container: node:20.7-bullseye
1919
services:
@@ -40,7 +40,7 @@ jobs:
4040
ports:
4141
- 6379:6379
4242
steps:
43-
- name: install dependencies
43+
- name: Install dependencies
4444
run: apt-get update && apt-get install -y lsb-release && apt-get clean all
4545
- name: Check out repository code
4646
uses: actions/checkout@v4
@@ -51,15 +51,64 @@ jobs:
5151
run: |
5252
python -m pip install -r core_backend/requirements.txt
5353
python -m pip install -r requirements-dev.txt
54-
- name: Run Unit Tests
54+
- name: Run Alembic Migration Tests
5555
env:
5656
PROMETHEUS_MULTIPROC_DIR: /tmp
5757
REDIS_HOST: ${{ env.REDIS_HOST }}
5858
run: |
5959
cd core_backend
6060
export POSTGRES_HOST=postgres POSTGRES_USER=$POSTGRES_USER \
61-
POSTGRES_PASSWORD=$POSTGRES_PASSWORD POSTGRES_DB=$POSTGRES_DB \
61+
POSTGRES_PASSWORD=$POSTGRES_PASSWORD POSTGRES_DB=$POSTGRES_DB
6262
ALIGN_SCORE_API=$ALIGN_SCORE_API
6363
python -m alembic upgrade head
6464
python -m pytest -m "not rails and alembic" tests/api/test_alembic_migrations.py
65-
python -m pytest -m "not rails and not alembic" tests
65+
unit-tests:
66+
runs-on: ubuntu-20.04
67+
container: node:20.7-bullseye
68+
services:
69+
postgres:
70+
image: pgvector/pgvector:pg16
71+
env:
72+
POSTGRES_PASSWORD: ${{ env.POSTGRES_PASSWORD }}
73+
POSTGRES_USER: ${{ env.POSTGRES_USER }}
74+
POSTGRES_DB: ${{ env.POSTGRES_DB }}
75+
options: >-
76+
--health-cmd pg_isready
77+
--health-interval 5s
78+
--health-timeout 5s
79+
--health-retries 5
80+
ports:
81+
- 5432:5432
82+
redis:
83+
image: redis:6.0-alpine
84+
options: >-
85+
--health-cmd "redis-cli ping || exit 1"
86+
--health-interval 5s
87+
--health-timeout 5s
88+
--health-retries 5
89+
ports:
90+
- 6379:6379
91+
steps:
92+
- name: Install dependencies
93+
run: apt-get update && apt-get install -y lsb-release && apt-get clean all
94+
- name: Check out repository code
95+
uses: actions/checkout@v4
96+
- uses: actions/setup-python@v4
97+
with:
98+
python-version: "3.10"
99+
- name: Install Python libraries
100+
run: |
101+
python -m pip install -r core_backend/requirements.txt
102+
python -m pip install -r requirements-dev.txt
103+
- name: Run Unit Tests (Excluding Alembic)
104+
env:
105+
PROMETHEUS_MULTIPROC_DIR: /tmp
106+
REDIS_HOST: ${{ env.REDIS_HOST }}
107+
run: |
108+
cd core_backend
109+
export POSTGRES_HOST=postgres POSTGRES_USER=$POSTGRES_USER \
110+
POSTGRES_PASSWORD=$POSTGRES_PASSWORD POSTGRES_DB=$POSTGRES_DB
111+
ALIGN_SCORE_API=$ALIGN_SCORE_API
112+
python -m alembic upgrade head
113+
python -m pytest -s -m "not rails and not alembic" --ignore-glob="tests/api/step_definitions/*" tests
114+
python -m pytest -m "not rails and not alembic" tests/api/step_definitions

‎.pylintrc

+2-1
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,8 @@ load-plugins=pylint.extensions.check_elif,
3838
pylint.extensions.docstyle,
3939
pylint.extensions.mccabe,
4040
pylint.extensions.overlapping_exceptions,
41-
pylint.extensions.redefined_variable_type
41+
pylint.extensions.redefined_variable_type,
42+
pylint_pytest
4243

4344
# Pickle collected data for later comparisons.
4445
persistent=yes

‎Makefile

+9-1
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,14 @@ fresh-env :
2929
pip install psycopg2-binary==2.9.9; \
3030
fi
3131

32+
# Linting
33+
lint-core-backend:
34+
black core_backend/
35+
ruff check core_backend/
36+
mypy core_backend/app --ignore-missing-imports --explicit-package-base
37+
mypy core_backend/tests --ignore-missing-imports --explicit-package-base
38+
pylint core_backend/
39+
3240
# Dev requirements
3341
setup-dev: setup-db setup-redis setup-llm-proxy
3442
teardown-dev: teardown-db teardown-redis teardown-llm-proxy
@@ -67,7 +75,7 @@ setup-redis:
6775
-p 6379:6379 \
6876
-d redis:6.0-alpine
6977

70-
make teardown-redis:
78+
teardown-redis:
7179
@docker stop redis-local
7280
@docker rm redis-local
7381

‎admin_app/src/app/content/page.tsx

+15-21
Original file line numberDiff line numberDiff line change
@@ -66,11 +66,12 @@ const CardsPage = () => {
6666
const [displayLanguage, setDisplayLanguage] = React.useState<string>(
6767
LANGUAGE_OPTIONS[0].label,
6868
);
69+
6970
const [searchTerm, setSearchTerm] = React.useState<string>("");
7071
const [tags, setTags] = React.useState<Tag[]>([]);
7172
const [filterTags, setFilterTags] = React.useState<Tag[]>([]);
72-
const [currAccessLevel, setCurrAccessLevel] = React.useState("readonly");
73-
const { token, accessLevel } = useAuth();
73+
const [editAccess, setEditAccess] = React.useState(false);
74+
const { token, userRole } = useAuth();
7475
const [snackMessage, setSnackMessage] = React.useState<{
7576
message: string | null;
7677
color: "success" | "info" | "warning" | "error" | undefined;
@@ -106,12 +107,11 @@ const CardsPage = () => {
106107
}
107108
};
108109
fetchTags();
109-
setCurrAccessLevel(accessLevel);
110110
} else {
111111
setTags([]);
112-
setCurrAccessLevel("readonly");
113112
}
114-
}, [accessLevel, token]);
113+
setEditAccess(userRole === "admin");
114+
}, [userRole, token]);
115115

116116
const SnackbarSlideTransition = (props: SlideProps) => {
117117
return <Slide {...props} direction="up" />;
@@ -169,7 +169,7 @@ const CardsPage = () => {
169169
}}
170170
>
171171
<CardsUtilityStrip
172-
editAccess={currAccessLevel === "fullaccess"}
172+
editAccess={editAccess}
173173
searchTerm={searchTerm}
174174
setSearchTerm={setSearchTerm}
175175
tags={tags}
@@ -185,7 +185,7 @@ const CardsPage = () => {
185185
filterTags={filterTags}
186186
openSidebar={openSearchSidebar || openChatSidebar}
187187
token={token}
188-
accessLevel={currAccessLevel}
188+
editAccess={editAccess}
189189
setSnackMessage={setSnackMessage}
190190
/>
191191
<Box
@@ -355,7 +355,7 @@ const CardsUtilityStrip: React.FC<CardsUtilityStripProps> = ({
355355
</Tooltip>
356356
<Tooltip title="Add new content">
357357
<>
358-
<AddButtonWithDropdown />
358+
<AddButtonWithDropdown editAccess={editAccess} />
359359
</>
360360
</Tooltip>
361361
<DownloadModal
@@ -400,8 +400,7 @@ const TagsFilter: React.FC<TagsFilterProps> = ({ tags, filterTags, setFilterTags
400400
);
401401
};
402402

403-
function AddButtonWithDropdown() {
404-
const [editAccess, setEditAccess] = useState(true);
403+
const AddButtonWithDropdown: React.FC<{ editAccess: boolean }> = ({ editAccess }) => {
405404
const [anchorEl, setAnchorEl] = useState<HTMLElement | null>(null);
406405
const openMenu = Boolean(anchorEl);
407406
const [openModal, setOpenModal] = useState(false);
@@ -416,15 +415,10 @@ function AddButtonWithDropdown() {
416415
return (
417416
<>
418417
<ButtonGroup variant="contained" disabled={!editAccess}>
419-
<Button
420-
disabled={!editAccess}
421-
component={Link}
422-
href="/content/edit"
423-
startIcon={<AddIcon />}
424-
>
418+
<Button component={Link} href="/content/edit" startIcon={<AddIcon />}>
425419
New
426420
</Button>
427-
<Button size="small" disabled={!editAccess} onClick={handleClick}>
421+
<Button size="small" onClick={handleClick}>
428422
<ArrowDropDownIcon />
429423
</Button>
430424
</ButtonGroup>
@@ -446,7 +440,7 @@ function AddButtonWithDropdown() {
446440
<ImportModal open={openModal} onClose={() => setOpenModal(false)} />
447441
</>
448442
);
449-
}
443+
};
450444

451445
const CardsGrid = ({
452446
displayLanguage,
@@ -455,7 +449,7 @@ const CardsGrid = ({
455449
filterTags,
456450
openSidebar,
457451
token,
458-
accessLevel,
452+
editAccess,
459453
setSnackMessage,
460454
}: {
461455
displayLanguage: string;
@@ -464,7 +458,7 @@ const CardsGrid = ({
464458
filterTags: Tag[];
465459
openSidebar: boolean;
466460
token: string | null;
467-
accessLevel: string;
461+
editAccess: boolean;
468462
setSnackMessage: React.Dispatch<
469463
React.SetStateAction<{
470464
message: string | null;
@@ -673,7 +667,7 @@ const CardsGrid = ({
673667
archiveContent={(content_id: number) => {
674668
return archiveContent(content_id, token!);
675669
}}
676-
editAccess={accessLevel === "fullaccess"}
670+
editAccess={editAccess}
677671
/>
678672
</Grid>
679673
);

‎admin_app/src/app/integrations/api.ts

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ import api from "../../utils/api";
33
const createNewApiKey = async (token: string) => {
44
try {
55
const response = await api.put(
6-
"/user/rotate-key",
6+
"/workspace/rotate-key",
77
{},
88
{
99
headers: {

‎admin_app/src/app/integrations/page.tsx

+8-8
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,12 @@ import { useAuth } from "@/utils/auth";
1111
import { KeyRenewConfirmationModal, NewKeyModal } from "./components/APIKeyModals";
1212
import ConnectionsGrid from "./components/ConnectionsGrid";
1313
import { LoadingButton } from "@mui/lab";
14-
import { getUser } from "../user-management/api";
14+
import { getCurrentWorkspace } from "../workspace-management/api";
1515

1616
const IntegrationsPage = () => {
1717
const [currAccessLevel, setCurrAccessLevel] = React.useState("readonly");
18-
const { token, accessLevel } = useAuth();
19-
18+
const { token, accessLevel, userRole } = useAuth();
19+
const editAccess = userRole == "admin";
2020
React.useEffect(() => {
2121
setCurrAccessLevel(accessLevel);
2222
}, [accessLevel]);
@@ -31,7 +31,7 @@ const IntegrationsPage = () => {
3131
maxWidth: "lg",
3232
}}
3333
>
34-
<KeyManagement token={token} editAccess={currAccessLevel === "fullaccess"} />
34+
<KeyManagement token={token} editAccess={editAccess} />
3535
<Layout.Spacer multiplier={3} />
3636
<Connections />
3737
</Box>
@@ -59,7 +59,7 @@ const KeyManagement = ({
5959
const setApiKeyInfo = async () => {
6060
setKeyInfoFetchIsLoading(true);
6161
try {
62-
const data = await getUser(token!);
62+
const data = await getCurrentWorkspace(token!);
6363
setCurrentKey(data.api_key_first_characters);
6464
const formatted_api_update_date = format(
6565
data.api_key_updated_datetime_utc,
@@ -108,7 +108,7 @@ const KeyManagement = ({
108108
}
109109
};
110110

111-
return (
111+
return editAccess ? (
112112
<Layout.FlexBox key={"key-management"} flexDirection="column" gap={sizes.baseGap}>
113113
<Typography variant="h4" color="primary">
114114
Your API Key
@@ -158,7 +158,7 @@ const KeyManagement = ({
158158
<Typography variant="body1">Generate your first API key</Typography>
159159
)}
160160
<LoadingButton
161-
variant="contained"
161+
variant={"contained"}
162162
onClick={currentKey ? handleConfirmationModalOpen : handleRenew}
163163
disabled={!editAccess}
164164
loading={keyGenerationIsLoading}
@@ -192,7 +192,7 @@ const KeyManagement = ({
192192
/>
193193
</Layout.FlexBox>
194194
</Layout.FlexBox>
195-
);
195+
) : null;
196196
};
197197

198198
const Connections = () => {

0 commit comments

Comments
 (0)