Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support secrets in tool requirements #19084

Draft
wants to merge 116 commits into
base: dev
Choose a base branch
from
Draft
Changes from all commits
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
6136e18
init add secrets to tools
arash77 Oct 29, 2024
dfd226c
add secret requirement in tools schema
arash77 Oct 30, 2024
58fe28d
check the required field with user_preferences_extra
arash77 Oct 30, 2024
71cda21
validate secret type and store for tool interface in user preferences
arash77 Oct 30, 2024
c2c8796
Add secrets into tools
arash77 Oct 30, 2024
99ff642
Add tests for secrets in tools
arash77 Oct 30, 2024
fcbe6a2
Avoid log vault_key
arash77 Oct 31, 2024
3695368
fix typo
arash77 Oct 31, 2024
f7207a1
cast app for using vault into StructuredApp
arash77 Oct 31, 2024
d3913aa
Add secrets parameter to parse_requirements_and_containers method in …
arash77 Oct 31, 2024
dafe1e2
add secrets into cwl and yml
arash77 Oct 31, 2024
b8dba5d
Fix tool parsing test to get secrets
arash77 Oct 31, 2024
8425f2c
Fix tool tests to include secrets
arash77 Oct 31, 2024
9cfc9db
Rename 'secrets' to 'credentials' in tool parsing
arash77 Nov 28, 2024
218ec24
Refactor test cases to remove unused TestSecretsInExtraUserPreference…
arash77 Dec 2, 2024
489d8b5
updating the credentials to the new format
arash77 Dec 3, 2024
82275c7
Refactor credential classes (Variable and Secret)
arash77 Dec 3, 2024
9080483
user credential model
arash77 Dec 6, 2024
16ebd12
Add API and schema for user credentials management
arash77 Dec 6, 2024
3420daa
Remove unused Union import from credentials service
arash77 Dec 6, 2024
1909f09
Add basic ToolCredentials component and related interfaces for managi…
davelopez Dec 5, 2024
2748f2e
Refactor ToolCredentials component
davelopez Dec 5, 2024
6c07263
Add badges to indicate optional and required credentials in Credentia…
davelopez Dec 5, 2024
a663273
Add WIP user credentials store
davelopez Dec 5, 2024
bd6f6cc
Refactor ToolCredentials component to enhance user messaging
davelopez Dec 5, 2024
0eecfcf
Initialize credentials in ManageToolCredentials to use a copy
davelopez Dec 5, 2024
8fe488b
Refactor credential validation logic
davelopez Dec 5, 2024
a4ead0b
update OpenAPI schema for credentials endpoints
arash77 Dec 9, 2024
94e8f19
Add update credentials API and payload models
arash77 Dec 9, 2024
355147c
Refactor credentials API schema
arash77 Dec 9, 2024
e70e5b1
Enhance ToolEvaluator to read secrets from UserVaultWrapper and query…
arash77 Dec 9, 2024
21551f6
Add new models for user credentials and tool credentials management
davelopez Dec 9, 2024
7560c2d
fixing model name
arash77 Dec 16, 2024
970a89c
updating models for user credentials
arash77 Dec 16, 2024
ec510d6
partial update for credentials api to match the new changes
arash77 Dec 16, 2024
87b625b
fix linting and schema
arash77 Dec 16, 2024
1b991fd
update credentials api
arash77 Dec 16, 2024
6b28e86
update credentials models
arash77 Dec 16, 2024
3a4d18b
update schema
arash77 Dec 16, 2024
d3d3bc1
fix linting
arash77 Dec 16, 2024
2443b66
update
arash77 Dec 18, 2024
f4ea213
refactor credentials API and schema
arash77 Dec 18, 2024
165d4e1
refactor user credentials model and introduce variable and secret model
arash77 Dec 18, 2024
4035a92
add user credentials, user credentials group,
arash77 Dec 19, 2024
925d986
update openapi schema
arash77 Dec 19, 2024
ef1cb45
rename variable and secret tables to credential_variable and credenti…
arash77 Dec 19, 2024
9527e1e
add credentials attribute to MockTool class for enhanced testing
arash77 Dec 19, 2024
248b822
rename parse_requirements_and_containers to parse_requirements,
arash77 Dec 20, 2024
5c1b38a
remove vault in evaluation and MinimalToolApp
arash77 Dec 20, 2024
2db9b5f
Fix credentials access control in API
davelopez Dec 20, 2024
b344e23
Refactor user credentials store
davelopez Dec 20, 2024
e8b486b
Refactor ToolCredentials and related UI components
davelopez Dec 20, 2024
276adc9
Update user_id type to support "current" in credentials
arash77 Jan 2, 2025
b9d0cdc
Fix user credentials retrieval to handle None case and ensure proper …
arash77 Jan 2, 2025
8096fe7
Add integration tests for user credentials API
arash77 Jan 2, 2025
720b751
Fix API endpoint string formatting in integration tests for user cred…
arash77 Jan 2, 2025
3c85c7e
Add ondelete cascade to user credential group foreign keys and update…
arash77 Jan 3, 2025
606c1dd
Refactor credential deletion logic to use lists instead of sets and a…
arash77 Jan 3, 2025
19a3138
Add ondelete cascade to foreign keys in user credentials and credenti…
arash77 Jan 3, 2025
5086dc8
Draft: Implement UserCredentialsConfigurator to manage environment va…
arash77 Jan 9, 2025
c3108bc
Refactor UserCredentialsConfigurator to handle multiple results
arash77 Jan 10, 2025
973a3d5
Refactor CredentialsService to improve credential handling and deleti…
arash77 Jan 14, 2025
c4022a9
Enhance test coverage for credential management
arash77 Jan 14, 2025
32afcdf
Refactor ToolEvaluator
arash77 Jan 14, 2025
ccdbbc9
Refactor CredentialsService to skip None values for variables and sec…
arash77 Jan 14, 2025
68c774d
Fix v-model bindings in ServiceCredentials component for variable and…
davelopez Jan 14, 2025
4302f34
Add secret placeholder handling in ManageToolCredentials and userCred…
davelopez Jan 14, 2025
96b412b
Try to avoid password manager autocompletion
davelopez Jan 14, 2025
fe783c8
Improve variable and secret handling in ManageToolCredentials
davelopez Jan 14, 2025
4dd7483
Fix new group not selected on creation by default
davelopez Jan 14, 2025
1f8a202
Enhance manage new credential sets
davelopez Jan 14, 2025
c446284
Fix bug when updating user credentials to get the updated one
arash77 Jan 15, 2025
e86127f
Add integration test for adding a new group to user credentials
arash77 Jan 15, 2025
2d27a7d
Refactoring CredentialsService and Add CredentialsManager for seperat…
arash77 Jan 15, 2025
e1777e1
Fix parameter naming in delete_credentials method calls for clarity
arash77 Jan 15, 2025
61c5aa6
Remove group_name parameter from user credentials API and related met…
arash77 Jan 15, 2025
8604232
Enhance credential management interface
davelopez Jan 15, 2025
167912f
Add functionality to delete credential groups in UI
davelopez Jan 15, 2025
cf7d2ee
Introduces update_current_group method for better group management
arash77 Jan 15, 2025
2e96b70
Refactor integration tests for user credentials
arash77 Jan 15, 2025
a6d0a1f
Refactor + add more integration tests for user credential management
davelopez Jan 16, 2025
dcabf7e
Refactors credentials management methods to only include db connectio…
arash77 Jan 16, 2025
9892db2
Add tool_id attribute to MockTool class to fix the test
arash77 Jan 20, 2025
a067009
Adds session instance to CredentialsManager constructor
arash77 Jan 20, 2025
9192515
Add unit tests for CredentialsManager functionality
arash77 Jan 20, 2025
7103686
Add test to ensure anonymous users cannot provide credentials
davelopez Jan 20, 2025
4fcadfb
Add test to ensure other users cannot list credentials
davelopez Jan 20, 2025
1796862
Refactors credential management and improves type annotations
arash77 Jan 21, 2025
6efc539
Refactor test_CredentialsManager to improve type annotations for rows…
arash77 Jan 21, 2025
e7cbb86
Refactor credentials management to improve type annotations and conso…
arash77 Jan 21, 2025
f4c7c1c
Refactor test_CredentialsManager to consolidate imports
arash77 Jan 21, 2025
dc89ac4
Renames 'reference' to 'service_reference'
arash77 Jan 27, 2025
1f90709
Fix credential check to ensure tool ID is present before accessing cr…
arash77 Jan 28, 2025
700cd78
Refactor deleteCredentialsGroupForTool function for improved readability
arash77 Jan 28, 2025
3a160e4
Replaces Secret and Variable with Credential
arash77 Jan 29, 2025
e2beb89
Update secret property references to is_set
davelopez Jan 29, 2025
9c75312
Add source_version, service_name, and service_version to user credent…
arash77 Feb 3, 2025
3feb1f6
Remove unnecessary line breaks in credentials.py for improved code cl…
arash77 Feb 3, 2025
255ad3b
Update GetToolCredentialsDefinition to return None when the tool with…
arash77 Feb 3, 2025
243f9db
Enhance error message in CredentialsService to include source type, I…
arash77 Feb 3, 2025
9a4c608
Refactor error handling in CredentialsService to raise ObjectNotFound…
arash77 Feb 3, 2025
5c750dc
Adapt user credentials UI to backend changes
davelopez Feb 3, 2025
662de35
Refactor ServiceCredentials prototype layout
davelopez Feb 3, 2025
7be9f14
Add optional vault attribute to MockApp for enhanced flexibility
arash77 Feb 4, 2025
3476041
Format ServiceCredentials.vue
arash77 Feb 4, 2025
564d98e
Refactor MinimalToolApp to include vault and install_model attributes…
arash77 Feb 5, 2025
eea7372
Remove unused cast import from evaluation.py
arash77 Feb 5, 2025
cae8215
Refactor UserCredentialsConfigurator to accept vault, session and use…
arash77 Feb 5, 2025
6d9bad0
Refactor ToolApp and evaluation logic to utilize StructuredApp attrib…
arash77 Feb 5, 2025
246fe2b
Redesign ManageToolCredentials UX
davelopez Feb 7, 2025
104797c
Refactor ServiceCredentials component
davelopez Feb 7, 2025
605688d
Add name conflict validation to ServiceCredentials component
davelopez Feb 7, 2025
253ecdf
Add indicator when the user has not provided credentials for a service
davelopez Feb 7, 2025
b39e085
Fix credential validation logic in UI
davelopez Feb 7, 2025
672ac60
Refactor credential management dialog to toggle visibility and improv…
davelopez Feb 10, 2025
fa860a6
Refactor credential handling to support null values and streamline up…
arash77 Feb 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
367 changes: 367 additions & 0 deletions client/src/api/schema/schema.ts

Large diffs are not rendered by default.

54 changes: 53 additions & 1 deletion client/src/api/users.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { GalaxyApi } from "@/api";
import { type components, GalaxyApi } from "@/api";
import { toQuotaUsage } from "@/components/User/DiskUsage/Quota/model";
import { rethrowSimple } from "@/utils/simple-error";

@@ -35,3 +35,55 @@ export async function fetchCurrentUserQuotaSourceUsage(quotaSourceLabel?: string

return toQuotaUsage(data);
}

export type CreateSourceCredentialsPayload = components["schemas"]["CreateSourceCredentialsPayload"];
export type ServiceCredentialPayload = components["schemas"]["ServiceCredentialPayload"];
export type ServiceGroupPayload = components["schemas"]["ServiceGroupPayload"];
export type UserCredentials = components["schemas"]["UserCredentialsResponse"];
export type ServiceVariableDefinition = components["schemas"]["CredentialDefinitionResponse"];

export function transformToSourceCredentials(
toolId: string,
toolCredentialsDefinition: ServiceCredentialsDefinition[]
): SourceCredentialsDefinition {
const services = new Map(
toolCredentialsDefinition.map((service) => [getKeyFromCredentialsIdentifier(service), service])
);
return {
sourceType: "tool",
sourceId: toolId,
services,
};
}

export interface ServiceCredentialsIdentifier {
name: string;
version: string;
}

export function getKeyFromCredentialsIdentifier(credentialsIdentifier: ServiceCredentialsIdentifier): string {
return `${credentialsIdentifier.name}-${credentialsIdentifier.version}`;
}

/**
* Represents the definition of credentials for a particular service.
*/
export interface ServiceCredentialsDefinition extends ServiceCredentialsIdentifier {
label?: string;
description?: string;
secrets: ServiceVariableDefinition[];
variables: ServiceVariableDefinition[];
}

/**
* Represents the definition of credentials for a particular source.
* A source can be a tool, a workflow, etc.Base interface for credentials definitions.
* A source may accept multiple services, each with its own credentials.
*
* The `services` map is indexed by the service name and version using the `getKeyFromCredentialsIdentifier` function.
*/
export interface SourceCredentialsDefinition {
sourceType: string;
sourceId: string;
services: Map<string, ServiceCredentialsDefinition>;
}
7 changes: 7 additions & 0 deletions client/src/components/Tool/ToolCard.vue
Original file line number Diff line number Diff line change
@@ -14,6 +14,7 @@ import { useUserStore } from "@/stores/userStore";
import ToolSelectPreferredObjectStore from "./ToolSelectPreferredObjectStore";
import ToolTargetPreferredObjectStorePopover from "./ToolTargetPreferredObjectStorePopover";

import ToolCredentials from "./ToolCredentials.vue";
import ToolHelpForum from "./ToolHelpForum.vue";
import ToolTutorialRecommendations from "./ToolTutorialRecommendations.vue";
import ToolFavoriteButton from "components/Tool/Buttons/ToolFavoriteButton.vue";
@@ -174,6 +175,12 @@ const showHelpForum = computed(() => isConfigLoaded.value && config.value.enable
</div>
</div>

<ToolCredentials
v-if="props.options.credentials"
:tool-id="props.id"
:tool-version="props.version"
:tool-credentials-definition="props.options.credentials" />

<div id="tool-card-body">
<FormMessage variant="danger" :message="errorText" :persistent="true" />
<FormMessage :variant="props.messageVariant" :message="props.messageText" />
249 changes: 249 additions & 0 deletions client/src/components/Tool/ToolCredentials.vue
Original file line number Diff line number Diff line change
@@ -0,0 +1,249 @@
<script setup lang="ts">
import { BAlert, BButton } from "bootstrap-vue";
import { computed, ref } from "vue";

import { isRegisteredUser } from "@/api";
import {
type CreateSourceCredentialsPayload,
type ServiceCredentialsDefinition,
type ServiceCredentialsIdentifier,
type SourceCredentialsDefinition,
transformToSourceCredentials,
type UserCredentials,
} from "@/api/users";
import { useUserCredentialsStore } from "@/stores/userCredentials";
import { useUserStore } from "@/stores/userStore";

import LoadingSpan from "@/components/LoadingSpan.vue";
import ManageToolCredentials from "@/components/User/Credentials/ManageToolCredentials.vue";

interface Props {
toolId: string;
toolVersion: string;
toolCredentialsDefinition: ServiceCredentialsDefinition[];
}

const props = defineProps<Props>();

const userStore = useUserStore();
const userCredentialsStore = useUserCredentialsStore(
isRegisteredUser(userStore.currentUser) ? userStore.currentUser.id : "anonymous"
);

const isBusy = ref(true);
const busyMessage = ref<string>("");
const userCredentials = ref<UserCredentials[] | undefined>(undefined);

const credentialsDefinition = computed<SourceCredentialsDefinition>(() => {
return transformToSourceCredentials(props.toolId, props.toolCredentialsDefinition);
});

const hasUserProvidedRequiredCredentials = computed<boolean>(() => {
if (!userCredentials.value || userCredentials.value.length === 0) {
return false;
}
return userCredentials.value.every((credentials) => areRequiredSetByUser(credentials));
});

const hasUserProvidedAllCredentials = computed<boolean>(() => {
if (!userCredentials.value || userCredentials.value.length === 0) {
return false;
}
return userCredentials.value.every(areAllSetByUser);
});

const hasSomeOptionalCredentials = computed<boolean>(() => {
for (const service of credentialsDefinition.value.services.values()) {
if (
service.secrets.some((secret) => secret.optional) ||
service.variables.some((variable) => variable.optional)
) {
return true;
}
}
return false;
});

const hasSomeRequiredCredentials = computed<boolean>(() => {
for (const service of credentialsDefinition.value.services.values()) {
if (
service.secrets.some((secret) => !secret.optional) ||
service.variables.some((variable) => !variable.optional)
) {
return true;
}
}
return false;
});

const provideCredentialsButtonTitle = computed(() => {
return hasUserProvidedRequiredCredentials.value ? "Manage credentials" : "Provide credentials";
});

const bannerVariant = computed(() => {
if (isBusy.value) {
return "info";
}
return hasUserProvidedRequiredCredentials.value ? "success" : "warning";
});

const showModal = ref(false);

/**
* Check if the user has credentials for the tool.
* @param providedCredentials - The provided credentials to check. If not provided, the function will fetch the
* credentials from the store if they exist.
*/
async function checkUserCredentials(providedCredentials?: UserCredentials[]) {
busyMessage.value = "Checking your credentials...";
isBusy.value = true;
try {
if (userStore.isAnonymous) {
return;
}

if (!providedCredentials) {
providedCredentials =
userCredentialsStore.getAllUserCredentialsForTool(props.toolId) ??
(await userCredentialsStore.fetchAllUserCredentialsForTool(props.toolId));
}

userCredentials.value = providedCredentials;
} catch (error) {
// TODO: Implement error handling.
console.error("Error checking user credentials", error);
} finally {
isBusy.value = false;
}
}

function areAllSetByUser(credentials: UserCredentials): boolean {
const selectedGroup = credentials.groups[credentials.current_group_name];
if (!selectedGroup) {
return false;
}
return (
credentials.credential_definitions.variables.every((v) => {
const variable = selectedGroup.variables.find((dv) => v.name === dv.name);
return variable?.is_set ?? false;
}) &&
credentials.credential_definitions.secrets.every((s) => {
const secret = selectedGroup.secrets.find((ds) => s.name === ds.name);
return secret?.is_set ?? false;
})
);
}

function areRequiredSetByUser(credentials: UserCredentials): boolean {
const selectedGroup = credentials.groups[credentials.current_group_name];
if (!selectedGroup) {
return false;
}
return (
credentials.credential_definitions.variables.every((v) => {
const variable = selectedGroup.variables.find((dv) => v.name === dv.name);
return variable ? variable.is_set : v.optional;
}) &&
credentials.credential_definitions.secrets.every((s) => {
const secret = selectedGroup.secrets.find((ds) => s.name === ds.name);
return secret ? secret.is_set : s.optional;
})
);
}

function toggleDialog() {
showModal.value = !showModal.value;
}

async function onSavedCredentials(providedCredentials: CreateSourceCredentialsPayload) {
showModal.value = false;
busyMessage.value = "Saving your credentials...";
try {
isBusy.value = true;
userCredentials.value = await userCredentialsStore.saveUserCredentialsForTool(providedCredentials);
} catch (error) {
// TODO: Implement error handling.
console.error("Error saving user credentials", error);
} finally {
isBusy.value = false;
}
}

async function onDeleteCredentialsGroup(serviceId: ServiceCredentialsIdentifier, groupName: string) {
busyMessage.value = "Updating your credentials...";
isBusy.value = true;
try {
userCredentialsStore.deleteCredentialsGroupForTool(props.toolId, serviceId, groupName);
} catch (error) {
// TODO: Implement error handling.
console.error("Error deleting user credentials group", error);
} finally {
isBusy.value = false;
}
}

checkUserCredentials();
</script>

<template>
<div>
<BAlert show :variant="bannerVariant" class="tool-credentials-banner">
<LoadingSpan v-if="isBusy" :message="busyMessage" />
<div v-else-if="userStore.isAnonymous">
<span v-if="hasSomeRequiredCredentials">
<strong>
This tool requires credentials to access its services and you need to be logged in to provide
them.
</strong>
</span>
<span v-else>
This tool <strong>can use additional credentials</strong> to access its services
<strong>or you can use it anonymously</strong>.
</span>
<br />
Please <a href="/login/start">log in or register here</a>.
</div>
<div v-else class="d-flex justify-content-between align-items-center">
<div class="credentials-info">
<span v-if="hasUserProvidedRequiredCredentials">
<strong>You have already provided credentials for this tool.</strong> You can update or delete
your credentials, using the <i>{{ provideCredentialsButtonTitle }}</i> button.
<span v-if="hasSomeOptionalCredentials && !hasUserProvidedAllCredentials">
<br />
You can still provide some optional credentials for this tool.
</span>
</span>
<span v-else-if="hasSomeRequiredCredentials">
This tool <strong>requires you to enter credentials</strong> to access its services. Please
provide your credentials before using the tool using the
<i>{{ provideCredentialsButtonTitle }}</i> button.
</span>
<span v-else>
This tool <strong>can use credentials</strong> to access its services. If you don't provide
credentials, you can still use the tool, but you will access its services
<strong>anonymously</strong> and in some cases, with limited functionality.
</span>
</div>

<BButton variant="primary" size="sm" class="provide-credentials-btn" @click="toggleDialog">
{{ provideCredentialsButtonTitle }}
</BButton>
</div>
</BAlert>
<ManageToolCredentials
v-if="showModal"
:tool-id="props.toolId"
:tool-version="props.toolVersion"
:tool-credentials-definition="credentialsDefinition"
:tool-user-credentials="userCredentials"
@delete-credentials-group="onDeleteCredentialsGroup"
@save-credentials="onSavedCredentials"
@close="toggleDialog" />
</div>
</template>

<style scoped>
.tool-credentials-banner {
margin-bottom: 1rem;
}
</style>
Loading
Loading