Skip to content

Commit 925b974

Browse files
abrantesarthurabrantesarthur
authored
add support for syncing Assessments from OneTrust to Transcend (#376)
* create flattenOneTrustAssessment helper * flatten questionReponses * create helper getOneTrustRisk * update return type of getOneTrustRIsk * convert OneTrust interfaces to codecs * fix codecs * fix codecs * create flattenOneTrustSectionHeaders helper * improve codecs and create flattenOneTrustQuestions helper * improve variable names * create extractProperties helper * implement flattenOneTrustNestedQuestionsOptions * improve flattenList * improve flattenList * update flattenOneTrustNestedQuestionsOptions to use aggregateObjects * more changes * update const flattenOneTrustSectionHeaders = ( * update const flattenOneTrustSectionHeaders = ( * commit * commit * create more helpers and add tests * create enrichRiskStatisticsWithDefault * fix bug * remove extra test * create more codecs * flatten risks * flatten approvers, respondents, and primaryEntityDetails * update flattenOneTrustAssessment type * create DEFAULT_ONE_TRUST_COMBINED_ASSESSMENT * add comments * update * improve createDefatulCodec * add missing fields to codec * fix codecs * undo changes to cli-pull-ot * improve codecs * done * update writeOneTrustAssessment to write in csv format * import assessment types from privacy types * update type-utils * remove enrich helpers * import createDefaultCodec from type-utils * rename cli-pull-ot -> cli-sync-ot * add dryRun argument * add dryRun argument * move some logic from writeOneTrustAssessment to cli-sync-ot * ship more improvements to writeOneTrustAssessment * reorganize folder structure * create syncOneTrustAssessments and enrichOneTrustAssessment helpers * call syncOneTrustAssessments from cli-sync-ot * create oneTrustAssessmentToJson helper * create oneTrustAssessmentToCsv helper * create oneTrustAssessmentToCsvRecord helper * update readme * move constants to heleprs * add transcendUrl to the list of arguments * add ability to sync to Transcend * update error messages * update extractProperties documentation * add fixme comment * write docs for aggregateObjects * document flattenObject * remove some TODOs * add TODOs * remove TODOs * add a fixme * simplify flattenOneTrustAssessment * improve flattenOneTrustSections * improve flattenOneTrustSectionHeaders * simplify flatten functions * import aggregateObjects and flattenObject from @transcend/type-utils * use transposeObjectArray from @transcend/type-utils * improve flattenOneTrustAssessment * update Readme * update cli-sync-ot docs * update package.version * update commments * remove potential bugs from flattenOneTrustAssessment * add fixmes * fix flattenOneTrustQuestionResponses * fix flattenOneTrustNestedQuestionsOptions and add tests * correctly flatten risks and add tests * make progress in risk categories * write more tests * test flattenOneTrustSectionHeaders * add flattenOneTrustSections tests * default for pushing to transcend is json * update enrichOneTrustAssessment to enrich creator * create getOneTrustUser and cache fetched users * remove FIXME * enrich assessment with approver details * enrich assessments with respondents information * use privacy-types 4.105.3 * fix bugs and import codecs from privacy-types * fix bug in syncONeTrustASsessmentToDisk * rewrite syncOneTrustAssessments to run in parallel and series * fix some bugs * nits * fix pre-commit * fix tests
1 parent 801ba52 commit 925b974

33 files changed

+2036
-743
lines changed

‎.pnp.cjs

+10-10
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.yarn/cache/@transcend-io-privacy-types-npm-4.103.0-b8d1864632-4661368b34.zip

-370 KB
Binary file not shown.

‎.yarn/cache/@transcend-io-type-utils-npm-1.5.0-125f1a01fb-0d7d85e794.zip

-76.8 KB
Binary file not shown.

‎README.md

+40-13
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,7 @@
147147
- [Authentication](#authentication-34)
148148
- [Arguments](#arguments-33)
149149
- [Usage](#usage-34)
150+
- [tr-sync-ot](#tr-sync-ot)
150151
- [tr-build-xdi-sync-endpoint](#tr-build-xdi-sync-endpoint)
151152
- [Authentication](#authentication-35)
152153
- [Arguments](#arguments-34)
@@ -176,7 +177,7 @@ yarn add -D @transcend-io/cli
176177

177178
# cli commands available within package
178179
yarn tr-pull --auth=$TRANSCEND_API_KEY
179-
yarn tr-pull-ot --auth=$ONE_TRUST_OAUTH_TOKEN --hostname=$ONE_TRUST_HOSTNAME --file=$ONE_TRUST_OUTPUT_FILE
180+
yarn tr-sync-ot --auth=$ONE_TRUST_OAUTH_TOKEN --hostname=$ONE_TRUST_HOSTNAME --file=$ONE_TRUST_OUTPUT_FILE
180181
yarn tr-push --auth=$TRANSCEND_API_KEY
181182
yarn tr-scan-packages --auth=$TRANSCEND_API_KEY
182183
yarn tr-discover-silos --auth=$TRANSCEND_API_KEY
@@ -217,7 +218,7 @@ npm i -D @transcend-io/cli
217218

218219
# cli commands available within package
219220
tr-pull --auth=$TRANSCEND_API_KEY
220-
tr-pull-ot --auth=$ONE_TRUST_OAUTH_TOKEN --hostname=$ONE_TRUST_HOSTNAME --file=$ONE_TRUST_OUTPUT_FILE
221+
tr-sync-ot --auth=$ONE_TRUST_OAUTH_TOKEN --hostname=$ONE_TRUST_HOSTNAME --file=$ONE_TRUST_OUTPUT_FILE
221222
tr-push --auth=$TRANSCEND_API_KEY
222223
tr-scan-packages --auth=$TRANSCEND_API_KEY
223224
tr-discover-silos --auth=$TRANSCEND_API_KEY
@@ -577,9 +578,9 @@ tr-pull --auth=./transcend-api-keys.json --resources=consentManager --file=./tra
577578

578579
Note: This command will overwrite the existing transcend.yml file that you have locally.
579580

580-
### tr-pull-ot
581+
### tr-sync-ot
581582

582-
Pulls resources from a OneTrust instance. For now, it only supports retrieving OneTrust Assessments. It sends a request to the [Get List of Assessments](https://developer.onetrust.com/onetrust/reference/getallassessmentbasicdetailsusingget) endpoint to fetch a list of all Assessments in your account. Then, it queries the [Get Assessment](https://developer.onetrust.com/onetrust/reference/exportassessmentusingget) and [Get Risk](https://developer.onetrust.com/onetrust/reference/getriskusingget) endpoints to enrich these assessments with more details such as respondents, approvers, assessment questions and responses, and assessment risks. Finally, it syncs the enriched resources to disk in the specified file and format.
583+
Pulls resources from a OneTrust and syncs them to a Transcend instance. For now, it only supports retrieving OneTrust Assessments. It sends a request to the [Get List of Assessments](https://developer.onetrust.com/onetrust/reference/getallassessmentbasicdetailsusingget) endpoint to fetch a list of all Assessments in your account. Then, it queries the [Get Assessment](https://developer.onetrust.com/onetrust/reference/exportassessmentusingget), [Get Risk](https://developer.onetrust.com/onetrust/reference/getriskusingget), and [Get User](https://developer.onetrust.com/onetrust/reference/getuserbyid) endpoints to enrich these assessments with more details such as respondents, approvers, assessment questions and responses, and assessment risks. Finally, it syncs the enriched resources to disk in the specified file and format.
583584

584585
This command can be helpful if you are looking to:
585586

@@ -593,25 +594,51 @@ In order to use this command, you will need to generate a OneTrust OAuth Token w
593594
- [GET /v2/assessments](https://developer.onetrust.com/onetrust/reference/getallassessmentbasicdetailsusingget)
594595
- [GET /v2/assessments/{assessmentId}/export](https://developer.onetrust.com/onetrust/reference/exportassessmentusingget)
595596
- [GET /risks/{riskId}](https://developer.onetrust.com/onetrust/reference/getriskusingget)
597+
- [GET /v2/Users/{userId}](https://developer.onetrust.com/onetrust/reference/getuserusingget)
596598

597599
To learn how to generate the token, see the [OAuth 2.0 Scopes](https://developer.onetrust.com/onetrust/reference/oauth-20-scopes) and [Generate Access Token](https://developer.onetrust.com/onetrust/reference/getoauthtoken) pages.
598600

601+
If syncing the resources to Transcend, you will also need to generate an API key on the Transcend Admin Dashboard (https://app.transcend.io/infrastructure/api-keys).
602+
603+
The API key needs the following scopes when pushing the various resource types:
604+
605+
| Resource | Scope |
606+
| ----------- | ------------------ |
607+
| assessments | Manage Assessments |
608+
599609
#### Arguments
600610

601-
| Argument | Description | Type | Default | Required |
602-
| ---------- | ------------------------------------------------------------------------------------------------- | ------- | ----------- | -------- |
603-
| auth | The OAuth access token with the scopes necessary to access the OneTrust Public APIs. | string | N/A | true |
604-
| hostname | The domain of the OneTrust environment from which to pull the resource (e.g. trial.onetrust.com). | string | N/A | true |
605-
| file | Path to the file to pull the resource into. Its format must match the fileFormat argument. | string | N/A | true |
606-
| fileFormat | The format of the output file. For now, only json is supported. | string | json | false |
607-
| resource | The resource to pull from OneTrust. For now, only assessments is supported. | string | assessments | false |
608-
| debug | Whether to print detailed logs in case of error. | boolean | false | false |
611+
| Argument | Description | Type | Default | Required |
612+
| ------------- | ------------------------------------------------------------------------------------------------- | ------------ | ------------------------ | -------- |
613+
| hostname | The domain of the OneTrust environment from which to pull the resource (e.g. trial.onetrust.com). | string | N/A | true |
614+
| oneTrustAuth | The OAuth access token with the scopes necessary to access the OneTrust Public APIs. | string | N/A | true |
615+
| transcendAuth | The Transcend API Key to with the scopes necessary to access Transcend's Public APIs. | string | N/A | false |
616+
| transcendUrl | URL of the Transcend backend. Use https://api.us.transcend.io for US hosting. | string - URL | https://api.transcend.io | false |
617+
| file | Path to the file to pull the resource into. Its format must match the fileFormat argument. | string | N/A | false |
618+
| fileFormat | The format of the output file. | string | json | false |
619+
| resource | The resource to pull from OneTrust. For now, only assessments is supported. | string | assessments | false |
620+
| dryRun | Whether to export the resource to a file rather than sync to Transcend. | boolean | false | false |
621+
| debug | Whether to print detailed logs in case of error. | boolean | false | false |
609622

610623
#### Usage
611624

625+
```sh
626+
# Syncs all assessments from the OneTrust instance to Transcend
627+
tr-sync-ot --hostname=trial.onetrust.com --oneTrustAuth=$ONE_TRUST_OAUTH_TOKEN --transcendAuth=$TRANSCEND_API_KEY
628+
```
629+
630+
Alternatively, you can set dryRun to true and sync the resource to disk:
631+
632+
```sh
633+
# Writes out file to ./oneTrustAssessments.csv
634+
tr-sync-ot --hostname=trial.onetrust.com --oneTrustAuth=$ONE_TRUST_OAUTH_TOKEN --dryRun=true --file=./oneTrustAssessments.csv
635+
```
636+
637+
You can also sync to disk in json format:
638+
612639
```sh
613640
# Writes out file to ./oneTrustAssessments.json
614-
tr-pull-ot --auth=$ONE_TRUST_OAUTH_TOKEN --hostname=trial.onetrust.com --file=./oneTrustAssessments.json
641+
tr-sync-ot --hostname=trial.onetrust.com --oneTrustAuth=$ONE_TRUST_OAUTH_TOKEN --dryRun=true --fileFormat=json --file=./oneTrustAssessments.json
615642
```
616643

617644
### tr-push

‎package.json

+4-4
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"author": "Transcend Inc.",
33
"name": "@transcend-io/cli",
44
"description": "Small package containing useful typescript utilities.",
5-
"version": "6.13.2",
5+
"version": "6.14.0",
66
"homepage": "https://github.com/transcend-io/cli",
77
"repository": {
88
"type": "git",
@@ -28,7 +28,6 @@
2828
"tr-pull-consent-metrics": "./build/cli-pull-consent-metrics.js",
2929
"tr-pull-consent-preferences": "./build/cli-pull-consent-preferences.js",
3030
"tr-pull-datapoints": "./build/cli-pull-datapoints.js",
31-
"tr-pull-ot": "./build/cli-pull-ot.js",
3231
"tr-push": "./build/cli-push.js",
3332
"tr-request-approve": "./build/cli-request-approve.js",
3433
"tr-request-cancel": "./build/cli-request-cancel.js",
@@ -42,6 +41,7 @@
4241
"tr-retry-request-data-silos": "./build/cli-retry-request-data-silos.js",
4342
"tr-scan-packages": "./build/cli-scan-packages.js",
4443
"tr-skip-request-data-silos": "./build/cli-skip-request-data-silos.js",
44+
"tr-sync-ot": "./build/cli-sync-ot.js",
4545
"tr-update-consent-manager": "./build/cli-update-consent-manager-to-latest.js",
4646
"tr-upload-consent-preferences": "./build/cli-upload-consent-preferences.js",
4747
"tr-upload-cookies-from-csv": "./build/cli-upload-cookies-from-csv.js",
@@ -68,9 +68,9 @@
6868
"@transcend-io/handlebars-utils": "^1.1.0",
6969
"@transcend-io/internationalization": "^1.6.0",
7070
"@transcend-io/persisted-state": "^1.0.4",
71-
"@transcend-io/privacy-types": "^4.103.0",
71+
"@transcend-io/privacy-types": "^4.105.3",
7272
"@transcend-io/secret-value": "^1.2.0",
73-
"@transcend-io/type-utils": "^1.5.0",
73+
"@transcend-io/type-utils": "^1.8.0",
7474
"bluebird": "^3.7.2",
7575
"cli-progress": "^3.11.2",
7676
"colors": "^1.4.0",

‎src/cli-pull-ot.ts

-76
This file was deleted.

‎src/cli-sync-ot.ts

+70
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
#!/usr/bin/env node
2+
import { logger } from './logger';
3+
4+
import colors from 'colors';
5+
import { parseCliSyncOtArguments, createOneTrustGotInstance } from './oneTrust';
6+
import { OneTrustPullResource } from './enums';
7+
import { syncOneTrustAssessments } from './oneTrust/helpers/syncOneTrustAssessments';
8+
import { buildTranscendGraphQLClient } from './graphql';
9+
10+
/**
11+
* Pull configuration from OneTrust down locally to disk
12+
*
13+
* Dev Usage:
14+
* yarn ts-node ./src/cli-sync-ot.ts --hostname=customer.my.onetrust.com --oneTrustAuth=$ONE_TRUST_OAUTH_TOKEN --transcendAuth=$TRANSCEND_API_KEY
15+
*
16+
* Standard usage
17+
* yarn cli-sync-ot --hostname=customer.my.onetrust.com --oneTrustAuth=$ONE_TRUST_OAUTH_TOKEN --transcendAuth=$TRANSCEND_API_KEY
18+
*/
19+
async function main(): Promise<void> {
20+
const {
21+
file,
22+
fileFormat,
23+
hostname,
24+
oneTrustAuth,
25+
transcendAuth,
26+
transcendUrl,
27+
resource,
28+
// debug,
29+
dryRun,
30+
} = parseCliSyncOtArguments();
31+
32+
// use the hostname and auth token to instantiate a client to talk to OneTrust
33+
const oneTrust = createOneTrustGotInstance({ hostname, auth: oneTrustAuth });
34+
35+
// try {
36+
if (resource === OneTrustPullResource.Assessments) {
37+
await syncOneTrustAssessments({
38+
oneTrust,
39+
file,
40+
fileFormat,
41+
dryRun,
42+
...(transcendAuth && transcendUrl
43+
? {
44+
transcend: buildTranscendGraphQLClient(transcendUrl, transcendAuth),
45+
}
46+
: {}),
47+
});
48+
}
49+
// } catch (err) {
50+
// logger.error(
51+
// colors.red(
52+
// `An error occurred syncing the resource ${resource} from OneTrust: ${
53+
// debug ? err.stack : err.message
54+
// }`,
55+
// ),
56+
// );
57+
// process.exit(1);
58+
// }
59+
60+
// Indicate success
61+
logger.info(
62+
colors.green(
63+
`Successfully synced OneTrust ${resource} to ${
64+
dryRun ? `disk at "${file}"` : 'Transcend'
65+
}!`,
66+
),
67+
);
68+
}
69+
70+
main();

‎src/codecs.ts

+30
Original file line numberDiff line numberDiff line change
@@ -2073,3 +2073,33 @@ export const PathfinderPromptRunMetadata = t.partial({
20732073
export type PathfinderPromptRunMetadata = t.TypeOf<
20742074
typeof PathfinderPromptRunMetadata
20752075
>;
2076+
2077+
/** The columns of a row of a OneTrust Assessment form to import into Transcend. */
2078+
const OneTrustAssessmentColumnInput = t.intersection([
2079+
t.type({
2080+
/** The title of the column */
2081+
title: t.string,
2082+
}),
2083+
t.partial({
2084+
/** The optional value of the column */
2085+
value: t.string,
2086+
}),
2087+
]);
2088+
2089+
/** A row with information of the OneTrust assessment form to import into Transcend */
2090+
const OneTrustAssessmentRowInput = t.type({
2091+
/** A list of columns within this row. */
2092+
columns: t.array(OneTrustAssessmentColumnInput),
2093+
});
2094+
2095+
/** Input for importing multiple OneTrust assessment forms into Transcend */
2096+
export const ImportOnetrustAssessmentsInput = t.partial({
2097+
/** 'The rows of the CSV file.' */
2098+
rows: t.array(OneTrustAssessmentRowInput),
2099+
/** 'The json record representing the assessment.' */
2100+
json: t.string,
2101+
});
2102+
/** Type override */
2103+
export type ImportOnetrustAssessmentsInput = t.TypeOf<
2104+
typeof ImportOnetrustAssessmentsInput
2105+
>;

‎src/graphql/gqls/assessment.ts

+13
Original file line numberDiff line numberDiff line change
@@ -283,3 +283,16 @@ export const ASSESSMENTS = gql`
283283
}
284284
}
285285
`;
286+
287+
export const IMPORT_ONE_TRUST_ASSESSMENT_FORMS = gql`
288+
mutation TranscendCliImportOneTrustAssessmentForms(
289+
$input: ImportOnetrustAssessmentsInput!
290+
) {
291+
importOneTrustAssessmentForms(input: $input) {
292+
assessmentForms {
293+
id
294+
title
295+
}
296+
}
297+
}
298+
`;

0 commit comments

Comments
 (0)