Skip to content

Commit 4c40cbd

Browse files
tjazerzentjazerzen
authored
Rate limit and attachment support (#11)
> Note: Should be reviewed alongside devrev/airdrop-shared#24. This PR introduces support for - Rate limiting (through proxy) - Attachments handling (I can now confirm that acceptance tests for attachment passing means that the attachments will also get extracted to the DevRev platform) - Simplify flow for the incremental mode NOTE: Regarding attachments: In Trello, you have to construct separate endpoint for streaming attachments (Refer to `trello-openapi-compact.yaml` for more. This is handled during attachment normalization phase (in data extraction). This endpoint has for some reason separate authentication. Changes include: - Simplify every acceptance test by removing text `Make sure to replace placeholders in the resource with The Trello API Key, The Trello OAuth Token, and The Organization ID.`. Instead, we do that only once on the toplevel and it seems to working fine - Acceptance tests in `Data fetching` for rate limiting (testing API calls rate limits directly) - Acceptance tests in `Pushing data to DevRev server` for rate limiting (testing if events get emitted properly) - Added instructions for constructing the attachment URL (Trello-specific) - Added acceptance test that actually tests if attachment extraction works - Added event payload for incremental mode - Simplify and improve acceptance test for incremental mode - Update `test_data/external_domain_metadata_event_payload.json` - Update `test_data/trello_external_sync_unit_check.json` - Updated `trello-openapi-compact.yaml` to: - Also include information about rate limiting documentation - Add docs for endpoint for streaming attachments Issues connected: - [ISS-217157](https://app.devrev.ai/devrev/works/ISS-217157)
1 parent fc3cfe7 commit 4c40cbd

14 files changed

+616
-161
lines changed

‎devrev-trello-snapin.plain‎

Lines changed: 143 additions & 74 deletions
Large diffs are not rendered by default.

‎mappings/external_domain_metadata_cards.plain‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
- The record type 'cards' (Name: Cards) should have the following fields:
2-
- name (display name: "Name", is required, type: text)
3-
- url (display name: "URL", is required, type: text)
4-
- description (display name: "Description", is required, type: rich text)
5-
- id_members (display name: "ID Members", is required, type: reference)
6-
- Field id_members refers to the record type "#record:users".
7-
- Type of field id_members is an array with max_length 50.
1+
The record type 'cards' (Name: Cards) should have the following fields:
2+
- name (display name: "Name", is required, type: text)
3+
- url (display name: "URL", is required, type: text)
4+
- description (display name: "Description", is required, type: rich text)
5+
- id_members (display name: "ID Members", is required, type: reference)
6+
- Field id_members refers to the record type "#record:users".
7+
- Type of field id_members is an array with max_length 50.

‎mappings/external_domain_metadata_users.plain‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
- The record type 'users' (Name: Users) should have two fields:
2-
- full_name (display name: "Full Name", is required, type: text)
3-
- username (display name: "Username", is required, type: text)
1+
The record type 'users' (Name: Users) should have two fields:
2+
- full_name (display name: "Full Name", is required, type: text)
3+
- username (display name: "Username", is required, type: text)

‎mappings/initial_domain_mapping_cards.plain‎

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,15 @@
1-
- The record_type_mappings "cards" should have the following properties:
2-
- Default mapping should map each external card to a "issue" object.
3-
- There should be a single "possible_record_type_mappings" element, specifying:
4-
- The mapping is one-way (reverse is false, forward is true)
5-
- There should be no custom fields in the mapping.
6-
- The following The Stock Field Mapping Fields should be mapped using The External Transformation Method:
7-
- field "name" should be mapped to "title"
8-
- field "url" should be mapped to "item_url_field"
9-
- field "description" should be mapped to "body" (rich text)
10-
- field "id_members" should be mapped to "owned_by_ids" (use directly)
11-
- The following The Stock Field Mapping Fields should be mapped using The Fixed Transformation Method:
12-
- field "priority" should contain fixed value "P2"
13-
- field "stage" should contain fixed value "triage"
14-
- The following The Stock Field Mapping Fields should be mapped using The DevRev Record Transformation Method:
15-
- field "applies_to_part_id" should refer to the "product" object type
1+
The record_type_mappings "cards" should have the following properties:
2+
- Default mapping should map each external card to a "issue" object.
3+
- There should be a single "possible_record_type_mappings" element, specifying:
4+
- The mapping is one-way (reverse is false, forward is true)
5+
- There should be no custom fields in the mapping.
6+
- The following The Stock Field Mapping Fields should be mapped using The External Transformation Method:
7+
- field "name" should be mapped to "title"
8+
- field "url" should be mapped to "item_url_field"
9+
- field "description" should be mapped to "body" (rich text)
10+
- field "id_members" should be mapped to "owned_by_ids" (use directly)
11+
- The following The Stock Field Mapping Fields should be mapped using The Fixed Transformation Method:
12+
- field "priority" should contain fixed value "P2"
13+
- field "stage" should contain fixed value "triage"
14+
- The following The Stock Field Mapping Fields should be mapped using The DevRev Record Transformation Method:
15+
- field "applies_to_part_id" should refer to the "product" object type

‎mappings/initial_domain_mapping_users.plain‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
- The record_type_mappings "users" should have the following properties:
2-
- Default mapping should map each external user to a "devu" user object.
3-
- There should be a single "possible_record_type_mappings" element, specifying:
4-
- The mapping is one-way (reverse is false, forward is true).
5-
- There should be no custom fields in the mapping.
6-
- The following The Stock Field Mapping Fields should be mapped using The External Transformation Method:
7-
- field "full_name" should be mapped to "full_name".
8-
- field "username" should be mapped to "display_name".
1+
The record_type_mappings "users" should have the following properties:
2+
- Default mapping should map each external user to a "devu" user object.
3+
- There should be a single "possible_record_type_mappings" element, specifying:
4+
- The mapping is one-way (reverse is false, forward is true).
5+
- There should be no custom fields in the mapping.
6+
- The following The Stock Field Mapping Fields should be mapped using The External Transformation Method:
7+
- field "full_name" should be mapped to "full_name".
8+
- field "username" should be mapped to "display_name".

‎templates/trello_incremental_mode_acceptance_tests_first.plain‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
Validate the incremental mode with the following flow:
2+
- Step 1:
3+
- Execute request `curl -X POST "http://localhost:8003/external-worker.update-last-successful-sync/<sync_unit_id>" -H "Content-Type: application/json" -d '{"snap_in_version_id":"test-version-id","extend_state":{"users":{"completed":true},"cards":{"completed":true},"attachments":{"completed":true}}}'`, where `<sync_unit_id>` is `event["payload"]["event_context"]["sync_unit_id"]` in the resource [data_extraction_test.json](test_data/data_extraction_test.json).
4+
- Step 2:
5+
- Execute request `curl --request PUT --url 'https://api.trello.com/1/cards/688725fd3e26ebcf364bff4a?key=<TRELLO_API_KEY>&token=<TRELLO_TOKEN>&name="Card50-<uuid>"' --header 'Accept: application/json'` (replace <TRELLO_API_KEY>, <TRELLO_TOKEN> and <uuid> with the actual values).
6+
- Expect the request to succeed.
7+
- Step 3:
8+
- Invoke The Extraction Function using the resource [data_extraction_incremental_test.json](test_data/data_extraction_incremental_test.json).
9+
- Expect The Callback Server to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE".
10+
- Let `artifact_array = <event_from_callback_server>["event_data"]["artifacts"]` (array of objects).
11+
- Expect `artifact_array` to not be empty array.
12+
- Out of `artifact_array`, expect one of the elements to have "item_type" equal to "cards" (`:= cards_artifact`). Expect `cards_artifact["item_count"]=1`.
13+
- Expect that there is no element in `artifact_array` with "item_type" equal to "attachments". If there is, this indicates that the attachments data was pushed to the DevRev servers, which is wrong.
14+
- Expect that there is no element in `artifact_array` with "item_type" equal to "users". If there is, this indicates that the users data was pushed to the DevRev servers, which is wrong.

‎templates/trello_incremental_mode_acceptance_tests_second.plain‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
Validate the incremental mode with the following flow:
2+
- Step 1:
3+
- Execute request `curl -X POST "http://localhost:8003/external-worker.update-last-successful-sync/<sync_unit_id>" -H "Content-Type: application/json" -d '{"snap_in_version_id":"test-version-id","extend_state":{"users":{"completed":true},"cards":{"completed":true},"attachments":{"completed":true}}}'`, where `<sync_unit_id>` is `event["payload"]["event_context"]["sync_unit_id"]` in the resource [data_extraction_test.json](test_data/data_extraction_test.json).
4+
- Step 2:
5+
- Execute request `curl --request PUT --url 'https://api.trello.com/1/cards/688725db990240b77167efef?key=<TRELLO_API_KEY>&token=<TRELLO_TOKEN>&name="Card1-<uuid>"' --header 'Accept: application/json'` (replace <TRELLO_API_KEY>, <TRELLO_TOKEN> and <uuid> with the actual values).
6+
- Expect the request to succeed.
7+
- Step 3:
8+
- Invoke The Extraction Function using the resource [data_extraction_incremental_test.json](test_data/data_extraction_incremental_test.json).
9+
- Expect The Callback Server to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE".
10+
- Let `artifact_array = <event_from_callback_server>["event_data"]["artifacts"]` (array of objects).
11+
- Expect `artifact_array` to not be empty array.
12+
- Out of `artifact_array`, expect one of the elements to have "item_type" equal to "cards" (`:= cards_artifact`). Expect `cards_artifact["item_count"]=1`.
13+
- Out of `artifact_array`, expect one of the elements to have "item_type" equal to "attachments" (`:= attachments_artifact`). Expect `attachments_artifact["item_count"]=2`.
14+
- Expect that there is no element in `artifact_array` with "item_type" equal to "users". If there is, this indicates that the users data was pushed to the DevRev servers, which is wrong.

‎test_data/attachments_extraction_continue_test.json‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
[
2+
{
3+
"payload": {
4+
"connection_data": {
5+
"key": "key=<TRELLO_API_KEY>&token=<TRELLO_TOKEN>",
6+
"key_type": "",
7+
"org_id": "<TRELLO_ORGANIZATION_ID>",
8+
"org_name": "Trello Workspace"
9+
},
10+
"event_context": {
11+
"callback_url": "http://localhost:8002/callback",
12+
"dev_oid": "DEV-36shCCBEAA",
13+
"dev_org": "DEV-36shCCBEAA",
14+
"dev_org_id": "DEV-36shCCBEAA",
15+
"dev_uid": "DEVU-6",
16+
"dev_user": "DEVU-6",
17+
"dev_user_id": "DEVU-6",
18+
"event_type_adaas": "",
19+
"external_sync_unit": "688725dad59c015ce052eecf",
20+
"external_sync_unit_id": "688725dad59c015ce052eecf",
21+
"external_sync_unit_name": "cards-pagination-test-2025-07-28-092514",
22+
"external_system": "6752eb95c833e6b206fcf388",
23+
"external_system_id": "6752eb95c833e6b206fcf388",
24+
"external_system_name": "Trello",
25+
"external_system_type": "ADaaS",
26+
"import_slug": "trello-snapin-devrev",
27+
"initial_sync_scope": "full-history",
28+
"mode": "INITIAL",
29+
"request_id": "ff894fd5-2290-42bb-9f89-0785e49b4049",
30+
"request_id_adaas": "ff894fd5-2290-42bb-9f89-0785e49b4049",
31+
"run_id": "cbbe2419-1f86-4737-aa78-6bb7118ce52c",
32+
"sequence_version": "17",
33+
"snap_in_slug": "trello-snapin-devrev",
34+
"snap_in_version_id": "don:integration:dvrv-eu-1:devo/36shCCBEAA:snap_in_package/787b97af-95a8-4b57-809e-8d55f4e72f40:snap_in_version/50d4660e-dad9-41D6-9169-8a7e96b2d7fa",
35+
"sync_run": "cbbe2419-1f86-4737-aa78-6bb7118ce52c",
36+
"sync_run_id": "cbbe2419-1f86-4737-aa78-6bb7118ce52c",
37+
"sync_tier": "sync_tier_2",
38+
"sync_unit": "don:integration:dvrv-eu-1:devo/36shCCBEAA:external_system_type/ADAAS:external_system/6752eb95c833e6b206fcf388:sync_unit/984c894e-71e5-4e94-b484-40b839c9a916",
39+
"sync_unit_id": "984c894e-71e5-4e94-b484-40b839c9a916",
40+
"uuid": "ff894fd5-2290-42bb-9f89-0785e49b4049",
41+
"worker_data_url": "http://localhost:8003/external-worker"
42+
},
43+
"event_type": "EXTRACTION_ATTACHMENTS_CONTINUE"
44+
},
45+
"context": {
46+
"dev_oid": "don:identity:dvrv-eu-1:devo/36shCCBEAA",
47+
"automation_id": "",
48+
"source_id": "",
49+
"snap_in_id": "don:integration:dvrv-eu-1:devo/36shCCBEAA:snap_in/04bf12fa-57bd-4057-b0b0-ed3f42d9813e",
50+
"snap_in_version_id": "don:integration:dvrv-eu-1:devo/36shCCBEAA:snap_in_package/787b97af-95a8-4b57-809e-8d55f4e72f40:snap_in_version/50d4660e-dad9-41D6-9169-8a7e96b2d7fa",
51+
"service_account_id": "don:identity:dvrv-eu-1:devo/36shCCBEAA:svcacc/101",
52+
"secrets": {
53+
"service_account_token": "test-service-account-token"
54+
},
55+
"user_id": "don:identity:dvrv-eu-1:devo/36shCCBEAA:devu/6",
56+
"event_id": "",
57+
"execution_id": "4481432207487786275"
58+
},
59+
"execution_metadata": {
60+
"request_id": "ff894fd5-2290-42bb-9f89-0785e49b4049",
61+
"function_name": "extraction",
62+
"event_type": "EXTRACTION_ATTACHMENTS_START",
63+
"devrev_endpoint": "http://localhost:8003"
64+
},
65+
"input_data": {
66+
"global_values": {},
67+
"event_sources": {},
68+
"keyrings": null,
69+
"resources": {
70+
"keyrings": {},
71+
"tags": {}
72+
}
73+
}
74+
}
75+
]

‎test_data/attachments_extraction_test.json‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
[
2+
{
3+
"payload": {
4+
"connection_data": {
5+
"key": "key=<TRELLO_API_KEY>&token=<TRELLO_TOKEN>",
6+
"key_type": "",
7+
"org_id": "<TRELLO_ORGANIZATION_ID>",
8+
"org_name": "Trello Workspace"
9+
},
10+
"event_context": {
11+
"callback_url": "http://localhost:8002/callback",
12+
"dev_oid": "DEV-36shCCBEAA",
13+
"dev_org": "DEV-36shCCBEAA",
14+
"dev_org_id": "DEV-36shCCBEAA",
15+
"dev_uid": "DEVU-6",
16+
"dev_user": "DEVU-6",
17+
"dev_user_id": "DEVU-6",
18+
"event_type_adaas": "",
19+
"external_sync_unit": "688725dad59c015ce052eecf",
20+
"external_sync_unit_id": "688725dad59c015ce052eecf",
21+
"external_sync_unit_name": "cards-pagination-test-2025-07-28-092514",
22+
"external_system": "6752eb95c833e6b206fcf388",
23+
"external_system_id": "6752eb95c833e6b206fcf388",
24+
"external_system_name": "Trello",
25+
"external_system_type": "ADaaS",
26+
"import_slug": "trello-snapin-devrev",
27+
"initial_sync_scope": "full-history",
28+
"mode": "INITIAL",
29+
"request_id": "ff894fd5-2290-42bb-9f89-0785e49b4049",
30+
"request_id_adaas": "ff894fd5-2290-42bb-9f89-0785e49b4049",
31+
"run_id": "cbbe2419-1f86-4737-aa78-6bb7118ce52c",
32+
"sequence_version": "17",
33+
"snap_in_slug": "trello-snapin-devrev",
34+
"snap_in_version_id": "don:integration:dvrv-eu-1:devo/36shCCBEAA:snap_in_package/787b97af-95a8-4b57-809e-8d55f4e72f40:snap_in_version/50d4660e-dad9-41D6-9169-8a7e96b2d7fa",
35+
"sync_run": "cbbe2419-1f86-4737-aa78-6bb7118ce52c",
36+
"sync_run_id": "cbbe2419-1f86-4737-aa78-6bb7118ce52c",
37+
"sync_tier": "sync_tier_2",
38+
"sync_unit": "don:integration:dvrv-eu-1:devo/36shCCBEAA:external_system_type/ADAAS:external_system/6752eb95c833e6b206fcf388:sync_unit/984c894e-71e5-4e94-b484-40b839c9a916",
39+
"sync_unit_id": "984c894e-71e5-4e94-b484-40b839c9a916",
40+
"uuid": "ff894fd5-2290-42bb-9f89-0785e49b4049",
41+
"worker_data_url": "http://localhost:8003/external-worker"
42+
},
43+
"event_type": "EXTRACTION_ATTACHMENTS_START"
44+
},
45+
"context": {
46+
"dev_oid": "don:identity:dvrv-eu-1:devo/36shCCBEAA",
47+
"automation_id": "",
48+
"source_id": "",
49+
"snap_in_id": "don:integration:dvrv-eu-1:devo/36shCCBEAA:snap_in/04bf12fa-57bd-4057-b0b0-ed3f42d9813e",
50+
"snap_in_version_id": "don:integration:dvrv-eu-1:devo/36shCCBEAA:snap_in_package/787b97af-95a8-4b57-809e-8d55f4e72f40:snap_in_version/50d4660e-dad9-41D6-9169-8a7e96b2d7fa",
51+
"service_account_id": "don:identity:dvrv-eu-1:devo/36shCCBEAA:svcacc/101",
52+
"secrets": {
53+
"service_account_token": "test-service-account-token"
54+
},
55+
"user_id": "don:identity:dvrv-eu-1:devo/36shCCBEAA:devu/6",
56+
"event_id": "",
57+
"execution_id": "4481432207487786275"
58+
},
59+
"execution_metadata": {
60+
"request_id": "ff894fd5-2290-42bb-9f89-0785e49b4049",
61+
"function_name": "extraction",
62+
"event_type": "EXTRACTION_ATTACHMENTS_START",
63+
"devrev_endpoint": "http://localhost:8003"
64+
},
65+
"input_data": {
66+
"global_values": {},
67+
"event_sources": {},
68+
"keyrings": null,
69+
"resources": {
70+
"keyrings": {},
71+
"tags": {}
72+
}
73+
}
74+
}
75+
]

‎test_data/data_extraction_incremental_test.json‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
[
2+
{
3+
"payload": {
4+
"connection_data": {
5+
"key": "key=5545475b3c2ef986196eef0c5fb9510e&token=ATTA5aa45117249df9e5e7ae2ccb96f7207586395123f2958be49679c1e27d5f4602C8F50D28",
6+
"key_type": "",
7+
"org_id": "6752eb95c833e6b206fcf388",
8+
"org_name": "Trello Workspace"
9+
},
10+
"event_context": {
11+
"callback_url": "http://localhost:8002/callback",
12+
"dev_oid": "DEV-36shCCBEAA",
13+
"dev_org": "DEV-36shCCBEAA",
14+
"dev_org_id": "DEV-36shCCBEAA",
15+
"dev_uid": "DEVU-1",
16+
"dev_user": "DEVU-1",
17+
"dev_user_id": "DEVU-1",
18+
"event_type_adaas": "",
19+
"external_sync_unit": "688725dad59c015ce052eecf",
20+
"external_sync_unit_id": "688725dad59c015ce052eecf",
21+
"external_sync_unit_name": "SaaS connectors",
22+
"external_system": "6752eb95c833e6b206fcf388",
23+
"external_system_id": "6752eb95c833e6b206fcf388",
24+
"external_system_name": "Trello",
25+
"external_system_type": "ADaaS",
26+
"import_slug": "trello-snapin-devrev",
27+
"mode": "INCREMENTAL",
28+
"request_id": "63c6f1c6-eabe-452f-a694-7f23a8f5c3cc",
29+
"request_id_adaas": "63c6f1c6-eabe-452f-a694-7f23a8f5c3cc",
30+
"run_id": "cbbe2419-1f86-4737-aa78-6bb7118ce52c",
31+
"sequence_version": "6",
32+
"snap_in_slug": "trello-snapin-devrev",
33+
"snap_in_version_id": "don:integration:dvrv-eu-1:devo/36shCCBEAA:snap_in_package/b66dda95-cf9e-48be-918c-8439ecdd548e:snap_in_version/50d4660e-dad9-41d6-9169-8a7e96b2d7fa",
34+
"sync_run": "cbbe2419-1f86-4737-aa78-6bb7118ce52c",
35+
"sync_run_id": "cbbe2419-1f86-4737-aa78-6bb7118ce52c",
36+
"sync_tier": "sync_tier_2",
37+
"sync_unit": "don:integration:dvrv-eu-1:devo/36shCCBEAA:external_system_type/ADAAS:external_system/6752eb95c833e6b206fcf388:sync_unit/984c894e-71e5-4e94-b484-40b839c9a916",
38+
"sync_unit_id": "984c894e-71e5-4e94-b484-40b839c9a916",
39+
"uuid": "63c6f1c6-eabe-452f-a694-7f23a8f5c3cc",
40+
"worker_data_url": "http://localhost:8003/external-worker"
41+
},
42+
"event_type": "EXTRACTION_DATA_START"
43+
},
44+
"context": {
45+
"dev_oid": "don:identity:dvrv-eu-1:devo/36shCCBEAA",
46+
"automation_id": "",
47+
"source_id": "",
48+
"snap_in_id": "don:integration:dvrv-eu-1:devo/36shCCBEAA:snap_in/03a783b1-5d9f-4af8-b958-e401f2022439",
49+
"snap_in_version_id": "don:integration:dvrv-eu-1:devo/36shCCBEAA:snap_in_package/b66dda95-cf9e-48be-918c-8439ecdd548e:snap_in_version/50d4660e-dad9-41d6-9169-8a7e96b2d7fa",
50+
"service_account_id": "don:identity:dvrv-eu-1:devo/36shCCBEAA:svcacc/42",
51+
"secrets": {
52+
"service_account_token": "test-service-account-token"
53+
},
54+
"user_id": "don:identity:dvrv-eu-1:devo/36shCCBEAA:devu/1",
55+
"event_id": "",
56+
"execution_id": "13765595327067933408"
57+
},
58+
"execution_metadata": {
59+
"request_id": "63c6f1c6-eabe-452f-a694-7f23a8f5c3cc",
60+
"function_name": "extraction",
61+
"event_type": "EXTRACTION_DATA_START",
62+
"devrev_endpoint": "http://localhost:8003"
63+
},
64+
"input_data": {
65+
"global_values": {},
66+
"event_sources": {},
67+
"keyrings": null,
68+
"resources": {
69+
"keyrings": {},
70+
"tags": {}
71+
}
72+
}
73+
}
74+
]

0 commit comments

Comments
 (0)