diff --git a/fern/docs/pages/airdrop/attachments-extraction.mdx b/fern/docs/pages/airdrop/attachments-extraction.mdx index b3079efa..579a540e 100644 --- a/fern/docs/pages/airdrop/attachments-extraction.mdx +++ b/fern/docs/pages/airdrop/attachments-extraction.mdx @@ -10,8 +10,8 @@ During the attachment extraction phase, the snap-in extracts attachments from the external system and uploads them as artifacts to DevRev. The snap-in must respond to Airdrop with a message with an event of type -`EXTRACTION_ATTACHMENTS_PROGRESS` together with an optional progress estimate and relevant artifacts -when it extracts some data and the maximum snap-in run time (12 minutes) has been reached. +`EXTRACTION_ATTACHMENTS_PROGRESS` together with an optional progress estimate +when the maximum snap-in runtime (13 minutes) has been reached. The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_DELAY` and specify a back-off time when the extraction has been rate-limited by the external system and @@ -42,6 +42,26 @@ domain object ID from the external system and the actor ID from the external sys The uploaded artifact is structured like a normal artifact containing extracted data in JSON Lines (JSONL) format and requires specifying `ssor_attachment` as the item type. +The snap-in must respond to Airdrop with a message, that either signals success, a delay, or an error. + +```typescript Success +await adapter.emit(ExtractorEventType.ExtractionAttachmentsDone); +``` + +```typescript Delay +await adapter.emit(ExtractorEventType.ExtractionAttachmentsDelay, { + delay: "30", +}); +``` + +```typescript Error +await adapter.emit(ExtractorEventType.ExtractionAttachmentsError, { + error: "Informative error message", +}); +``` + +The snap-in must always emit a single message. + ## Examples Here is an example of an SSOR attachment file: diff --git a/fern/docs/pages/airdrop/data-extraction.mdx b/fern/docs/pages/airdrop/data-extraction.mdx index c17ea0da..5d50078f 100644 --- a/fern/docs/pages/airdrop/data-extraction.mdx +++ b/fern/docs/pages/airdrop/data-extraction.mdx @@ -1,47 +1,120 @@ In the data extraction phase, the extractor is expected to call the external system's APIs -to retrieve all the items that were updated since the start of the last extraction. -If there was no previous extraction (the current run is an initial import), -then all the items should be extracted. +to retrieve all the items that should be synced with DevRev. -The extractor must store at what time it started each extraction in its state, -so that it can extract only items created or updated since this date in the next sync run. +If the current run is an initial sync, this means all the items should be extracted. +Otherwise the extractor should retrieve all the items that were changed since the start of the last extraction. + +Each snap-in invocation runs in a separate runtime instance with a maximum execution time of 13 minutes. +After 10 minutes, the Airdrop platform sends a message to the snap-in to gracefully exit. + +If a large amount of data needs to be extracted, it might not all be extracted within this time frame. +To handle such situations, the snap-in uses a state object. +This state object is shared across all invocations and keeps track of where the previous snap-in invocations ended in the extraction process. ## Triggering event -Airdrop initiates data extraction by starting the snap-in with a message with event of type +Airdrop initiates data extraction by starting the snap-in with a message with event type `EXTRACTION_DATA_START` when transitioning to the data extraction phase. During the data extraction phase, the snap-in extracts data from an external system, -prepares batches of data and uploads them in the form of artifacts to DevRev. +prepares batches of data and uploads them in the form of artifacts (files) to DevRev. -The snap-in must respond to Airdrop with a message with event of type `EXTRACTION_DATA_PROGRESS`, -together with an optional progress estimate and relevant artifacts -when it extracts some data and the maximum Airdrop snap-in runtime (12 minutes) has been reached. +The snap-in must respond to Airdrop with a message with event type of `EXTRACTION_DATA_PROGRESS`, +together with an optional progress estimate when the maximum Airdrop snap-in runtime (13 minutes) has been reached. If the extraction has been rate-limited by the external system and back-off is required, the snap-in -must respond to Airdrop with a message with event of type `EXTRACTION_DATA_DELAY` and specifying -back-off time with `delay` attribute. +must respond to Airdrop with a message with event type `EXTRACTION_DATA_DELAY` and specifying +back-off time with `delay` attribute (in seconds as an integer). -In both cases, Airdrop starts the snap-in with a message with event of type `EXTRACTION_DATA_CONTINUE`. -The restarting is immediate (in case of `EXTRACTION_DATA_PROGRESS`) or delayed -(in case of `EXTRACTION_DATA_DELAY`). +In both cases, Airdrop starts the snap-in with a message with event type `EXTRACTION_DATA_CONTINUE`. +In case of `EXTRACTION_DATA_PROGRESS` the restarting is immediate, +meanwhile in case of `EXTRACTION_DATA_DELAY` the restarting is delayed for the given number of seconds. -Once the data extraction is done, the snap-in must respond to Airdrop with a message with event of -type `EXTRACTION_DATA_DONE`. +Once the data extraction is done, the snap-in must respond to Airdrop with a message with event type `EXTRACTION_DATA_DONE`. If data extraction failed in any moment of extraction, the snap-in must respond to Airdrop with a -message with event of type `EXTRACTION_DATA_ERROR`. +message with event type `EXTRACTION_DATA_ERROR`. ## Implementation Data extraction should be implemented in the [data-extraction.ts](https://github.com/devrev/airdrop-template/blob/main/code/src/functions/extraction/workers/data-extraction.ts) file. -During the data extraction phase, the snap-in uploads batches of extracted items (with tooling from `@devrev/adaas-sdk`). +The snap-in must respond to Airdrop with a message, that signals either success, a delay, progress, or an error. + +```typescript Success +await adapter.emit(ExtractorEventType.ExtractionDataDone); +``` + +```typescript Delay +await adapter.emit(ExtractorEventType.ExtractionDataDelay, { + delay: "30", +}); +``` + +```typescript Progress +await adapter.emit(ExtractorEventType.ExtractionDataProgress); +``` + +```typescript Error +await adapter.emit(ExtractorEventType.ExtractionDataError, { + error: { + message: "Failed to extract data.", + }, +}); +``` + +The snap-in must always emit a single message. + +### Extracting and storing the data + +The SDK library includes a repository system for handling extracted items. +Each item type, such as users, tasks, or issues, has its own repository. +These are defined in the `repos` array as `itemType`. +The `itemType` name should match the `record_type` specified in the provided metadata. + +```typescript +const repos = [ + { + itemType: 'todos', + }, + { + itemType: 'users', + }, + { + itemType: 'attachments', + }, +]; +``` + +The `initializeRepos` function initializes the repositories and should be the first step when the process begins. + +```typescript +processTask({ + task: async ({ adapter }) => { + adapter.initializeRepos(repos); + // ... + }, + onTimeout: async ({ adapter }) => { + // ... + }, +}); +``` + +After initialization of repositories using `initializeRepos`, +items should be then retrieved from the external system and stored in the correct repository by calling the `push` function. + +```typescript +await adapter.getRepo('users')?.push(items); +``` + +Behind the scenes, the SDK library stores items pushed to the repository and uploads them in batches to the Airdrop platform. -Each artifact is submitted with an `item_type`, defining a separate domain object from the -external system and matching the `record_type` in the provided metadata. +### Data normalization -Extracted data must be normalized: +Extracted data must be normalized to fit the domain metadata defined in the `external-domain-metadata.json` file. +More details on this process are provided in the [Metadata extraction](/public/snapin-development/adaas/metadata-extraction) section. + +Normalization rules: - Null values: All fields without a value should either be omitted or set to null. For example, if an external system provides values such as "", –1 for missing values, @@ -52,8 +125,29 @@ Extracted data must be normalized: - Number fields must be valid JSON numbers (not strings). - Multiselect fields must be provided as an array (not CSV). -Each line of the file contains an `id` and the optional `created_date` and `modified_date` fields -in the beginning of the record. +Extracted items are automatically normalized when pushed to the `repo` if a normalization function is provided under the `normalize` key in the repo object. + +```typescript +const repos = [ + { + itemType: 'todos', + normalize: normalizeTodo, + }, + { + itemType: 'users', + normalize: normalizeUser, + }, + { + itemType: 'attachments', + normalize: normalizeAttachment, + }, +]; +``` + +For examples of normalization functions, refer to the [data-normalization.ts](https://github.com/devrev/airdrop-template/blob/main/code/src/functions/external-system/data-normalization.ts) file in the starter template. + +Each line of the file contains `id`, `created_date`, and `modified_date` fields +in the beginning of the record. These fields are required. All other fields are contained within the `data` attribute. ```json {2-4} @@ -67,7 +161,30 @@ All other fields are contained within the `data` attribute. "owner": "A3A", "rca": null, "severity": "fatal", - "summary": "Lorem ipsum" + "summary": "Lorem ipsum", + } +} +``` + +If the item you are normalizing is a work item (a ticket, task, issue, or similar), +it should also contain the `item_url_field` within the `data` attribute. +This field should be assigned a URL that points to the item in the external system. +This link is visible in the airdropped item in the DevRev app, +helping users to easily locate the item in the external system. + +```json {12} +{ + "id": "2102e01F", + "created_date": "1972-03-29T22:04:47+01:00", + "modified_date": "1970-01-01T01:00:04+01:00", + "data": { + "actual_close_date": "1970-01-01T02:33:18+01:00", + "creator": "b8", + "owner": "A3A", + "rca": null, + "severity": "fatal", + "summary": "Lorem ipsum", + "item_url_field": "https://external-system.com/issue/123" } } ``` @@ -88,12 +205,15 @@ echo '{}' | chef-cli fuzz-extracted -r issue -m external_domain_metadata.json > ## State handling -Since each snap-in invocation is a separate runtime instance (with a maximum execution time of 12 minutes), -it does not know what has been previously accomplished or how many records have already been extracted. -To enable information passing between invocations and runs, support has been added for saving a limited amount -of data as the snap-in `state`. Snap-in `state` persists between phases in one sync run as well as between multiple sync runs. +To enable information passing between invocations and runs, a limited amount of data can be saved as the snap-in `state`. +Snap-in `state` persists between phases in one sync run as well as between multiple sync runs. + You can access the `state` through SDK's `adapter` object. +```typescript +adapter.state['users'].completed = true; +``` + A snap-in must consult its state to obtain information on when the last successful forward sync started. - The snap-in's `state` is loaded at the start of each invocation and saved at its end. @@ -105,7 +225,15 @@ Effective use of the state and breaking down the problem into smaller chunks are The snap-in starter template contains an [example](https://github.com/devrev/airdrop-template/blob/main/code/src/functions/extraction/index.ts) of a simple state. Adding more data to the state can help with pagination and rate limiting by saving the point at which extraction was left off. -To test the state in development, you can decrease the timeout between snap-in invocations. +```typescript +export const initialExtractorState: ExtractorState = { + todos: { completed: false }, + users: { completed: false }, + attachments: { completed: false }, +}; +``` + +To test the state during snap-in development, you can pass in the option to decrease the timeout between snap-in invocations. ```typescript await spawn({ diff --git a/fern/docs/pages/airdrop/external-sync-units-extraction.mdx b/fern/docs/pages/airdrop/external-sync-units-extraction.mdx index e90d370d..df816bd5 100644 --- a/fern/docs/pages/airdrop/external-sync-units-extraction.mdx +++ b/fern/docs/pages/airdrop/external-sync-units-extraction.mdx @@ -9,7 +9,14 @@ sync units that it can extract from the external system API and send it to Airdr External sync unit extraction is executed only during the initial import. -### Implementation +## Triggering event + +Airdrop starts the external sync unit extraction by sending a message with the event type `EXTRACTION_EXTERNAL_SYNC_UNITS_START`. + +The snap-in must reply to Airdrop with an `EXTRACTION_EXTERNAL_SYNC_UNITS_DONE` message when finished, +or `EXTRACTION_EXTERNAL_SYNC_UNITS_ERROR` if an error occurs. + +## Implementation This phase should be implemented in the [`external-sync-units-extraction.ts`](https://github.com/devrev/airdrop-template/blob/main/code/src/functions/extraction/workers/external-sync-units-extraction.ts) file. @@ -52,4 +59,6 @@ await adapter.emit(ExtractorEventType.ExtractionExternalSyncUnitsError, { }); ``` +**The snap-in must always emit a single message.** + To test your changes, start a new airdrop in the DevRev App. If external sync units extraction is successful, you should be prompted to choose an external sync unit from the list. diff --git a/fern/docs/pages/airdrop/extraction-phases.mdx b/fern/docs/pages/airdrop/extraction-phases.mdx index 87bc9525..97b60916 100644 --- a/fern/docs/pages/airdrop/extraction-phases.mdx +++ b/fern/docs/pages/airdrop/extraction-phases.mdx @@ -1,24 +1,30 @@ Each snap-in must handle all the phases of Airdrop extraction. In a snap-in, you typically define a run function that iterates over events and invokes workers per extraction phase. -The SDK library exports `processTask` to structure the work within each phase, and `onTimeout` function -to handle timeouts. - The Airdrop snap-in extraction lifecycle consists of four phases: -* External sync units extraction +* External sync units extraction (only for initial sync) * Metadata extraction * Data extraction * Attachments extraction Each phase is defined in a separate file and is responsible for fetching the respective data. -The SDK library provides a repository management system to handle artifacts in batches. -The `initializeRepos` function initializes the repositories, and the `push` function uploads the -artifacts to the repositories. The `postState` function is used to post the state of the extraction task. + + Snap-in development is an iterative process. + It typically begins with retrieving some data from the external system. + The next step involves crafting an initial version of the external domain metadata and validating it through chef-cli. + This metadata is used to prepare the initial domain mapping and checking for any possible issues. + API calls to the external system are then corrected to fetch the missing data. + Start by working with one item type (we recommend starting with users), and once it maps well to DevRev objects and imports as desired, proceed with other item types. + + +The SDK library exports a `processTask` function, which takes an object parameter with two keys: +* `task`: a function that implements the functionality for the given phase. +* `onTimeout`: a function that handles timeouts, typically by simply emitting a message to the Airdrop platform. State management is crucial for snap-ins to maintain the state of the extraction task. -The `postState` function is used to post the state of the extraction task. -The state is stored in the adapter and can be retrieved using the `adapter.state` property. +State is saved to the Airdrop backend by calling the `postState` function. +During the extraction the state is stored in the adapter and can be retrieved using the `adapter.state` property. ```typescript import { AirdropEvent, EventType, spawn } from "@devrev/ts-adaas"; diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index 83983136..7d8efe9b 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -30,15 +30,18 @@ consider gathering the following information: - **Error handling**: Learn about error response formats and codes. Knowing this helps in handling errors and exceptions in your integration. -## Terminology +## Basic concepts ### Sync unit -A _sync unit_ is one self encompassing unit of data that is synced to an external system. Examples: +A _sync unit_ is one self-encompassing unit of data that is synced to an external system. For example: - A project in Jira. - An account in SalesForce. - An organization Zendesk. +In Jira, users often have multiple projects. Each project acts as an individual sync unit. +In contrast, Zendesk operates with a single large pool of tickets and agents. Here, the entire Zendesk instance can be synced in a single airdrop. + ### Sync run Airdrop extractions are done in _sync runs_. @@ -61,13 +64,13 @@ An **extractor** function in the snap-in is responsible for extracting data from A _reverse sync_ is a sync run from DevRev to an external system. It uses a **loader** function, to create or update data in the external system. -### Initial import +### Initial sync -An _initial import_ is the first import of data from the external system to DevRev. +The first sync is called the _initial sync_. It is triggered manually by the end user in DevRev's **Airdrops** UI. -In initial import all data needs to be extracted to create a baseline (while in incremental runs only -updated objects need to be extracted). +During the initial sync, all data from the external sync unit is extracted from the external system and loaded into DevRev. +This process typically involves a large import and may take some time. An _initial import_ consists of the following phases: @@ -76,16 +79,13 @@ An _initial import_ consists of the following phases: 3. Data extraction 4. Attachments extraction -### 1-way (incremental) sync +### 1-way (incremental sync) A _1-way sync_ (or _incremental sync_) refers to any extraction after the initial sync run has been successfully completed. -An extractor extracts data that was created or updated in the external system after the start -of the latest successful forward sync, including any changes that occurred during the forward sync, -but were not picked up by it. -A snap-in must consult its state to get information on when the last successful forward sync started. -Airdrop snap-ins must maintain their own states that persists between phases in a sync run, -as well as between sync runs. +An extractor extracts data that was created or updated in the external system after the start +of the latest successful forward sync. +This includes any changes that happened after the start of the previous sync, but were not picked up by it. A 1-way sync consists of the following phases: @@ -93,4 +93,12 @@ A 1-way sync consists of the following phases: 2. Data extraction 3. Attachments extraction -A 1-way sync extracts only the domain objects updated or created since the previous successful sync run. +### 2-way sync + +A _2-way sync_ is a reverse sync. +The loader receives information about changes in DevRev since the last successful reverse sync and updates the data in the external system. + +A 2-way sync consists of the following phases: + +1. Data loading +2. Attachments loading \ No newline at end of file diff --git a/fern/docs/pages/airdrop/initial-domain-mapping.mdx b/fern/docs/pages/airdrop/initial-domain-mapping.mdx index 262aced7..d880ab83 100644 --- a/fern/docs/pages/airdrop/initial-domain-mapping.mdx +++ b/fern/docs/pages/airdrop/initial-domain-mapping.mdx @@ -1,8 +1,11 @@ Initial domain mapping is a process that establishes relationships between -external data schemas and DevRev's native record types. This mapping is configured -once and then becomes available to all users of your integration, +external data schemas and DevRev's native record types. +This mapping is configured once and then becomes available to all users of your snap-in, allowing them to import data while maintaining semantic meaning from their source systems. +The initial domain mapping is installed with your snap-in. +The extractor automatically triggers a function to upload these mappings to the Airdrop system. + ## Chef-cli initial domain mapping setup ### Prerequisites diff --git a/fern/docs/pages/airdrop/load-attachments.mdx b/fern/docs/pages/airdrop/load-attachments.mdx index 4a1ac311..d1e2e0f4 100644 --- a/fern/docs/pages/airdrop/load-attachments.mdx +++ b/fern/docs/pages/airdrop/load-attachments.mdx @@ -2,7 +2,14 @@ In the load attachments phase, the snap-in saves each attachment to the external ## Triggering event -Airdrop initiates the load attachments phase by starting the snap-in with a message containing an event of type `START_LOADING_ATTACHMENTS`. +Airdrop initiates data loading by sending a message with the event type `START_LOADING_ATTACHMENTS` to the snap-in. + +If the maximum Airdrop snap-in runtime (13 minutes) has been reached, +the snap-in must respond to Airdrop with a message with event type of `ATTACHMENT_LOADING_PROGRESS`, together with an optional progress estimate. + +In case of `ATTACHMENT_LOADING_PROGRESS`, Airdrop starts the snap-in with a message with event type `CONTINUE_LOADING_ATTACHMENTS`. + +Once the data extraction is done, the snap-in must respond to Airdrop with a message with event type `ATTACHMENT_LOADING_DONE`. ## Implementation @@ -10,6 +17,8 @@ This phase is defined in [load-attachments.ts](https://github.com/devrev/airdrop The loading process involves providing the `create` function to add attachments to the external system. The `create` function is responsible for making API calls to the external system to create the attachments, as well as handling errors and the external system's rate limiting. The function should return the `id` and optionally `modifiedDate` of the record in the external system or indicates a rate-limiting back-off or logs errors if the attachment could not be created. +**The snap-in must always emit a single message.** + ```typescript processTask({ task: async ({ adapter }) => { diff --git a/fern/docs/pages/airdrop/load-data.mdx b/fern/docs/pages/airdrop/load-data.mdx index 5e18d25c..f7b55b90 100644 --- a/fern/docs/pages/airdrop/load-data.mdx +++ b/fern/docs/pages/airdrop/load-data.mdx @@ -2,7 +2,14 @@ The load data phase manages the creation and updating of items in the external s ## Triggering event -Airdrop initiates data loading by starting the snap-in with a message containing an event of type `START_LOADING_DATA`. +Airdrop initiates data loading by sending a message with the event type `START_LOADING_DATA` to the snap-in. + +If the maximum Airdrop snap-in runtime (13 minutes) has been reached, +the snap-in must respond to Airdrop with a message with event type of `DATA_LOADING_PROGRESS`, together with an optional progress estimate. + +In case of `DATA_LOADING_PROGRESS`, Airdrop starts the snap-in with a message with event type `CONTINUE_LOADING_DATA`. + +Once the data extraction is done, the snap-in must respond to Airdrop with a message with event type `DATA_LOADING_DONE`. ## Implementation @@ -12,7 +19,7 @@ Loading is performed by providing a list of item types to load (`itemTypesToLoad Each item type must provide `create` and `update` functions, which handle the denormalization of records to the schema of the external system and facilitate HTTP calls to the external system. Both loading functions must manage rate limiting for the external system and handle errors. The `create` and `update` functions should return an `id` of the record in the external system and optionally also `modifiedDate`. If a record cannot be created or updated, they indicate the rate-limiting offset or errors. -Same as with extraction, the SDK library exports the `processTask` function to structure the work within each phase and the `onTimeout` function to handle timeouts. +The snap-in must always emit a single message. ```typescript processTask({ diff --git a/fern/docs/pages/airdrop/loading-phases.mdx b/fern/docs/pages/airdrop/loading-phases.mdx index 495857c9..41909b5b 100644 --- a/fern/docs/pages/airdrop/loading-phases.mdx +++ b/fern/docs/pages/airdrop/loading-phases.mdx @@ -46,7 +46,12 @@ export default run; ## State handling -Loading phases run as separate runtime instances, similar to extraction phases, with a maximum execution time of 12 minutes. +Loading phases run as separate runtime instances, similar to extraction phases, with a maximum execution time of 13 minutes. These phases share a `state`, defined in the `LoaderState` interface. It is important to note that the loader state is separate from the extractor state. + Access to the `state` is available through the SDK's `adapter` object. + +## Creating items in DevRev + +To create an item in DevRev and sync it with the external system, start by creating an item with a **subtype** that was established during the initial sync. After selecting the subtype, fill out the necessary details for the item. \ No newline at end of file diff --git a/fern/docs/pages/airdrop/local-development.mdx b/fern/docs/pages/airdrop/local-development.mdx index 77b04257..6610b944 100644 --- a/fern/docs/pages/airdrop/local-development.mdx +++ b/fern/docs/pages/airdrop/local-development.mdx @@ -9,7 +9,7 @@ For easier development you can run your Airdrop snap-in locally and receive logs ## Run the template -DevRev provides a starter template, which you can run and test out right away. +DevRev offers a starter Airdrop snap-in template that is ready for immediate use and testing. 1. Create a new repository: - Create a new repository from this template by clicking the "Use this template" button in the upper right corner and then "Create a new repository". @@ -46,6 +46,8 @@ DevRev provides a starter template, which you can run and test out right away. devrev snap_in activate ``` +# Initial sync + Now that you have a running snap-in, you can start an airdrop. Go to DevRev app and click **Airdrops** -> **Start Airdrop** -> **Your snap-in**. diff --git a/fern/docs/pages/airdrop/manifest.mdx b/fern/docs/pages/airdrop/manifest.mdx index cd87fd13..6af49f7c 100644 --- a/fern/docs/pages/airdrop/manifest.mdx +++ b/fern/docs/pages/airdrop/manifest.mdx @@ -46,18 +46,18 @@ Ensure that `extractor_function` and `loader_function` names correspond with tho ## Establish a connection to the external system -_Keyrings_ are a collection of authentication information, used by a snap-in to authenticate to the external system in API calls. This can include a key (for example, a PAT token or API key), its type, the organization ID for which a key is valid, and in some cases the organization name. +_Keyrings_ provide a secure way to store and manage credentials within your DevRev snap-in. -Keyrings provide a secure way to store and manage credentials within your DevRev snap-in. -This eliminates the need to expose sensitive information like passwords or access tokens directly -within your code or configuration files, enhancing overall security. -They also provide a valid token by abstracting OAuth token renewal from the end user. +Keyrings are a collection of authentication information, used by a snap-in to authenticate to the external system in API calls. +This can include a key (for example, a PAT token or API key), its type and the organization ID for which a key is valid. -They are called **Connections** in the DevRev app. +This eliminates the need to expose sensitive information like passwords or access tokens directly within your code or configuration files. They also provide a valid token by abstracting OAuth token renewal from the end user, so less work is needed on the developer's side. + +Keyrings are called **Connections** in the DevRev app. ### Configure a keyring -Keyrings are configured in the `manifest.yaml` by configuring a `keyring_type`, like in the [example](https://github.com/devrev/airdrop-template/blob/main/manifest.yaml). +Keyrings are configured in the `manifest.yaml` by configuring a `keyring_type`, like in the [example](https://github.com/devrev/airdrop-template/blob/main/manifest.yaml): ```yaml keyring_types: @@ -67,8 +67,7 @@ keyring_types: # The kind field specifies the type of keyring. kind: <"secret"/"oauth2"> # is_subdomain field specifies whether the keyring contains a subdomain. - # Enabling this field allows the keyring to get the subdomain from the user during creation. - # This is useful when the keyring requires a subdomain as part of the configuration. + # Enabling this field allows the keyring to get the subdomain from the user during keyring creation. # Default is false. is_subdomain: # Name of the external system you are importing from. @@ -96,7 +95,7 @@ keyring_types: # Optional: query parameters to be included in the verification request. query_params: : # optional: query parameters to be included in the verification request. - # Fetching Organization Data: This allows you to retrieve additional information about the user's organization. + # The organization_data section takes care of fetching organization data and is required if is_subdomain option is false. organization_data: type: "config" # The URL to which the request is sent to fetch organization data. @@ -106,7 +105,16 @@ keyring_types: headers: : # The jq filter used to extract the organization data from the response. + # It should provide an object with id and name, depending on what the external system returns. + # For example "{id: .data[0].id, name: .data[0].name }". response_jq: ``` +There are some options to consider: + +* `kind` + The `kind` option can be either "secret" or "oauth2". The "secret" option is intended for storing various tokens, such as a PAT token. Use of OAuth2 is encouraged when possible. More information is available for [secret](/public/snapin-development/references/keyrings/secret-configuration) and [oauth2](/oauth-configuration). + +* `is_subdomain` + The `is_subdomain` field relates to the API endpoints being called. When the endpoints for fetching data from an external system include a slug representing the organization—such as for example `https://subdomain.freshdesk.com/api/v2/tickets`—set this key to "true". In this scenario, users creating a new connection are prompted to insert the subdomain. -You can find more information about keyrings and keyring types [here](/snapin-development/references/keyrings/keyring-intro). \ No newline at end of file + If no subdomain is present in the endpoint URL, set this key to "false". In this case, provide the `organization_data` part of the configuration. Specify the endpoint in the `url` field to fetch organization data. Users creating a new connection are prompted to select the organization from a list of options, as retrieved from the `organization_data.url` value. \ No newline at end of file diff --git a/fern/docs/pages/airdrop/metadata-extraction.mdx b/fern/docs/pages/airdrop/metadata-extraction.mdx index 9a364202..f191f11c 100644 --- a/fern/docs/pages/airdrop/metadata-extraction.mdx +++ b/fern/docs/pages/airdrop/metadata-extraction.mdx @@ -39,6 +39,8 @@ processTask({ }); ``` +The snap-in must always emit a single message. + ## Validating metadata The extraction function of the snap-in must provide a valid metadata file. @@ -243,7 +245,7 @@ be changed by the end user at any time, such as mandatory fields or custom field A good practice is to retrieve the set of possible values for all enum fields from the external system's APIs in each sync run. You can mark specific enum values as deprecated using the `is_deprecated` property. - `ID` (primary key) of the record, `created_date`, and `modified_date` must not be declared. + **`ID` (primary key) of the record, `created_date`, and `modified_date` must not be declared.** Example: diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index 66b1cb10..94476db7 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -1,11 +1,14 @@ -_Airdrop_ is DevRev’s solution to migrate data. It allows our customers to bring in their existing -data from external systems to DevRev, export data back to external systems, and keep data in sync -between DevRev and the external systems. +_Airdrop_ is DevRev’s solution for migrating data between DevRev and an external system. +It allows our users to import their existing data from an external system, +export any changes in DevRev back to the external system, and keep data in sync. DevRev provides a range of snap-ins available through the DevRev marketplace for integration with -various external systems like Jira and Zendesk. Sometimes, a specific snap-in may not be available -for a given external system. This is why DevRev enables creation of external connectors (extractors -and loaders) to other developers, allowing them to integrate with DevRev’s Airdrop and develop their -own Airdrop snap-ins. +various external systems like Jira and Zendesk. +These DevRev provided snap-ins are also called _native snap-ins_. + +Sometimes, a specific native snap-in may not be available for a given external system. +This is why DevRev enables creation of external snap-ins by third-party developers, +allowing them to integrate with DevRev’s Airdrop and develop their own Airdrop snap-ins. +These community-developed snap-ins can offer the same functionality as native snap-ins. You can learn more about Airdrop in the [general documentation](https://docs.devrev.ai/import#airdrop-features). diff --git a/fern/docs/pages/airdrop/publish-to-marketplace.mdx b/fern/docs/pages/airdrop/publish-to-marketplace.mdx index b98d3156..72cf010e 100644 --- a/fern/docs/pages/airdrop/publish-to-marketplace.mdx +++ b/fern/docs/pages/airdrop/publish-to-marketplace.mdx @@ -1,4 +1,6 @@ -When your snap-in is ready for use by you or other organizations, publish it to the DevRev marketplace. Before starting the publication process, ensure your snap-in is deployed to your organization. If you need a refresher, refer to the [instructions](/snapin-development/adaas/development-guide/getting-started#publishing-the-snap-in) for deploying your snap-in. +When your snap-in is ready for use by you or other organizations, you can publish it to the DevRev marketplace. +Before starting the publication process, make sure your snap-in is deployed to your organization. +If you need a refresher, refer to the [instructions](/snapin-development/adaas/development-guide/deploy-to-organization) for deploying your snap-in. ### 1. Create marketplace listing @@ -38,4 +40,4 @@ To make sure that marketplace item was published you can retrieve it using its I devrev marketplace_items show [marketplace_item_id] | jq '{name: .name, id: .id, state: .state}' ``` -Learn more about the markeplace [here](/snapin-development/marketplace-listings). +Learn more about the markeplace [here](/snapin-development/marketplace-listings). \ No newline at end of file diff --git a/fern/docs/pages/airdrop/snap-in-template.mdx b/fern/docs/pages/airdrop/snap-in-template.mdx index 6425f9d9..1df32f3a 100644 --- a/fern/docs/pages/airdrop/snap-in-template.mdx +++ b/fern/docs/pages/airdrop/snap-in-template.mdx @@ -1,9 +1,11 @@ -Airdrop snap-in starter template is available [here](https://github.com/devrev/airdrop-template). +The starter Airdrop snap-in template is available [here](https://github.com/devrev/airdrop-template). + +Starting with the template is recommended, as it includes all the necessary boilerplate and structure for a new project. The template makes use of **[DevRev's SDK library](https://www.npmjs.com/package/@devrev/ts-adaas)**, -which helps developers build snap-ins that integrate with DevRev’s Airdrop platform. -This library simplifies the workflow for handling data extraction and loading, event-driven actions, -state management, and artifact handling. +which provides abstraction over the DevRev’s Airdrop control protocol. +The SDK simplifies the workflow for data extraction and loading, handling events, +state management, and artifacts (batches of extracted items). It provides features such as: