Skip to content

Commit 87b000d

Browse files
authored
Merge branch 'main' into feature/ISS-189649-new-api-mask-documentation
2 parents 6cf9d1c + bda6037 commit 87b000d

File tree

2 files changed

+61
-1
lines changed

2 files changed

+61
-1
lines changed

fern/docs/pages/airdrop/data-extraction.mdx

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -279,3 +279,57 @@ await spawn({
279279
```
280280

281281
This lets you simulate a soft timeout and validate that your worker shuts down.
282+
283+
## Time-scoped syncs
284+
285+
Time-scoped syncs allow using a custom timestamp to control the scope of data extraction. This capability enables more granular control over which data gets synchronized between external systems and DevRev.
286+
287+
Enable time-scoped syncs by first adding the capability to your manifest:
288+
289+
```yaml
290+
imports:
291+
- # slug and other import information ...
292+
capabilities:
293+
- TIME_SCOPED_SYNCS
294+
```
295+
296+
Your data extraction implementation must handle two optional parameters from the event's `EventContext`:
297+
298+
- **`extract_from`**: Timestamp in RFC3339 format indicating the starting point of extraction. This applies to both initial and incremental syncs.
299+
- **`reset_extract_from`**: A boolean flag for incremental syncs that determines whether data should be re-extracted.
300+
301+
The extraction logic depends on the sync type and parameter combination:
302+
303+
**Initial syncs**: Use `extract_from` as the starting timestamp for data extraction.
304+
305+
**Incremental syncs**:
306+
- If `reset_extract_from` is `true`: Start from `extract_from` if it is provided; otherwise, extract all data.
307+
- If `reset_extract_from` is `false` or not provided: Use the `adapter.state.lastSuccessfulSyncStarted` timestamp.
308+
309+
```typescript
310+
const { reset_extract_from, extract_from } = adapter.event.payload.event_context;
311+
312+
// The start of a new sync.
313+
if (adapter.event.payload.event_type === EventType.ExtractionDataStart) {
314+
315+
// Handle extract_from parameter for any sync type
316+
if (extract_from) {
317+
console.log(`Starting extraction from given timestamp: ${extract_from}.`);
318+
// ...
319+
}
320+
321+
// Handle incremental sync logic
322+
if (adapter.event.payload.event_context.mode === SyncMode.INCREMENTAL) {
323+
324+
// If `reset_extract_from` is true, the extraction should start from extract_from (if provided)
325+
// or from the beginning (if extract_from is not provided).
326+
if (reset_extract_from) {
327+
console.log(`reset_extract_from is true. Starting extraction from provided timestamp (${extract_from}) or from the beginning.`);
328+
// If reset_extract_from is false or not provided, it should use the lastSuccessfulSyncStarted timestamp to get only the new or updated data.
329+
} else {
330+
console.log(`Starting extraction from lastSuccessfulSyncStarted: (${adapter.state.lastSuccessfulSyncStarted}).`);
331+
// ...
332+
}
333+
}
334+
}
335+
```

fern/docs/pages/airdrop/getting-started.mdx

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,4 +100,10 @@ The loader receives information about changes in DevRev since the last successfu
100100
A 2-way sync consists of the following phases:
101101

102102
1. Data loading
103-
2. Attachments loading
103+
2. Attachments loading
104+
105+
### Time-scoped sync
106+
107+
Normally, an initial sync extracts all data from an organization, and an incremental sync only extracts changes since the last successful sync. A _time-scoped sync_, on the other hand, allows for the extraction of data beginning from a custom timestamp.
108+
109+
This approach is useful for initially testing the integration with a smaller data set—such as data from the last month—before conducting a full import of all available data.

0 commit comments

Comments
 (0)