-
Notifications
You must be signed in to change notification settings - Fork 460
LDVRecipes
A demonstration recipe for how to process a large amount of records in serial chunks using Queueables. The idea behind this recipe is that Queueables, in production, have no max-queue depth. Meaning that so long as you only enqueue one new queueable, it can keep cycling through until the entire data set is processed. This is useful for instance, when you want to process hundreds of thousands of records.
Note: You're not able to re-enqueue within a test context, so the unit test for this code is limited to the same number of records as chunkSize below.
Note: This should be refactored to be an abstract class that you can extend named 'Ouroboros'. (Ouroboros = the snake eating it's own tail)
Group LDV Recipes
Implements
Queueable
private final chunkSizeInteger
private offsetIdId
private objectsToProcessList<ContentDocumentLink>
TESTVISIBLE
private static chunksExecutedInteger
No param constructor. Use for starting the chain.
public LDVRecipes()Constructor accepting an ID to use as an offset. Use this version to continue the chain.
public LDVRecipes(Id offsetId)| Name | Type | Description |
|---|---|---|
| offsetId | Id |
This method contains the 'what' happens to each chunk of records. Note, that this example doesn't actually do any processing. In a real-life use case you'd iterate over the records stored in this.objectsToProcess.
public void execute(System.QueueableContext queueableContext)| Name | Type | Description |
|---|---|---|
| queueableContext | System.QueueableContext |
void
Returns a 'cursor' - a set of records of size X from a given offset. Note: We originally intended to use OFFSET - the SOQL keyword, but discovered the max OFFSET size is 2000. This obviously won't work for large data volumes greater than 2000 so we switched to using the ID of the record. Since ID is an indexed field, this should also allow us to prevent full table scans even on the largest tables.
private List<ContentDocumentLink> getRecordsToProcess(Id offsetId)| Name | Type | Description |
|---|---|---|
| offsetId | Id | The offset ID is used to demarcate already processed |
| records. |
List<ContentDocumentLink>
private Boolean safeToReenqueue()Boolean