diff --git a/CHANGELOG.md b/CHANGELOG.md index bccbb5b..898eeea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,15 @@ # Changelog +## [1.5.0] - 2025-12-18 + +- 💥 Breaking change: Poll files node + - Added `pollIntervalSeconds` field to output payload alongside `nextPoll` + - "Every poll" output port is now hidden by default, with a checkbox option to enable it + - Added new "Deleted results" output that fires when files are removed from the Data Link + +**Warning**: The change in the available ports on this node could affect your workflows. +Please update with care - make sure that the correct output ports are still connected! + ## [1.4.1] - 2025-11-16 - Bugfix: Don't use absolute paths for internal API calls for config setup diff --git a/docs/img/poll_files_node_edit.png b/docs/img/poll_files_node_edit.png index ad1d615..2fe6df3 100644 Binary files a/docs/img/poll_files_node_edit.png and b/docs/img/poll_files_node_edit.png differ diff --git a/docs/seqera_nodes/poll_files.md b/docs/seqera_nodes/poll_files.md index be7c93c..074b1f3 100644 --- a/docs/seqera_nodes/poll_files.md +++ b/docs/seqera_nodes/poll_files.md @@ -1,8 +1,8 @@ # Poll files -**Periodically list a Seqera Data Explorer Data Link and emit messages when new objects appear.** +**Periodically list a Seqera Data Explorer Data Link and emit messages when objects are added or deleted.** -This node automatically monitors a Data Link for _changes_, making it perfect for event-driven workflows that trigger when new files are uploaded. +This node automatically monitors a Data Link for _changes_, making it perfect for event-driven workflows that trigger when files are uploaded or removed. !!! note @@ -15,52 +15,69 @@ This node automatically monitors a Data Link for _changes_, making it perfect fo ## Configuration -!!! info +This node works much like the [list files node](list_files.md), but instead of triggering when recieving a message input, it polls repeatedly and outputs events when it detects a change. + +!!! warning + + File addition / deletion is only detected across poll events. + This means that if you delete a file and create a new one with the same name + _inbetween_ two poll events, the change **will not be detected**. - This node works much like the [list files node](list_files.md), but instead of triggering when recieving a message input, it polls repeatedly and outputs events when it detects a change. + If this kind of activity is likely, make sure that you set the polling frequency + to be fast enough that the node will check the file listing between events + _(poll -> delete file -> poll -> add file -> poll)_. - **Seqera config**: Reference to the seqera-config node containing API credentials and default workspace settings. - **Node name**: Optional custom name for the node in the editor. - **Data Link name** (required): Display name of the Data Link. Supports autocomplete. +- **Return type** (default **files**): `files`, `folders` or `all`. +- **Poll frequency** (default **15 min**): Interval between polls. +- **Every poll / Show output port** (default **off**): When enabled, adds an extra output that emits all files on every poll. - **Base path**: Path within the Data Link to start from. - **Prefix**: Prefix filter applied to both files and folders. - **Pattern**: Regular-expression filter applied to files after the prefix filter. -- **Return type** (default **files**): `files`, `folders` or `all`. - **Max results** (default **100**): Maximum number of objects to return per poll. - **Depth** (default **0**): Folder recursion depth. -- **Poll frequency** (default **15 min**): Interval between polls. - **Workspace ID**: Override the workspace ID from the Config node. All properties work the same as the [list files](list_files.md) node, plus automatic polling. -## Outputs (two) +## Outputs + +By default, the node has two outputs: + +1. **New results** – Emitted only when one or more _new_ objects are detected since the last poll. +2. **Deleted results** – Emitted only when one or more objects are _deleted_ since the last poll. -The node has two outputs that fire at different times: +When **Every poll / Show output port** is enabled, the node has three outputs: 1. **All results** – Emitted every poll with the full, filtered list of files. 2. **New results** – Emitted only when one or more _new_ objects are detected since the last poll. +3. **Deleted results** – Emitted only when one or more objects are _deleted_ since the last poll. -Both messages include the same properties: +All outputs include the same properties: - `msg.payload.files` – Array of file objects from the API. - `msg.payload.resourceType`, `msg.payload.resourceRef`, `msg.payload.provider` – Data Link metadata. - `msg.files` – Convenience array of fully-qualified object names (strings). - `msg.payload.nextPoll` (only on **All results** output) – ISO timestamp of the next scheduled poll. +- `msg.payload.pollIntervalSeconds` (only on **All results** output) – Poll interval duration in seconds. -## How new files are detected +## How changes are detected The node tracks seen files in its context storage. On each poll: 1. Fetch the current list of files from the Data Link 2. Compare against the list from the previous poll -3. If new files are found, emit them on output 2 -4. Update the stored list for the next comparison +3. If new files are found, emit them on the "New results" output +4. If files are missing (deleted), emit them on the "Deleted results" output +5. Update the stored list for the next comparison -The comparison is based on the full file path. Files that are deleted and re-uploaded will be detected as "new". +The comparison is based on the full file path. Files that are deleted and re-uploaded will be detected as "deleted" then "new" on subsequent polls. !!! info - The very first poll after the node is created sees everything as new and is handled as a special case. It does not output new results. + The very first poll after the node is created sees everything as new and is handled as a special case. It does not output new or deleted results. ## Required permissions @@ -74,7 +91,7 @@ See the [configuration documentation](configuration.md#required-token-permission 1. Add a **poll-files** node and configure the Data Link 2. Set **pollFrequency** to your desired interval (e.g., `5:00` for 5 minutes) -3. Connect output 2 (New results) to a **workflow-launch** node +3. Connect the output (New results) to a **workflow-launch** node 4. Configure the launch node to use the file paths from `msg.files` 5. Deploy @@ -83,12 +100,12 @@ Now every time a new file appears in the Data Link, a workflow will automaticall ### Trigger only on specific file types 1. Set **pattern**: `.*\.bam$` to only detect BAM files -2. Connect output 2 to your processing logic +2. Connect the output to your processing logic 3. The node will only emit when new BAM files appear ## Notes -- The first poll after deployment/restart does **not** emit to the "New results" output (it initializes the tracking state) +- The first poll after deployment/restart does **not** emit to the "New results" or "Deleted results" outputs (it initializes the tracking state) - The tracking is reset on each Node-RED restart or flow redeployment - Very frequent polling (< 30 seconds) may impact API rate limits - Custom message properties are preserved in outputs (e.g., `msg._context`) diff --git a/examples/02 - Launch on file upload.json b/examples/02 - Launch on file upload.json index 6c8dd31..a8200a1 100644 --- a/examples/02 - Launch on file upload.json +++ b/examples/02 - Launch on file upload.json @@ -31,7 +31,7 @@ "returnType": "files", "x": 100, "y": 100, - "wires": [[], ["2b7682159053d63c"]] + "wires": [["2b7682159053d63c"], []] }, { "id": "2b7682159053d63c", @@ -127,5 +127,13 @@ "x": 570, "y": 200, "wires": [[], [], []] + }, + { + "id": "2801ba6dc68b2335", + "type": "global-config", + "env": [], + "modules": { + "@seqera/node-red-seqera": "1.4.1" + } } ] diff --git a/nodes/datalink-poll.html b/nodes/datalink-poll.html index c2b7075..356a612 100644 --- a/nodes/datalink-poll.html +++ b/nodes/datalink-poll.html @@ -9,6 +9,20 @@ + +
+ + +
+
+ + +
+
@@ -20,20 +34,23 @@
- -
- - -
-
- - + +
+ +
@@ -66,30 +83,37 @@ ### Inputs -: pollFrequency (string) : Poll frequency (default `15 minutes`). Can be configured in seconds, minutes, hours, or days. : dataLinkName (string) : The name of the data explorer link. +: returnType (string) : Select whether to return files, folders or everything. +: pollFrequency (string) : Poll frequency (default `15 minutes`). Can be configured in seconds, minutes, hours, or days. +: outputAllPolls (boolean) : Show the "every poll" output port. Emits an output file listing on every poll event, irrespective of changes (disabled by default). : basePath (string) : Path within the data link to start browsing. Leave blank for the root. : prefix (string) : Optional prefix filter for results (applies to folders and files) : pattern (string) : Optional regex pattern filter for results (applies to files only) -: returnType (string) : Select whether to return files, folders or everything. : maxResults (number) : Maximum number of results to return (default 100). +: depth (number) : Folder recursion depth (default 0). : workspaceId (string) : Override the workspace ID from the config node. -All inputs support msg._, flow._, global.\*, env, or JSONata expressions via the **typedInput**. +All inputs support `msg`, `flow`, `global`, `env`, or JSONata expressions via the **typedInput**. ### Outputs -The node has two outputs: +The node has two or three outputs depending on configuration: -1. All results on every poll. -2. New objects since the previous poll (nothing sent if no new objects). +1. **All results** (optional, disabled by default) - Emitted every poll with the full list of files. +2. **New results** - Emitted only when new objects are detected since the previous poll. +3. **Deleted results** - Emitted only when objects are deleted since the previous poll. -Both outputs have the following properties: +All outputs have the following properties: -: payload (array) : Fle information aggregated from the API (array of objects). +: payload (array) : File information aggregated from the API (array of objects). : files (array) : File names (array of strings). -All typed-input fields are identical to the _List files_ node with the addition of **poll frequency**. +The "All results" output also includes: + +: payload.nextPoll (string) : ISO timestamp of the next scheduled poll. +: payload.pollIntervalSeconds (number) : Poll interval duration in seconds. +