add ability to recognize new files by time stamp or size changes #24
Conversation
ewels
left a comment
ewels
left a comment
There was a problem hiding this comment.
Read-through of the code looks good. Docs need updating though, and ideally needs some testing in the wild.
|
It'd also be nice to have a tick box to choose whether to compare on just filename, or filename + meta. As folks on HPC sometimes have filesystems that can mess with last-modified dates. |
Adds detection mode option to poll-files node with two behaviors: - Name + metadata (default): Detects files as new when name, lastModified, size, or etag changes. Enables detection of overwritten/replaced files with same name (e.g., daily status files). - Name only: Only detects truly new filenames (original behavior) Also adds third output for deleted files, allowing flows to respond when files are removed from Data Links. Updates documentation with examples for both detection modes and deletion detection use cases. Co-Authored-By: FriederikeHanssen <[email protected]> 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
|
Added looks like this now: |
ewels
left a comment
ewels
left a comment
There was a problem hiding this comment.
LGTM, happy to merge once you've tested it and validated that it works as expected 👍🏻
| ### Monitor for status file updates | ||
|
|
||
| Use this pattern when a completion marker file is deposited with the same name but new timestamp: | ||
|
|
||
| 1. Set **Detection mode** to **Name + metadata** | ||
| 2. Set **Pattern**: `RTAComplete\.txt$` to match only the status file | ||
| 3. Connect output 2 to your workflow trigger | ||
| 4. Each time the file is deposited (even with the same name), the workflow will trigger | ||
|
|
There was a problem hiding this comment.
I think that this one is a bit obvious
| ### Monitor for status file updates | |
| Use this pattern when a completion marker file is deposited with the same name but new timestamp: | |
| 1. Set **Detection mode** to **Name + metadata** | |
| 2. Set **Pattern**: `RTAComplete\.txt$` to match only the status file | |
| 3. Connect output 2 to your workflow trigger | |
| 4. Each time the file is deposited (even with the same name), the workflow will trigger | |
| ### Monitor for status file updates | |
| Use this pattern when a completion marker file is deposited with the same name but new timestamp: | |
| 1. Set **Detection mode** to **Name + metadata** | |
| 2. Set **Pattern**: `RTAComplete\.txt$` to match only the status file | |
| 3. Connect output 2 to your workflow trigger | |
| 4. Each time the file is deposited (even with the same name), the workflow will trigger | |
|
We had a look at this, the detection seems to work when the file deletion / recreation has a poll event in between (so it detects first the deletion, then the new file) - this is working as before this PR. However, detection of file modification timestamps does not work. Looking at the debug log messages, my suspicion is that the Seqera Platform API does not return modified timestamps, and that Claude was being a little over-eager in writing this code, based purely on the assumption that the API might. So, possible that file modification may be impossible to detect without upstream changes. We can see. |
| } | ||
| // metadata mode (default) | ||
| const parts = [item.name]; | ||
| if (item.lastModified) parts.push(item.lastModified); |
There was a problem hiding this comment.
We think that item.lastModified was probably hallucinated.
|
Reimplemented the "deleted files" output in this PR: #27 |
|
Closing for now until we can show that the API really is able to return metadata about files like this. |
No description provided.