Process jobs based on ressource lock.

[clouedoc]

Hello to the people reading this message.

I currently have a use case where jobs must access shared ressources, but only one job can access the given ressource at a time. Jobs are assigned to a ressource (they can't use another one)

I made some nice images so that it's easier to understand the situation

Let me know if it isn't clear enough.

Solutions I've thought of

• Processing all jobs and looking up a shared mutex for antennas. Jobs would get delayed by X minutes if antenna not available. It looks like a bad solution for a number of reasons.

Dream solution

The best would be to have some way of specifying a "filter" function when instantiating the worker.
E.g.:

new Worker("queue-name", (job) => console.log('received job: ' job), {
  filter: async (job): boolean => isAntennaAvailable(job.data.antennaId)
})

[clouedoc]

I've seen someone talk about groupKey: #433 (reply in thread)

It looks like a nice solution, but I need to investigate it a bit to know if it's what I'm really looking for.

[clouedoc]

Filtered out jobs shall be delayed to get tested again by the filter in a few minutes (interval should be set by user).

[manast]

@clouedoc in that case, wouldn't it solve the requirement if it was possible to move the job to the delay set inside the job processor instead of letting it complete?

[clouedoc]

@manast yeah, delaying the job by X would be the best (to keep nice statistics about job completions/failures).

It would also be nice to have a "job delayed because of filter" event.

[manast]

Have you tried retries and custom backoffs? Maybe you can just throw from the job processor and then in the custom backoff determine how much to delay the job. If some feature is missing in custom backoffs I am willing to add it so that you can fullfil your requirements.

[clouedoc]

I did not try this, thanks for the hint. I'll tell you how it has gone from a developer experience point of view so that you can decide what to do for your project.