[Deferred] Periodic task scheduling

[rsamoilov]

Description

Rage::Deferred currently supports enqueuing one-off tasks and delayed tasks, but there is no built-in way to run tasks on a recurring schedule. Today, users who need periodic jobs (e.g. cleaning up expired records, refreshing caches, syncing data) have to reach for external tools like cron or a separate scheduling library.

The goal of this issue is to add native periodic task scheduling to Rage::Deferred, keeping it consistent with Rage's philosophy of collapsing backend concerns into a single runtime.

Proposed interface

# config/application.rb or config/environments/<environment>.rb
Rage.configure do
  config.deferred.schedule do
    every 1.hour, task: CleanupExpiredInvites
    every 1.minute, task: ResetCache
  end
end

Where CleanupExpiredInvites and ResetCache are standard Rage::Deferred::Task classes with a perform method.

Design considerations

These are open questions that should be addressed in the design proposal:

• First run timing. After boot, should the first run happen immediately or wait for the first interval to elapse?
• Leader election in multi-process deployments. If Rage is running multiple worker processes, only one process should own the scheduling responsibility to avoid duplicate task execution. Also, consider what happens when the leader process restarts — do timers reset?
• Overlap policy. What happens if a scheduled task is still running when the next interval fires?
• Arguments. Should every support passing arguments to the task's perform method?

Before You Start

This issue requires a design proposal before implementation. Please open a discussion or comment on this issue with a high-level plan that outlines:

1. The public API (confirming or adjusting the interface above).
2. How scheduling will be implemented (timer mechanism, which process owns it).
3. The leader election strategy for multi-process deployments.
4. How the edge cases above (overlap, first run, restarts) will be handled.

PRs without prior design discussion are unlikely to be accepted.

Tips

• Check the Deferred docs to understand how Rage::Deferred works, especially the WAL and task execution model.
• Look at how Rage::Cable selects a process to manage Redis subscriptions in multi-server setups.
• Check the architecture doc that shows how Rage's core components interact with each other and outlines the design principles.
• Feel free to ask any questions or request help in the comments below!

[ShashantNagpure]

Hello @rsamoilov . I have drafted a solution for this, this would be my first contribution for open source :D , could you please help and review it? If the design looks good, could I please assigned to work on it? Thanks

Design Proposal for Periodic scheduling for deferred tasks

Public API:

#   config.deferred.schedule do
#     every 2, task: PeriodicCounterTask, overlap: :allow
#   end

Scheduling
How scheduling will be implemented (timer mechanism, which process owns it)?
No new scheduler process/thread. We reuse Iodine's built-in timer primitives:

    Iodine.run_after(entry.first_run_delay_ms) do
      # Execute the first run
      execute_entry(entry)

      # Phase 2: Recurring execution
      Iodine.run_every(entry.interval_ms) do
        execute_entry(entry)

      end
    end

And then we're reusing the existing mechanism that regular one-off deferred jobs use to execute tasks.
Rage::Deferred::Task.enqueue

Overlap policy: What happens if a scheduled task is still running when the next interval fires?

We give the user two choices:

:skip (default) — Don't start a new one. Wait until the running one finishes, then resume on the next timer.
:allow — Start a new one anyway, even though the old one is still running.

Leader Election (Multi-Process)
Rage already uses File#flock for leader election in its WebSocket/Cable adapter. We use exactly the same pattern.
At boot, each process tries a non-blocking advisory exclusive lock on a shared Tempfile:
The first process to win the lock becomes the scheduler leader and calls for every periodic registered task.
All other processes silently stand by.
When the leader process dies, the OS releases the file lock automatically. The next time any standby worker restarts, it will win the next election and take over.

What happens when leader process restarts
yes the timers will reset, If a task runs every 60s and the leader dies at second 50, there's no catch up. The new leader waits a full 60s first_run_delay || interval_ms before the first run. Could this be something to look into?

Should every support passing arguments to the task's perform method?
No maybe. Unlike one-off enqueued jobs where you pass dynamic data (ProcessOrderJob.perform_later(order_id: 42)), scheduled tasks run on a timer and always get the same static values — so args and kwargs seems mostly noise.

[rsamoilov]

Hi @ShashantNagpure

This is a great start!

We reuse Iodine's built-in timer primitives

👍

Rage already uses File#flock for leader election in its WebSocket/Cable adapter. We use exactly the same pattern.

👍

yes the timers will reset, If a task runs every 60s and the leader dies at second 50, there's no catch up

👍

Overlap policy: What happens if a scheduled task is still running when the next interval fires? We give the user two choices:

I would suggest removing this config option. Have you checked how projects like que-scheduler or sidekiq-scheduler behave in this situation?

[ShashantNagpure]

que-scheduler
No overlap control at all. It simply enqueues jobs on cron schedule. It has schedule_type with two options, but those are about missed runs during downtime, not about overlap:
default — coalesces missed cron matches (only enqueues once on recovery)
every_event — replays every missed cron match on recovery

sidekiq-scheduler
No overlap control either. It pushes jobs to the Redis queue on schedule — period. If the previous job is still running when the next cron fires, a new job gets enqueued anyway.

For preventing concurrent runs, Sidekiq users are expected to use a separate layer: sidekiq-unique-jobs gem, Sidekiq Enterprise's unique_for, or application-level advisory locks.

The pattern
Both follow the same philosophy: the scheduler's only job is to enqueue on time. Overlap prevention is a separate concern.

Yes— we're mixing two responsibilities:

When to fire (scheduler's job)
Whether to allow concurrent runs (should be the task/queue layer's job)

sidekiq-unique-jobs uses Redis locks to prevent the same job from running concurrently. It provides several lock strategies:

class CleanupJob
  include Sidekiq::Job

  sidekiq_options lock: :while_executing,
                  lock_timeout: 0  # don't wait, just skip

  def perform
    # Only one instance of this job runs at a time.
    # If another is already running, this one is dropped.
  end
end

Suggestion:

Class CleanupTask
  include Rage::Deferred::Task
  self.unique = :while_executing  # or a simpler boolean
  
  def perform
    # ...
  end

If we were to follow this pattern, we'd remove overlpa from the scheduler DSL and instead add something like above

[rsamoilov]

If allowing overlaps is the default behavior for Sidekiq, Que, and most others, then what if we do the same and allow overlaps?