"JetStream resource limits exceeded for server" error while there is enough resources

[adaqus]

Observed behavior

A stream with compression s2 turned on stops receiving new messages after its nominal size (before compression) reaches Max Storage value reported in NATS logs during server startup, even though stream size on disk is 141 MB.

During server startup, we can see that Max Storage is 679.64 MB:

[1] 2024/12/13 20:39:02.186294 [INF] Starting JetStream
[1] 2024/12/13 20:39:02.186505 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[1] 2024/12/13 20:39:02.186508 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[1] 2024/12/13 20:39:02.186509 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[1] 2024/12/13 20:39:02.186510 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[1] 2024/12/13 20:39:02.186511 [INF] 
[1] 2024/12/13 20:39:02.186515 [INF]          https://docs.nats.io/jetstream
[1] 2024/12/13 20:39:02.186517 [INF] 
[1] 2024/12/13 20:39:02.186517 [INF] ---------------- JETSTREAM ----------------
[1] 2024/12/13 20:39:02.186520 [INF]   Max Memory:      46.46 GB
[1] 2024/12/13 20:39:02.186521 [INF]   Max Storage:     574.38 MB
[1] 2024/12/13 20:39:02.186522 [INF]   Store Directory: "/data/jetstream"
[1] 2024/12/13 20:39:02.186523 [INF] -------------------------------------------

After sending some messages to a stream with compression turned on, no new messages appear on it. Stream size reported by nats stream ls is 680 MiB:

❯ nats -s localhost:4222 stream ls
╭──────────────────────────────────────────────────────────────────────────────────────╮
│                                        Streams                                       │
├─────────────┬─────────────┬─────────────────────┬───────────┬─────────┬──────────────┤
│ Name        │ Description │ Created             │ Messages  │ Size    │ Last Message │
├─────────────┼─────────────┼─────────────────────┼───────────┼─────────┼──────────────┤
│ benchstream │             │ 2024-12-13 21:14:49 │ 4,597,794 │ 680 MiB │ 3m1s         │
╰─────────────┴─────────────┴─────────────────────┴───────────┴─────────┴──────────────╯

And in server logs we can see following messages:

[1] 2024/12/13 20:21:59.844541 [ERR] JetStream resource limits exceeded for server
[1] 2024/12/13 20:22:09.844573 [ERR] JetStream resource limits exceeded for server

Stream size on disk:

❯ sudo du -sh /mnt/nats_data/jetstream/\$G/streams/benchstream
141M	/mnt/nats_data/jetstream/$G/streams/benchstream

Stream info:

❯ nats -s localhost:4222 stream info
? Select a Stream benchstream
Information for Stream benchstream created 2024-12-13 21:14:49

              Subjects: test.>
              Replicas: 1
               Storage: File
           Compression: S2 Compression

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
            Direct Get: true
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

State:

              Messages: 4,597,794
                 Bytes: 680 MiB
        First Sequence: 1 @ 2024-12-13 21:14:54
         Last Sequence: 4,597,794 @ 2024-12-13 21:21:59
      Active Consumers: 0
    Number of Subjects: 1

Expected behavior

The stream continues to receive messages and error JetStream resource limits exceeded for server appears after stream size on disk (not size reported by stream ls) exceeds Max Storage.

Server and client version

Server version:

[1] 2024/12/13 20:13:33.306281 [INF] Starting nats-server
[1] 2024/12/13 20:13:33.306358 [INF]   Version:  2.10.7

Note: upgrading to NATS 2.10.23 (latest) doesn't fix this.

Client version:

❯ nats --version
v0.1.5

Host environment

Docker on Ubuntu 24.04, Kubernetes 2.28

Steps to reproduce

On Ubuntu:

1. In order to simulate limited disk space create 1GB image:

sudo dd if=/dev/zero of=/var/lib/docker/nats_data.img bs=1G count=0 seek=1

2. Format the image

sudo mkfs.ext4 /var/lib/docker/nats_data.img

3. Mount the image

sudo mkdir /mnt/nats_data
sudo mount -o loop /var/lib/docker/nats_data.img /mnt/nats_data

4. Start NATS JetStream node

docker run -v /mnt/nats_data/:/data -p 4222:4222 nats -js -sd /data

5. Create a stream with S2 compression (the rest of options are defaults)

nats -s localhost:4222 stream add benchstream --ack --compression s2 --subjects test.\>

6. Start sending 100 million messages to the stream

nats -s localhost:4222 pub test.msg --count 100000000 "Some message test with some random content {{ ID }} and additional content so a message has some length"

After some time we can see in NATS logs:

[1] 2024/12/13 20:21:59.844541 [ERR] JetStream resource limits exceeded for server
[1] 2024/12/13 20:22:09.844573 [ERR] JetStream resource limits exceeded for server
[1] 2024/12/13 20:22:19.844624 [ERR] JetStream resource limits exceeded for server
[1] 2024/12/13 20:22:29.844651 [ERR] JetStream resource limits exceeded for server

[MauriceVanVeen]

At the moment this is indeed the way how it's implemented. All limits use the total bytes as reported through nats stream info/ls/report, and not the underlying size on disk.

I'm guessing that that hasn't been implemented yet, as when using compression there's a variable rate of how efficient it will be, and deleting/erasing a message could actually increase size after compression. So those are some things that would need to be accounted for when fixing this. Although not sure yet when this would get fixed as there's more priority on making other fixes at the moment.