You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Why does strict allow 'highWaterMark' pending writes, instead of just 1? #22
There may very well be a good answer to this, so this is partly just for my own understanding and to document the reasoning, since I can't see anything about this written down anywhere.
The arguments I can see for pending writes = 1 in strict mode:
It bounds the buffered chunks much closer to what you'd naively expect: if you set highWaterMark = N, it's surprising that a writer will hold 2N in flight. N + 1 is much closer & more justifiable bound given what you asked for.
If you do correct single-writer backpressure, you never need more than one pending write (multi-writer backpressure is complex and I think for strict + async at least it's not obvious how to do that safely, and an arbitrary parallelism limit is even worse, so better to fail clearly upfront anyway).
It tightens the constraint & strictness, so it's much more likely to fail faster if you're doing something wrong (instead of failing unpredictably later).
If you aren't doing correct backpressure in some subtle way you're eventually going to get into a world of trouble with strict regardless, and relaxing this doesn't help there - it'd be better to use block and accept this explicitly.
The Node docs themselves say:
If you properly await each write, you can only ever have one pending write at a time (yours), so you never hit the pending writes limit.
So why allow more than 1?
(Obviously inspired by #20, since this applies there too, and the double bound is extra surprising - why not just always allow exactly 1 pending slot?)
The way I've envisioned it is like a hose filling up a bucket. Both the bucket and the hose each have a capacity. The water in the bucket (the slot queue) is what we're currently processing. The water in the hose (the pending queue) is what we're processing immediately next. What we don't want is water spilling on the ground.
The strict policy just says that we can't overflow both the bucket or the hose. However, in order to keep data flowing smoothly we always want to try to keep water in both the bucket and the hose flowing at a steady rate. For strict, ideally the capacity of the hose matches the capacity of the bucket.
The highWaterMark primarily specifies the capacity of the bucket, but we also use it to determine the capacity of our hose such that in strict, the hose never contains more than what the bucket can hold. This means that in strict, the actual effective total capacity is highWaterMark times 2.
push({ highWaterMark: 3, backpressure: 'strict' })
Producer awaits each write. Consumer is slow.
Step 1: write('a') — bucket has space, goes directly in
hose bucket
+-+-+-+ +-+-+-+
. . . =======> . . a
+-+-+-+ +-+-+-+
Step 2: write('b') — bucket has space
hose bucket
+-+-+-+ +-+-+-+
. . . =======> . b a
+-+-+-+ +-+-+-+
Step 3: write('c') — bucket is now full
hose bucket
+-+-+-+ +-+-+-+
. . . =======> c b a
+-+-+-+ +-+-+-+
Step 4: write('d') — bucket full, write goes into hose (pending queue).
Producer blocks (awaits) until space opens up.
hose bucket
+-+-+-+ +-+-+-+
. . d =======> c b a
+-+-+-+ +-+-+-+
Step 5: write('e') — also goes into hose. Producer blocks.
hose bucket
+-+-+-+ +-+-+-+
. e d =======> c b a
+-+-+-+ +-+-+-+
Step 6: Consumer reads! Bucket drains completely. Hose refills bucket.
Pending writes resolve; producers unblock.
before read:
hose bucket
+-+-+-+ +-+-+-+
. e d =======> c b a
+-+-+-+ +-+-+-+
consumer gets: [c, b, a]
after drain + promote:
hose bucket
+-+-+-+ +-+-+-+
. . . =======> . e d
+-+-+-+ +-+-+-+
Step 7: write('f') — bucket has space again
hose bucket
+-+-+-+ +-+-+-+
. . . =======> f e d
+-+-+-+ +-+-+-+
And the fire-and-forget case that `strict` catches:
What if the producer does NOT await?
hose bucket
+-+-+-+ +-+-+-+
f e d =======> c b a — both full
+-+-+-+ +-+-+-+
write('g') — no room in bucket OR hose!
strict: RangeError "too many pending writes"
(the water would spill on the ground)
So that example is assuming multiple parallel producers, is that right? I think if it were just a single producer, AFAICT it blocks at step 4 and doesn't continue until the read happens, and so for that single well-behaved producer case your example never writes more than a single value into the pending queue at a time.
So in effect, in a strict + correct-backpressure world, we're using the pending queue size purely to set the number of parallel async-writing producers supported, is that right? That does make sense, and I can see how >1 is required to support that. It's a bit confusing though - it's not obvious that in the above example this is safe with 3 producers, but could fail at any time with 4, even if they're independently handling backpressure correctly.
I do see from your example nearly why the hose should never be bigger then the bucket though - that's required for efficient promotion, otherwise we have to go through an unbounded number of reads to empty all pending writes, whereas a pending size <= buffer size guarantees that we'll always have a pending slot after read, and need two reads at most to empty everything. Neat.
Interesting to think about anyway. We could consider defining this better in the docs to make the concurrent-producer behaviour (and expected patterns) clearer in general imo. Right now there's clear solutions for concurrent reads, but concurrent writes is an exercise for the reader. Not sure if you already have a suggested best practice & external contract for that somewhere?
So that example is assuming multiple parallel producers, is that right?
The number and type of producer is irrelevant here and the model makes no such assumptions.
I think if it were just a single producer, AFAICT it blocks at step 4 and doesn't continue until the read happens, and so for that single well-behaved producer case your example never writes more than a single value into the pending queue at a time.
For strict, the writes would not start rejecting unless both the slot queue (the bucket) and the pending queue (the hose) are both full.
The slots here represent number of write operations, not amount of data. In the current design, if we specify highWaterMark: 1, that places no constraints on the actual number of Uint8Arrays included in the write.
Ah, I see my example is a bit off at step 4. In both strict and block, with an async writers, when a write goes directly to the bucket, the returned promise is returned resolved (meaning, the write was immediately accepted into the bucket). If the write goes into the pending, the returned promise is pending. In that example, for strict, the caller can write up to two more times without running afoul of the strict policy and having writes reject. With the block policy, the pending writes just keep accumulating in the pending queue.
So that example is assuming multiple parallel producers, is that right?
The number and type of producer is irrelevant here and the model makes no such assumptions.
Yep, I think from your second comment you see what I mean. I was confused by blocking in step 4, but then continuing to step 5 without a read.
More generally though, I've done some tests, and I think there is a real parallel producer problem: I think if you have highWaterMark = N on a strict stream and N + 1 parallel producers, even if they always correctly use the patterns from the docs, currently any backpressure will always trigger a violation. Demo:
import*asSIfrom'stream/iter';const{ writer, readable }=SI.push({highWaterMark: 3});asyncfunctionstartWriting(id){for(constitemof['a','b','c','d','e']){console.log(`${id} writing ${item}...`);awaitwriter.write(item);console.log(`${id} wrote ${item}`);}}constPRODUCER_COUNT=4;for(leti=0;i<PRODUCER_COUNT;i++){startWriting(i);}// Sleep - don't read immediatelyawaitnewPromise(resolve=>setTimeout(resolve,1000));consti=readable[Symbol.asyncIterator]();console.log('Reading...');const{ value }=awaiti.next();console.log('Read',value);
That's the canonical correct writer pattern, and they crash immediately: 4 producers write a, 4th producers sees backpressure & waits, 3 remaining producers all write('b') => Backpressure violation.
Parallel producers can detect that there's backpressure (desiredSize === 0) but AFAICT we don't expose any promise to wait for drain except via write(), so there's no way they can safely & independently wait for it even if they want to.
Only solution would be a global backpressure promise that all producers carefully share and manage somehow, but that is a major constraint that's non-trivial to implement due to the TOCTOU & coordination involved. We don't have a way to handle this in our own code either - N+1 pipeTo fails under backpressure the same way.
Is there any way to rewrite the above example (4 independent producers writing to a 3-slot writer) so it handles backpressure correctly today?
For strict, the writes would not start rejecting unless both the slot queue (the bucket) and the pending queue (the hose) are both full.
They won't reject, but they'll block, and that guarantees a maximum one item in the pending queue given a single producer I think. Defining that more clearly:
This blocks after writing 'c' => that's 3 slots used (a/b/c), one pending queue (d). AFAICT there is no reader behaviour where this producer (the correct backpressure example from the Node docs) ever uses more than one pending slot. Every producer flow I can see that does use the queue requires either parallel producers, or the single producer to ignore a backpressure signal before a write() call.
Activity
jasnell commentedon Jun 2, 2026
The way I've envisioned it is like a hose filling up a bucket. Both the bucket and the hose each have a capacity. The water in the bucket (the slot queue) is what we're currently processing. The water in the hose (the pending queue) is what we're processing immediately next. What we don't want is water spilling on the ground.
The
strictpolicy just says that we can't overflow both the bucket or the hose. However, in order to keep data flowing smoothly we always want to try to keep water in both the bucket and the hose flowing at a steady rate. Forstrict, ideally the capacity of the hose matches the capacity of the bucket.The
highWaterMarkprimarily specifies the capacity of the bucket, but we also use it to determine the capacity of our hose such that in strict, the hose never contains more than what the bucket can hold. This means that in strict, the actual effective total capacity ishighWaterMarktimes 2.jasnell commentedon Jun 2, 2026
Here's a more structured example
pimterry commentedon Jun 2, 2026
So that example is assuming multiple parallel producers, is that right? I think if it were just a single producer, AFAICT it blocks at step 4 and doesn't continue until the read happens, and so for that single well-behaved producer case your example never writes more than a single value into the pending queue at a time.
So in effect, in a strict + correct-backpressure world, we're using the pending queue size purely to set the number of parallel async-writing producers supported, is that right? That does make sense, and I can see how >1 is required to support that. It's a bit confusing though - it's not obvious that in the above example this is safe with 3 producers, but could fail at any time with 4, even if they're independently handling backpressure correctly.
I do see from your example nearly why the hose should never be bigger then the bucket though - that's required for efficient promotion, otherwise we have to go through an unbounded number of reads to empty all pending writes, whereas a pending size <= buffer size guarantees that we'll always have a pending slot after read, and need two reads at most to empty everything. Neat.
Interesting to think about anyway. We could consider defining this better in the docs to make the concurrent-producer behaviour (and expected patterns) clearer in general imo. Right now there's clear solutions for concurrent reads, but concurrent writes is an exercise for the reader. Not sure if you already have a suggested best practice & external contract for that somewhere?
jasnell commentedon Jun 2, 2026
The number and type of producer is irrelevant here and the model makes no such assumptions.
For
strict, the writes would not start rejecting unless both the slot queue (the bucket) and the pending queue (the hose) are both full.The slots here represent number of write operations, not amount of data. In the current design, if we specify
highWaterMark: 1, that places no constraints on the actual number ofUint8Arraysincluded in the write.Whether the writes are coming from multiple producers (sync, async, whatever) is not a factor.
jasnell commentedon Jun 2, 2026
Ah, I see my example is a bit off at step 4. In both
strictandblock, with an async writers, when a write goes directly to the bucket, the returned promise is returned resolved (meaning, the write was immediately accepted into the bucket). If the write goes into the pending, the returned promise is pending. In that example, for strict, the caller can write up to two more times without running afoul of the strict policy and having writes reject. With theblockpolicy, the pending writes just keep accumulating in the pending queue.pimterry commentedon Jun 3, 2026
Yep, I think from your second comment you see what I mean. I was confused by blocking in step 4, but then continuing to step 5 without a read.
More generally though, I've done some tests, and I think there is a real parallel producer problem: I think if you have
highWaterMark = Non a strict stream andN + 1parallel producers, even if they always correctly use the patterns from the docs, currently any backpressure will always trigger a violation. Demo:That's the canonical correct writer pattern, and they crash immediately: 4 producers write
a, 4th producers sees backpressure & waits, 3 remaining producers allwrite('b')=> Backpressure violation.Parallel producers can detect that there's backpressure (
desiredSize === 0) but AFAICT we don't expose any promise to wait for drain except viawrite(), so there's no way they can safely & independently wait for it even if they want to.Only solution would be a global backpressure promise that all producers carefully share and manage somehow, but that is a major constraint that's non-trivial to implement due to the TOCTOU & coordination involved. We don't have a way to handle this in our own code either - N+1 pipeTo fails under backpressure the same way.
Is there any way to rewrite the above example (4 independent producers writing to a 3-slot writer) so it handles backpressure correctly today?
They won't reject, but they'll block, and that guarantees a maximum one item in the pending queue given a single producer I think. Defining that more clearly:
This blocks after writing 'c' => that's 3 slots used (a/b/c), one pending queue (d). AFAICT there is no reader behaviour where this producer (the correct backpressure example from the Node docs) ever uses more than one pending slot. Every producer flow I can see that does use the queue requires either parallel producers, or the single producer to ignore a backpressure signal before a
write()call.pimterry commentedon Jul 13, 2026
Resolved - as of #23, pending writes is now 1.