Where do object streams fit in?

[vweevers]

One thing I like about node.js streams is composability. It's easy to compose pipelines that mix binary and object streams. For example, parsing a csv file, transforming the rows (objects), then writing back to a file. When raw performance isn't important (it often isn't) then node.js streams are pretty great.

Bob is binary-only, so what will replace object streams? If the answer is "async iterators", how does one compose a pipeline as easily as x.pipe(y).pipe(z)? And if each of those is an async iterator, wouldn't that hurt throughput, as you can no longer transform multiple objects in one tick?

[Fishrock123]

There is no planned support for object streams. It is possible that a user-land module with a similar api would expose a spec / stream base classes / validator for an object-mode version, but the plan is specifically to not support such behavior for core apis.

That being said, an object mode version of this would basically be the same except you'd swap the buffer for an object?

And if each of those is an async iterator, wouldn't that hurt throughput, as you can no longer transform multiple objects in one tick?

Node streams call process.nextTick() anyways so there is no difference regarding that.

[vweevers]

It is possible that a user-land module with a similar api would expose [..] an object-mode version

That would even be preferable. Similar to the role that readable-stream has in userland streams now. I do hope that node core takes this use case into account, and not let it be an afterthought.

Node streams call process.nextTick() anyways so there is no difference regarding that.

In transform streams the _transform function may be called multiple times, if data is available.

[Fishrock123]

BOB does not have any arbitrary delays by default. If you use async iterators to do the transform, that is a cost you are implementing.

Data does not get buffered at each step though, and so you'd do another pull cycle, which may end up doing some kind fo async read, assuming the source hasn't buffered more than you can read.

See #30 regarding the possibility and implications of concurrent data pulls.

[vweevers]

If you use async iterators to do the transform, that is a cost you are implementing.

My question is about a larger scope than bob I guess. Because if bob eventually replaces the streams of node core (will it?) then users have no choice but to use async iterators for non-binary things.

[Fishrock123]

Sorry I'm not following - could you elaborate on why? BOB doesn't use promises at all in its current state. It's just a bunch of classes linked together who call each other's methods.

[vweevers]

I mixed up some things, so I'll take a step back and rephrase.

My worry is that current streams will be replaced by bob, which creates a gap where object streams used to be. Recently node folks like @mcollina have been advocating async iterators as well. So I thought, maybe async iterators are intended to fill that gap.

How will these efforts fit together in the end?

[Fishrock123]

Those two things will be disjoint.

If BOB makes it to node core, it is most likely that existing streams will be built ontop of bob, at least for the binary case.

We can't really get rid of e.g. Readable with objectMode, so that will likely continue to function separately.

Async iterators on the other hand can and will be built ontop of both. In the binary case, it would give you a buffer. In the object case, an object.

Does that help clear things up?

[vweevers]

It does, thanks!