Does this require the entire JSON input to be buffered in memory?

[prdoyle]

Thank you for writing this!

I don't see any logic for handling buffer boundaries. Have I missed it? Or does this library require all the input to be in one big buffer?

[piotrrzysko]

Currently, the entire JSON must be loaded. Some time ago, I tried to add support for streaming, but the performance wasn’t good. I think it would be very useful to have this feature.

[prdoyle]

Makes sense.

Stage 1, at least, already works block-by-block, right? I wonder if it might suffice to save these variables and initialize them correctly when the next block arrives.

[piotrrzysko]

Yes, but the question is what to do with phase 2. Wait until the entire input is processed by stage 1, or try to move forward while waiting for another block? I think I tried the latter.

Additionally, back when I attempted to implement stream processing, there were some specific requirements for how vectorized processing should be implemented to make the most out of it. Please take a look at #53. I’m assuming that when we start breaking the input into smaller batches, pausing, and resuming processing, we might lose some of the gains that vectorization would otherwise offer.

BTW, I'm not saying this is impossible, just sharing what I remember :)

[prdoyle]

Makes sense.

I encountered your work while working on my own high-performance JSON object mapper. I guess the path of least resistance is probably for me to write it much like SchemaBasedJsonIterator. Mine compiles the object mapping logic down into efficient bytecode, but it's currently bottlenecked on the naive character-by-character JSON parsing I'm doing.