Error: IncompleteRead when data has empty values.

[RBhupi]

I think this error has two aspects to it, 1. the plugin.publish is accepting empty values and 2. sage_data_client is not able to send it.
This blocks the other good data. It will be expected from the user to either ignore these empty values or replace them with a None.

The sage_data_client.query() function, throws following error.

431992 rows × 15 columns

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/anaconda3/envs/data/lib/python3.12/http/client.py:579, in HTTPResponse._get_chunk_left(self)
    578 try:
--> 579     chunk_left = self._read_next_chunk_size()
    580 except ValueError:

File ~/anaconda3/envs/data/lib/python3.12/http/client.py:546, in HTTPResponse._read_next_chunk_size(self)
    545 try:
--> 546     return int(line, 16)
    547 except ValueError:
    548     # close the connection as protocol synchronisation is
    549     # probably lost

ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

IncompleteRead                            Traceback (most recent call last)
File ~/anaconda3/envs/data/lib/python3.12/http/client.py:595, in HTTPResponse._read_chunked(self, amt)
    594 try:
--> 595     while (chunk_left := self._get_chunk_left()) is not None:
    596         if amt is not None and amt <= chunk_left:

File ~/anaconda3/envs/data/lib/python3.12/http/client.py:581, in HTTPResponse._get_chunk_left(self)
...
    605     return b''.join(value)
    606 except IncompleteRead as exc:
--> 607     raise IncompleteRead(b''.join(value)) from exc

IncompleteRead: IncompleteRead(56727 bytes read)

Use the following code to query data:

df = sage_data_client.query(start="2024-07-27T10:00:00Z",
                            end="2024-07-28T10:00:00Z",
                            filter={
                                "vsn": "W09E",
                                "plugin": '10.31.81.1:5000/local/plugin-metek-sonic3d-sampler'
                            })

[seanshahkarami]

Thanks for submitting this.

I was able to reproduce this issue in your example and now I'm using the API directly to see if this is hitting some limit on how much data can be transmitted in a single response.