Read JSON document from socket and file descriptor

[Lucas286]

Hello,
I really like the library. Is there any safe way to load a document from a socket having its descriptor? How do I deal with framing and merging two different documents?

[penduin]

Glad you're enjoying it! Look into the WJReader docs (also on the wiki here on github) about reader callbacks to get data from anywhere, including a socket. It wouldn't hurt to accumulate some examples/helpers if you want to contribute what you come up with. (The same goes for WJWriter, you can write to a socket or anything else with a pretty simple callback.)

I'm not sure what you're asking when it comes to merges.
WJEMergeObjects handles stuff like:
{ a:1, b:2 } + { c:3 } = { a: 1, b:2, c:3 }
If by framing you mean something more like:
{ name:"shmoe" } + { age:123, smart:false } = { name: "shmoe", "attrs": { age:123, smart:false } }
then you'll want something like:
WJERename(from, "attrs");
WJEAttach(into, from);

Is any of that helpful?

[Lucas286]

In general, it is about non-blocking reading of the data stream from the socket. This stream is a json document. There can be many documents one after the other. Currently I have solved it using Newline Delimited JSON (ndjson.org). It's just that individual JSON documents are separated by '\ n'. Then reading and parsing documents is done with the following code:

// ipc client
typedef struct
{
	int                       fd;
	struct timespec   timeout;
	char*                   buff;
	int                        buff_idx;

} client_t;

// ****************************************************************************************************

int load_client_ndjson(client_t *client)
{
	struct timeval t;
	fd_set fdr;

	do
	{
		bzero(&t, sizeof(t));
		FD_ZERO(&fdr);
		FD_SET(client->fd, &fdr);

		if (select(client->fd + 1, &fdr, NULL, NULL, &t) > 0)
		{
			if (FD_ISSET(client->fd, &fdr))
			{
				char c;

				if (read(client->fd, &c, 1))
				{
					client->buff[client->buff_idx++] = c;
				}
				else
				{
					// socket disconected
					ipc_close_client(client);
					break;
				}

				if (c == '\n')
				{
					return 1;
				}
			}
		}
		else
		{
			return 0;
		}
	}
	while (client->buff_idx <  IPC_TCP_BUFFER_SIZE - 1);

	// overflow, clear buffer
	bzero(client->buff, IPC_TCP_BUFFER_SIZE);
	client->buff_idx = 0;

	return 0;

}

// ****************************************************************************************************

void ipc_client_process(client_t *client)
{
       // here descriptor i ready to read, but I am not sure that the entire document is readable

	// load full Newline Delimited JSON
	if (load_client_ndjson(client))
	{
               WJElement ndjson = NULL;

		ndjson = WJEParse(client->buff);

		if (ndjson)
		{
                       // wjelement operations....

                }
            
        }

}

I was wondering if this could be solved in a different way without using NDJSON.

[minego]

Well, it might be doable, but I think your current solution is better. The parser should stop parsing when it reaches the end of a document, as long as the JSON was not malformed in any way. In theory you can then start parsing another. I think that relying on finishing a document is going to be error prone though. The approach you're using of having a marker between documents is safer. - Micah

…

On Mon, Oct 05, 2020 at 11:50:26PM -0700, Lucas286 wrote: In general, it is about non-blocking reading of the data stream from the socket. This stream is a json document. There can be many documents one after the other. Currently I have solved it using Newline Delimited JSON [1](ndjson.org). It's just that individual JSON documents are separated by '\ n'. Then reading and parsing documents is done with the following code: // ipc client typedef struct { int fd; struct timespec timeout; char* buff; int buff_idx; } client_t; // ***************************************************************************** *********************** int load_client_ndjson(client_t *client) { struct timeval t; fd_set fdr; do { bzero(&t, sizeof(t)); FD_ZERO(&fdr); FD_SET(client->fd, &fdr); if (select(client->fd + 1, &fdr, NULL, NULL, &t) > 0) { if (FD_ISSET(client->fd, &fdr)) { char c; if (read(client->fd, &c, 1)) { client->buff[client->buff_idx++] = c; } else { // socket disconected ipc_close_client(client); break; } if (c == '\n') { return 1; } } } else { return 0; } } while (client->buff_idx < IPC_TCP_BUFFER_SIZE - 1); // overflow, clear buffer bzero(client->buff, IPC_TCP_BUFFER_SIZE); client->buff_idx = 0; return 0; } // ***************************************************************************** *********************** void ipc_client_process(client_t *client) { // here descriptor i ready to read, but I am not sure that the entire doc ument is readable // load full Newline Delimited JSON if (load_client_ndjson(client)) { WJElement ndjson = NULL; ndjson = WJEParse(client->buff); if (ndjson) { // wjelement operations.... } } } I was wondering if this could be solved in a different way without using NDJSON. — You are receiving this because you are subscribed to this thread. Reply to this email directly, [2]view it on GitHub, or [3]unsubscribe. References 1. http://ndjson.org/ 2. #92 (comment) 3. https://github.com/notifications/unsubscribe-auth/AADSRCAE2PTXQ7KZ73ZLDW3SJK43FANCNFSM4SADVGQA