support setting headers for kv put

[williamstein]

Proposed change

If you are putting a value using the key:value store, NATS itself supports headers. However, this nats.js client does not. In particular, this function in kv/src/kv.ts obviously doesn't support setting arbitrary user-specified headers:

  async put(
    k: string,
    data: Payload,
    opts: Partial<KvPutOptions> = {},
  ): Promise<number> {
    const ek = this.encodeKey(k);
    this.validateKey(ek);

    const o = {} as JetStreamPublishOptions;
    if (opts.previousSeq !== undefined) {
      const h = headers();
      o.headers = h;
      h.set(PubHeaders.ExpectedLastSubjectSequenceHdr, `${opts.previousSeq}`);
    }
    try {
      const pa = await this.js.publish(this.subjectForKey(ek, true), data, o);
      return pa.seq;
    } catch (err) {
      return Promise.reject(err);
    }
  }

I propose adding the following code after the if block above (and changing the typings appropriately):

  if (opts.headers !== undefined) {
    for (const [key, value] of opts.headers) {
      if (o.headers == null) {
        o.headers = headers();
      }
      o.headers.set(key, value[0]);
    }
  }

This makes it possible to set a header whenever setting a key.

Use case

This is an extremely useful primitive to have. E.g., today (for cocalc.com) I implemented chunking on top of the kv store so that I could store arbitrarily large values transparently, even if the configured server message size is small. To implement this, I absolutely needed headers. As a workaround, I just copy/pasted the put function and made the suggested modification above.

Contribution

Yes.

[aricart]

@williamstein currently our specification (unless it changed - will check) doesn't support user-specified headers on KV.

[williamstein]

currently our specification (unless it changed - will check) doesn't support user-specified headers on KV.

Thanks. I assume this is the spec:

https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-8.md

I don't think it mentions user-specified headers. Moreover, since at least one header is currently used for the implementation of KV, e.g., Nats-Expected-Last-Subject-Sequence, there could be a subtle conflict.

All that said, I can't think of any good way of implementing arbitrary size values without using headers, whereas the implementation I already wrote using headers turned out very nice. So I'll stick with my workaround no matter what (yeah, open source). Please close this issue if you feel there is no chance of extending the spec to support user defined headers. But perhaps keep in mind that user-defined headers are a natural primitive that might be easy for you guys to support, and having them enables valuable things that are extremely painful otherwise. And I would be fine with a constraint like "only user defined headers that don't start with Nats- are allowed". I would also be fine with the value of the headers being reset to whatever they are the last time the value for a given key is set. I can see though that there are several non-obvious design questions (e.g., what about history?).

[aricart]

If you want to store arbitrary sized data, object store is the correct API.
From a query to the group, the idea is to explicitly not support hearders.

[williamstein]

Thanks. Just to be clear, my request here is supporting headers, not storing arbitrarily large objects -- that was just an example application, of many. The object store API is difficult to work with in comparison to KV (e.g., this). Thanks for checking with the group that they actively don't want to support headers, which would make KV much more consistent with how streams work. It's a good thing NATS is open source, so that users like me still have the option to improve the tools along the edges, based on our experience building real world products on top of NATS.

[williamstein]

I just hit another application of headers for the kv store, which has nothing to do with storing large values. The keys that NATS supports are fairly restricted (UTF-8 minus some symbols), and I think this kv implementation dramatically restricts them further to a small subset of ASCII (for no apparent reason?). One could compute a key this is valid according this to this library, e.g., by computing the sha1 hash of the actual key, and also store the actual key in a header. Storing the actual key is required if you're iterating over your data, since you need to know what the keys are (not just their hashes).

The only alternative to this would be to store the actual key in some other place in the key-value store, e.g., ${sha1}-key, or to make the value be a combination of the actual key and the value, which is tricky to manage (and a hard-to-maintain reinvention of headers).

I suspect there are other applications of headers, which support giving users of kv even more power to build interesting things, going beyond what the NATS developers envision.