Release v2.9.9

Changelog

Go Version

• 1.19.4: Both the release executables and Docker images are built with this Go release

Improved

• JetStream

  • Avoid full state snapshot for streams with many deleted items (#3680)
  • Refactor to make stream removal from server consistent (#3691)

• Help/Usage

  • Display two additional --reload signal options, ldm and term (#3683)

Fixed

• Authorization

  • Prevent returning no_auth_required when a client tries to connect in operator mode (#3667)
    • Any client (Java) that suppressed sending credentials before of this flag could be affected.
    • This only affects the 2.9.8 server version

• JetStream

  • Tag policies not honored during stream replica reassignment after a peer is removed (#3678)
  • Address issues when concurrent "create" requests for the same stream are issued (#3679)
  • Server panic when consumer state was not decoded correctly (#3688)
  • Ensure consumers that are deleted on startup are removed from the system (#3689)
  • Fixed JetStream remained disabled for reactivated JWT accounts (#3690, thank you @JulienVdG)

• Leafnodes

  • Do not delay PINGs for leaf connections (#3692, thank you @sandykellagher)

Complete Changes

v2.9.8...v2.9.9

Release v2.9.8

Changelog

Go Version

• 1.19.3: Both release executables and Docker images are built with this Go release.

Monitoring:

• JetStream:
  • Server might crash if a pull consumer with an activity threshold is deleted immediately after a message ack (#3658). Thanks to @jdhenke for the report.
• Server Info:
  • If no_auth_user is set, clear the auth required flag in the server info presented to the client (#3659)

Complete Changes

v2.9.7...v2.9.8

Release v2.9.7

Changelog

Go Version

• 1.19.3: Both release executables and Docker images are built with this Go release.

Improved:

• JetStream:
  • Processing of acknowledgments (#3624)
  • Stream ingest performance with large numbers of consumers (#3633)
  • StreamDetail now includes stream creation for /healthz reporting. Thanks to @raypinto for the contribution (#3646)

Fixed

• JetStream:
  • For "Interest" or "WorkQueue" policy streams, when a consumer was deleted, a stream view could cause high CPU or memory usage. Thanks to @Kaarel for the report (#3610)
  • Make sure to enforce HA asset limits during peer processing as well as assignment (#3614)
  • Account max streams/consumers not always honored (#3615)
  • Reduce some warning log messages "append entries" (#3619)
  • Logic bug that would prevent some stream messages from being deleted. Thanks to @Kaarel for the report (#3620, #3625)
  • Account removal leaks subscriptions. Thanks to @JulienVdG for the report (#3627)
  • Server fails to start on concurrent map read/write (#3629, #3634)
  • Possible panic on stream info when leader is not yet or elected or loses quorum. Thanks to @apuckey for the report (#3631)
  • Allow any type of ack to suppress auto-cleanup of consumer (#3635)
  • Make mirror consumers use filtered version of consumer create (#3638)
  • WorkQueue was not correctly rejecting overlapping consumers in some cases. Thanks to @Kilio22 for the report (#3640)
  • Make sure header keys do not have additional prefixes. Thanks to @osmanovv for the report (#3649)
• Routing:
  • TLS connections to discovered server may fail (#3611, #3613)
• Weighted subject mappings updates not applied (#3618)

Complete Changes

v2.9.6...v2.9.7

Release v2.9.6

Changelog

Go Version

• 1.19.3: Both release executables and Docker images are built with this Go release.

Fixed

• JetStream:
  • Possible panic on some rare cases where a clustered consumer monitor go routine started while the consumer was deleted or scaled down to an R1. Some tracing could have caused a panic (#3599)
  • On stream proposal failures, the server would incorrectly warn about high stream lag. Thanks to @kino71 for the report (#3601)
  • Stream sources with OptStartTime would get redelivered messages on server restart. Thanks to @DavidCockerill for the report (#3606)
  • Scaling down a replicated stream (R>1) to R1 while it has no quorum, for instance due to a network partition, would leave the stream in a bad state and there would be a constant report of "No quorum, stalled" for this stream, even after the network partition is resolved (#3608)
• LeafNode:
  • Possible duplicate messages in complex setup. Thanks to @chenchunping for the report (#3604)

Complete Changes

v2.9.5...v2.9.6

Release v2.9.5

Changelog

Go Version

• 1.19.3: Both release executables and Docker images are built with this Go release.

Fixed

• JetStream:
  • Honor MaxMsgsPerSubject on stream update (#3595)
  • Errors such as "wrong sequence for skipped msg" or "expected sequence does not match store " while processing snapshots of streams with expired messages. Thanks to @MauriceVanVeen and @sylrfor the reports (#3589, #3596)
• Sublist's cache hit rate would be wrong in presence of multiple accounts. Thanks to @aopetrov86 for the report and contribution (#3591)

Complete Changes

v2.9.4...v2.9.5

Release v2.9.4

Changelog

Go Version

• 1.19.2: Both release executables and Docker images are built with this Go release.

Fixed

• Configuration Reload:
  • The option allow_non_tls would be ignored after a configuration reload. Thanks to @JulienVdG for the report (#3583)
• JetStream:
  • Possible deadlock. Thanks to @ashumkin for the report and @neilalexander for the contribution (#3555)
  • Possible panic in disk full situations. Thanks to @fantashley for the contribution (#3560, #3568)
  • "First sequence mismatch" after a restart/deployment with streams that have message TTLs. Thanks to @MauriceVanVeen for the report (#3567)
  • Update of an R1 consumer would not get a response. The update was accepted by the server, but the client library or NATS CLI would timeout waiting for the response (#3574)
  • Update of a consumer's InactiveThreshold would not always take effect. Thanks to @neilalexander for the contribution (#3575)
  • A consumer may not be removed based on InactiveThreshold in presence of gateways (#3575)
  • Migration of ephemerals on server shutdown was not working and could create "ghost" consumers on servers restart, that is, consumers that would be listed by the meta leader, but getting information about this consumer would fail. Migration will no longer occur, instead, all R1 pull consumers will be notified that the server is shutting down, invalidating the pending requests (#3576)
  • Consumers on a Limits policy stream could have their replicas changed to R1 (#3576)
  • Ensure that RAFT communication is properly stopped when needed, which otherwise could cause server memory usage increase (#3577)
  • Adding a warning when the inbound of messages causes a lag with the storage layer. In future release, the producers may be notified through a PubAck failure that the message cannot be accepted (#3578)
  • Added pending messages/bytes to pull request errors and status: when the server responds to the client library that a request has timed-out, or server is shutdown, etc..., the response will now include the request pending messages and bytes (#3572, #3587)
  • More messages than the max_msgs_per_subject value could be recovered on server restart following an abnormal server exit (#3579, #3582)
• Leafnode:
  • Existing subscriptions would be sent to leafnodes even though they violated permissions. The publish side would be doing the right thing by not sending the messages over, but the subscription interest was still sent (#3585)
• MQTT:
  • Subjects mapping were not working. Thanks to @ozon2 for the report (#3552)
• Routing:
  • An implicit route may not reconnect, regardless of the ConnectRetries setting. This can happen in configurations where the routes[] block contains only the seed (and not as a name that could resolve to each IP of the cluster). If a route to a discovered server is disconnected, it may not try to reconnect due to the implicit nature of that connection. Thanks to @wubumihuo for the report (#3573)

Complete Changes

v2.9.3...v2.9.4

Release v2.9.3

Changelog

Go Version

• 1.19.2: Both release executables and Docker images are built with this Go release.

Fixed

• JetStream:
  • Unresponsiveness (health check failures, routes being blocked) while creating a RAFT group when disk performance if very slow. Thanks to @TomasVojacek for the report (#3519)
  • Purge with additional options may leave some messages in the stream (#3529)
  • Prevent stack overflow when an account imports its own export. CVE-2022-42709 (#3538)
  • Prevent panic on negative replicas count. CVE-2022-42708 (#3539)
  • User-provided ephemeral consumer name was not used in cluster mode, that is, the server would still pick a random name (#3537)
• Added missing command line options in the -help section. Thanks to @ariel-zilber for the contributions (#3523, #3527)

Complete Changes

v2.9.2...v2.9.3

Release v2.9.2

Changelog

Go Version

• 1.19.1: Both release executables and Docker images are built with this Go release.

Improved:

• Fan-out performance degraded between v2.8.4 and v2.9.0. This was mainly due to addition of message count/size accounting per-account. Some code refactoring restored or even increased the performance compared to v2.8.4 (#3504)

Fixed

• JetStream:
  • Prevent panic processing a consumer assignment. This could happen in rare situations where a stream would catchup and start processing consumer assignments while the stream itself was "stopped", for instance during a cluster-reset event following a sequence mismatch detection, etc... (#3498)
  • The FileStore implementation could have held into memory of message blocks for longer (about 5 seconds) than needed. In situations where the stream was filling up quickly, the amount of blocks held in memory could grow enough that even 5 seconds expiration could cause the memory growth to be noticeable, even more so if the garbage collection has no reason to trigger if the overall memory usage is below the host's limit (#3501)
  • Scaling a consumer down to a R=1 would work but not send a response to the CLI/application requesting that change. That is, if an application would call js.UpdateConsumer() (using the Golang client library) with a replica of 1, the call would timeout but the operation would actually succeed (#3502)
  • The consumer names paging had issues and could have returned only the API limit, which for this request is 1024 (#3510)
  • Some streams may fail to be recovered if a meta-layer snapshot was done but an upstream source or mirror changed it subjects (#3508)
  • Redeliveries for consumers with the "LastPerSubject" delivery policy were not honored. Thanks to @brentd for the report (#3511)
• Possible exit of NATS Server running as a Windows service without logging enabled when server would report invalid protocols. The workaround is to enable logging (specify a log file as opposed to have the logging directed to the Windows Event Logs). Thanks to @BentTranberg for the report (#3497)

Complete Changes

v2.9.1...v2.9.2

Release v2.9.1

Changelog

Go Version

• 1.19.1: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Ability to remove a server by peer ID instead of server name (#3473)
  • Peer ID in the meta_cluster of statsz or jsz monitoring endpoint (#3476)
  • Ability to apply a discard new policy per subject. A new JSON field in the stream configuration called discard_new_per_subject can now be set (along with discard new policy and max messages per subject > 0) (#3487)

Improved

• JetStream:
  • Optimize acknowledgment handling. Thanks to @neilalexander for the report and contribution (#3468, #3472)

Updated

• Dependencies (#3491)

Changed

• JetStream:
  • When filtering a source stream, use the new consumer create API subject (#3478)

Fixed

• JetStream:
  • Peer randomization when creating consumers group for replicas of 1. Thanks to @goku321 for the contribution (#3470)
  • Added an error if consumer's Name and Durable are not equal when sending to the new $JS.API.CONSUMER.CREATE.%s.%s.%s subject (#3471)
  • Server was not sending a 409 to the client library when a pull request was closed after sending at least a message but could not send more if that would exceed the max_bytes pull request limit (#3474)
  • Possible panic on peer remove on server shutdown (#3477)
  • Filtered consumers may also receive messages on other subjects from the stream. Thanks to @perestoronin for the report (#3486)
• LeafNode:
  • A server that accepts a leaf connection on the websocket port, and the websocket{} block had a no_auth_user defined, this user was not being used for the account binding for that leaf node connection (#3489)
• Edge condition handling in {{Split()}} subject mapping function (#3463)

Complete Changes

v2.9.0...v2.9.1

Release v2.9.0

Changelog

IMPORTANT NOTE: Leafnode connections will now be closed if the cluster name is detected to be the same on the "hub" and "spoke", and reconnect attempts will be delayed by 30 seconds. If you are unaware of this misconfiguration you may experience a split network for longer than expected during the upgrade process. See the CHANGED section below and more details in pull request #3232.

Go Version

• 1.19.1: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • A new stream configuration field AllowDirect allows capable client libraries to have a new API that retrieves a message from any member of the group (leader or replicas). Note that this can lead to non-coherent reads after write since a replica may respond to the request although that replica does not yet have the latest write. The configuration field MirrorDirect allow the mirror to be part of the origin's group (#3158, #3221, #3238, #3247, #3252, #3325, #3329, #3358, #3380, #3392, #3441)
  • Support for InactiveThreshold for durable consumers. If the inactivity threshold is specified, a durable that is offline for more than this duration will be removed (#3190)
  • Ability for an operator to move streams, and support for tags reload (#3217, #3236, #3234, #3270, #3354, #3376, #3419)
  • Support for filter subject in a mirror configuration (#3227)
  • Support for consumer replica change. Thanks to @goku321 for the report (#3293)
  • Support for account purge operation by sending a request to $JS.API.ACCOUNT.PURGE.<account name>. Thanks to @goku321 and @sourabhaggrawal for the report (#3319, #3378)
  • Support for AES-GCM cipher encryption for FileStore (#3371)
  • Ability to override the default server limit for stream catchup. For instance jetstream: { max_outstanding_catchup: 32MB }. This limit is how many bytes in total may be inflight during streams catchup. This can help lower network bandwidth saturation (#3418)
  • Pagination for StreamInfo requests. Note that not all clients may have the ability to provide the offset at the time of the server release (#3454)
• LeafNodes:
  • Support for a SignatureHandler in remote configurations. This is applicable to applications embedding the NATS Server. Thanks to @kung-foo for the suggestion (#3335)
• Monitoring:
  • Account specific in/out messages/bytes and slow consumers statistics in $SYS.ACCOUNT.%s.SERVER.CONNS message response (#3187)
  • New endpoint /accstatz to get specific account statistics (such as number of connections, messages/bytes in/out, etc...) (#3250, #3382)
  • The /healthz endpoint is now also available via the system account under the $SYS.REQ.SERVER.PING.HEALTHZ subject (#3250)
  • New options for the /healthz endpoint: /healthz?js-enabled=true to return an error if JetStream is disabled, and /healthz?js-server-only=true to skip the check of JetStream accounts, streams and consumers. Thanks to @mfaizanse for the contribution (#3326)
  • The /connz?auth=1 endpoint now includes a tls_peer_certs array with subject, subject public key and raw certificate sha256. Thanks to @RedShift1 for the suggestion (#3387)
  • The /jsz?accounts=true endpoint will now show non 0 values for reserved_memory and reserved_storage when applicable (#3435)
• MQTT:
  • A new configuration option consumer_inactive_threshold allow new QoS1 consumers to be removed if they are offline for more than the specified duration (based on #3190) (#3193)
• Use of a library that automatically sets GOMAXPROCS to match Linux container CPU quota. Thanks to @1995parham for the contribution (#3218, #3224, #3237, #3406)
• A new server configuration option DontListen that triggers the server to accept only "in memory" client connections. This is for embedded use-cases only and is paired with changes made in the client library nats.go. Thanks to @neilalexander for the contribution (#2360, #3225)
• Support for JWT account option DisallowBearer (#3127)
• Stubs for WebAssembly. This allows NATS Server to be built under the js/wasm target. Thanks to @neilalexander for the contribution (#2363)
• Symlink for the deb/rpm packages. Since v2.7.4, the server is installed under /usr/bin instead of /usr/local/bin as it used to. We now have added symlink to /usr/local/sbin. Thanks to @ismail0352 for the report (#3242)
• Templates to scoped signing key user permissions (#3367, #3373, #3390)
• New subject mapping functions: SplitFromLeft, SplitFromRight, SliceFromLeft, SliceFromRight and Split (#3305)
• Building of executable, deb and rpm packages for the s390x architecture (#3458)

Changed

• Gateway:
  • Phasing out of the "optimistic" mode whereby a server could send messages to the remote cluster without knowing if there was an interest or not. The remote cluster would reply with a "no interest" protocol. As of v2.9.0, servers that creates a gateway connection to a server of that version (and above) will no longer send messages in optimistic mode since it is assumed that all accounts will be switched to interest-only mode (where the subscription interest map is sent over) (#3383)
• JetStream:
  • Stream's RePublish configuration field is now a new RePublish object (and SubjectMapping has been removed) that allows for a new boolean field called HeadersOnly. This could be useful for large messages and having republish just be a signaling mechanism (#3157)
  • When a pull request exceeds the maximum bytes, the error returned will be a "409" now instead of a "408" (#3172)
  • Make pull consumers FIFO per message, not per request (#3241)
  • Accept Nats-Expected-Last-Sequence with a 0 value. The server used to ignore if the sequence was 0, but now it will treat it as a requirement that the stream be empty if the header is present with a value of 0. Thanks to @bruth for the suggestion (#3038)
  • A consumer "Maximum Deliveries" count can now be updated. Thanks to @abegaj for the contribution (#3265)
  • Encryption of meta and RAFT stores (#3308)
  • Now return an error if there is an overlap between a source/mirror filter subject and the existing origin stream's subjects (#3356)
  • Compression in RAFT and stream catchup traffic when nodes are v2.9.0+. This can reduce network bandwidth (#3419)
• LeafNodes:
  • On establishment, the connection will now be closed when the same cluster name is detected on the "hub" and "spoke" side (#3232)
• Default to essential client information. We did default to full sharing for an export from the system account, the main one being JetStream (#3220)
• The queue group named _sys_ is now reserved and an application attempting to use it would get a permission violation (#3246)

Improved

• JetStream:
  • Server under heavy load and low on resources like file descriptors (#3168)
  • Performance of wildcard filtered consumer with stream with many subjects (#3184)
  • Better distribution in placement of streams in a cluster when no "max bytes" is set (#3194)
  • When the user adds a stream that already exists with a different configuration, the error description returned to the application was "stream name already in use", it will now be "stream name already in use with a different configuration". Also, when the user tries to restore a stream, if the stream was already present, the operation would fail with "stream name already in use", it will now be "stream name already in use, cannot restore". Thanks to @AndrewDK for the suggestion (#3280)
  • Speed of storing new keys with a large number of pre-existing keys (#3320)
  • Replicas ordering and information regarding unknown replica in stream information API response (#3347)
  • Catchup logic (#3348)
  • Catchup aborted on requester failure and better flow control (#3349)
  • FileStore for large KeyValue streams (#3351, #3353, #3366, #3401, #3413, #3416)
  • Usage of the FileStore implementation for the RAFT logs (#3377, #3394)
  • General improvements to clustered streams during server restart and KV/Compare-And-Swap use cases (#3392)
  • Ability to secure the creation of consumer for libraries sending consumer creation requests to the new subject $JS.API.CONSUMER.CREATE.<stream>.<subject>.<filter> (#3409)
  • Better accounting for max-bytes for pull consumers (#3456)
  • Better error description when an asset placement fails (#3342, #3459)
• Server banner for Trusted Operators now displays the Expires date as Never instead of the epoch time. Thanks @mschneider82 for the contribution (#3452)

Updated

• Dependencies (#3153, #3263, #3286)
• JetStream:
  • Allow consumer filter subjects to be updated (#3216)
  • AckAll and NoAck are now allowed on pull consumers (#3442)

Fixed

• Configuration Reload:
  • Configuration reload would fail if a leafnodes block would contain an authorization{} block, even if no modification was done there. Thanks to @cleaton for the report (#3204)
  • Some data races that could also lead to a server panic when processing a subscription and trying to send it to routes (#3222)
• Gateway:
  • Possible panic if monitor endpoint is inspected too soon on startup (#3374)
  • Possible routing issues with System messages and JetStream when internal subscriptions are created and internal requests are sent through gateways. The replies may sometimes be missed (#3427)
• JetStream:
  • Suppress consumer and stream advisories on server restart and any direct stream get message (#3156, #3160, #3162)
  • Possibly fail to retrieve a newly stored message. This would happen when stores and load using "last for subject" were concurrent (#3159)
  • When using Republish feature, republish on the republish subject and place original subject in a Nate-Subject header - similar to stream direct get message...