[Lanayx]

Fixes #383

Problem

Since 3.19.8, a multi-topic consumer moves to a permanent Failed state as soon as one child consumer stops. After that, every call throws mt/consumer(...) not connected, and the only way out is to recreate the consumer. 3.16.0 could not recover either: the poller died and was only logged, so the consumer silently stopped delivering. The difference is that 3.19.8 reports it. This PR fixes the underlying problem.

Two paths take a regex consumer down. Both start with a matched topic going away.

1. A stopped child fails the parent's poller. Pattern children subscribe without creating topics, so when a matched topic is deleted, the child's resubscribe gets TopicDoesNotExistException and it moves to Failed. The next time the parent starts a read on that child (after a seek or redelivery, or when the poller picks up a read the child had already completed), the child throws NotConnectedException. That rethrow fails the poller and, through PollerFailed, the whole parent. A child mailbox failure took the same route.
2. Acknowledging a message from a removed topic crashes the parent mailbox. Acknowledge, NegativeAcknowledge, AcknowledgeCumulative and ReconsumeLater* used consumers[topic] directly in the mailbox loop. After pattern discovery removed a topic, acking a message the application still held from it threw KeyNotFoundException and failed the mailbox.

Fix

• A stopped child is detached instead of failing the parent. A child that returns, or throws, AlreadyClosedException/NotConnectedException posts ChildStopped, and its read parks. The parent removes it and then:
  • durable subscriptions, pattern consumers included: recreate the child with the client's backoff, off the mailbox, so the other topics keep being served. Until it is back, IsConnected and HasReachedEndOfTopic are false, and seeking or unsubscribing fails with NotConnectedException while the consumer keeps running. Skipping the child instead would report success while it kept its old position, or its subscription. Close and Unsubscribe wait for children that are still being created or disposed, so none stays subscribed after they return.
    Pattern children never create topics, so a deleted pattern topic can't be subscribed again and stays pending until the next discovery removes it. If the topic is recreated first, the next retry picks it up; before, a topic deleted and recreated between two discovery runs was never consumed again.
  • non-durable pattern consumers (non-partitioned child): drop the topic from the known topics, so the next discovery subscribes again if it still matches. The re-subscribed topic starts from the initial position, like any newly discovered topic.
  • other non-durable subscriptions (e.g. multi-topic readers): fail the parent, because a recreated child would restart from the initial position.
  • after a failed Unsubscribe, which leaves the consumer running so the call can be retried: fail the parent, because the child may already have been unsubscribed and must not be recreated. Ignoring the stopped child would leave its read, and pending receives, waiting forever. Before this PR, the poller failure already failed the parent in this case.
• New and recreated children start at the parent's consumer epoch. For Exclusive/Failover subscriptions the parent advances its epoch on every seek and redelivery, and it discards messages with an older epoch. A child created afterwards subscribed with epoch 0, so the parent silently dropped everything from it. This was already the case for partitions added by AutoUpdatePartitions and topics added by pattern discovery; recreation would have made it common.
• Children no longer dispose the parent's interceptors. Each child disposed the shared interceptors when it closed. Once one child stopped, or pattern discovery removed a topic, the remaining children kept calling disposed interceptors; OtelConsumerInterceptor, for example, stops tracing for good once disposed. Children now get a non-owning view, and the parent disposes the interceptors once, when it stops.
• Ack, nack, cumulative ack and reconsume for a topic without a consumer fail only that request, with NotConnectedException. The consumer keeps running. As with a single-topic consumer, a failed ack, cumulative ack or reconsume is passed to the interceptors' OnAcknowledge/OnAcknowledgeCumulative, so, for example, tracing ends for that message.
• Terminal failures keep their cause. NotConnectedException and AlreadyClosedException get an (message, innerException) constructor. A failed multi-topic consumer reports ... has failed and can't recover, it has to be recreated: <cause> and keeps the original exception as InnerException. ConnectionHandler remembers why it failed, so a failed child's errors carry the cause up to the parent too.
• A consumer whose topic no longer exists fails its pending receives with NotConnectedException, instead of leaving them waiting forever. This is how the parent learns a child has stopped. It also changes single-topic consumers: a pending ReceiveAsync on a deleted topic, when topic auto-creation is disabled, now fails instead of hanging.

Related fixes in code this PR touches:

• Child registration happens on the mailbox. Before, consumers.Add ran lazily inside sequences that TaskSeq's own thread consumed, racing the mailbox's reads of the same Dictionary after discovery or a partition update.
• handlePartitions enumerated its consumer-creation sequence a second time on failure, after the partition count had already been raised. Partly created consumers were never disposed, and the failed partitions were never retried. Consumers are now created once, and the new count is recorded only after all of them succeed. The receiver queue split now counts the new partitions, so it can't divide by zero when no child is currently registered. A failing partition check is logged instead of crashing the mailbox, as pattern discovery already did.
• processRemovedTopics matches children by their parent topic. It used to strip -partition- from child names, which did not match pattern topics that are themselves partition names. It also always forgets removed topics, even when they have no live child.
• LastDisconnectedTimestamp returns 0 when there are no children (e.g. a pattern that matches nothing) instead of failing the mailbox.
• ReconsumeLaterCumulative never completed its reply on success.

Tests

Integration tests that expected the old behaviour (a child failure stops the parent; an ack for an unknown topic fails the mailbox) were rewritten:

• Multi-topic consumer recreates a child whose mailbox fails: after a redelivery has moved the consumer epoch, a pending receive survives the failure, then gets a message from the recreated child.
• Child mailbox failure keeps a pending batch receive alive: the pending batch completes from the recreated child.
• Multi-topic consumer closes a recreated child at the broker
• Non-durable multi-topic consumer stops with the cause when a child mailbox fails
• Multi-topic consumer completes the request that faults its mailbox and Close queued behind a multi-topic consumer mailbox failure completes only after cleanup now fault the mailbox with a null poll result instead of a bogus ack. The first also checks that the cause is kept.

New tests:

• Multi-topic consumer fails only the acknowledgement for an unknown topic, which also checks that the interceptors are told about the failed acks
• Acknowledging a message from a removed pattern topic fails only that request
• Pattern consumer keeps running after a matched topic is deleted: force-deletes a matched topic, restarts every stream with a redelivery (this used to fail the consumer), checks that seeking and unsubscribing fail while the topic is pending, keeps consuming the other topic, then recreates the topic and checks it is subscribed again without waiting for discovery.
• Multi-topic consumer stops when a child stops after a failed unsubscribe: one child unsubscribes and the other has failed, so Unsubscribe fails; the pending receive then fails with AlreadyClosedException instead of waiting forever.
• Unsubscribing waits for a child that is still being recreated for a removed topic: holds the recreated child's broker lookup while discovery removes its topic, and checks Unsubscribe doesn't complete until that child is created and closed.
• A child recreated across a redelivery is created again at the new epoch: holds the recreated child's broker lookup while a redelivery moves the epoch on, checks the stale child is discarded and created again, that the new child's messages are delivered, and that neither child is still subscribed after Close.
• Closing a multi-topic consumer waits for a child that is being recreated: holds the recreated child's broker lookup, checks Close doesn't complete until that child is created and closed, then subscribes a new exclusive consumer to the topic.
• Multi-topic consumer refuses to seek or unsubscribe while a child is being recreated: uses a long client backoff so the failed child stays pending, and checks the consumer keeps consuming the other topic.
• Recreating a multi-topic consumer's child keeps the interceptors: the recreated child still runs the interceptors, and they are disposed exactly once, when the parent closes.

unwrap and assertNotConnected moved from Basic.fs to Common.fs so MultiTopic.fs can use them, next to the new assertNotConnectedBecauseOf and waitUntil helpers.

Verification

• dotnet build -c Release succeeds with no new warnings.
• Unit tests: 236 passed, 2 ignored.
• I did not run the integration tests locally; they run in this PR's Ubuntu CI.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Recovery still has unresolved shutdown, interceptor lifetime, unsubscribe, and end-of-topic correctness issues.

4 open issues

• Await in-flight child creation during Close · New
• Do not report end-of-topic while children are pending · New
• Handle pending child subscriptions before unsubscribe succeeds · New
• Prevent child disposal of shared interceptors · New

What changed in this PR

Addresses #383 by allowing multi-topic consumers to recover from stopped children while healthy topics continue delivering messages.

Changes:

• Adds child detachment and recreation, preserving consumer epochs across recovery.
• Isolates requests for removed topics and preserves terminal failure causes.
• Expands integration coverage for recovery, topic deletion, acknowledgements, and cleanup.

File	Description
tests/​IntegrationTests/​MultiTopic.fs	Tests pattern-topic removal, deletion, and rediscovery.
tests/​IntegrationTests/​Common.fs	Shares failure assertions and polling helpers.
tests/​IntegrationTests/​Basic.fs	Tests child recovery, terminal causes, and broker cleanup.
src/​Pulsar.Client/​Internal/​MultiTopicsConsumerImpl.fs	Implements child recovery and safer request handling.
src/​Pulsar.Client/​Internal/​ConsumerImpl.fs	Propagates failure causes and initializes child epochs.
src/​Pulsar.Client/​Internal/​ConnectionHandler.fs	Retains and exposes connection failure causes.
src/​Pulsar.Client/​Api/​PulsarClientException.fs	Adds constructors accepting inner exceptions.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Shutdown can miss outstanding child disposals, and stopped pattern children can be silently skipped by seek and unsubscribe operations.

1 open issue

• Await disposal of rejected recreated children · New

1 previously missed issue

In code that hasn't changed since last review

Track failed durable child topics as pending

src/​Pulsar.Client/​Internal/​MultiTopicsConsumerImpl.fs:818

When a durable pattern child's mailbox fails while its topic still exists, this branch forgets it without adding it to pendingChildren. Before the next discovery, SeekAsync and UnsubscribeAsync therefore pass their pending-child guards and report success after operating only on surviving topics, leaving the failed topic's cursor or durable subscription unchanged. Retain a pending marker until discovery confirms removal or successfully re-subscribes the topic, and include it in these guards.

4 resolved since last review

• Handle pending child subscriptions before unsubscribe succeeds
• Do not report end-of-topic while children are pending
• Await in-flight child creation during Close
• Prevent child disposal of shared interceptors

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Unsubscribe can complete before child cleanup finishes, and stalled polling conditions can hang the recovery tests.

2 open issues

• Wait for outstanding child creation during unsubscribe · New
• Bound condition checks by the remaining deadline · New

1 resolved since last review

• Await disposal of rejected recreated children

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

An unresolved lifecycle race and intertwined recovery and shutdown paths require human review and broker-backed integration verification.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Handle failed unsubscribe to cancel parked reads and receive waiters

src/​Pulsar.Client/​Internal/​MultiTopicsConsumerImpl.fs:846

If Unsubscribe fails before a queued ChildStopped is handled, it leaves the mailbox running in Failed (lines 1260–1262). This branch then ignores the stopped child without releasing its parked read or cancelling the receive token, so pending receives can hang indefinitely. Handle Failed explicitly by calling failConsumer ex and exiting the mailbox loop, which cancels parked reads and fails receive waiters.

2 resolved since last review

• Wait for outstanding child creation during unsubscribe
• Bound condition checks by the remaining deadline

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Missing-child request failures omit interceptor notifications, potentially leaving tracing activities open for the lifetime of the parent consumer.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Notify interceptors when acknowledgement requests fail

src/​Pulsar.Client/​Internal/​MultiTopicsConsumerImpl.fs:958

Missing-child acknowledgement and reconsume failures bypass the interceptor callbacks, so a retained message from a removed topic can leave its OpenTelemetry activity cached until parent shutdown when acknowledgement timeouts are disabled. Before failing each request, notify OnAcknowledge for individual operations or OnAcknowledgeCumulative for cumulative operations with the same exception. The same issue occurs in this file at lines 994, 1187, and 1203.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Concurrent child recovery, discovery, epoch changes, and shutdown coordination need final human review supported by confirmed broker-backed integration results.

No open issues 🎉

_{Review effort: Balanced}