Skip to content

Conversation

@MJarmo
Copy link

@MJarmo MJarmo commented Sep 26, 2025

Add persistent storage support to OTLP receiver

Summary

Introduces persistent storage support for the OTLP receiver to persist incoming telemetry (logs/traces/metrics) before export.

Goal

Prevent losing logs on high load/temporary backend outages by persisting queued telemetry.

Key changes

  • Add storage/persistence integration to the OTLP receiver (file-based storage extension wiring).
  • Ensure queue/receiver flow writes to persistent storage when configured.
  • Fixes and improvements:
    • Fixes losing logs during receiver shutdown or high load.
    • Fix context propagation/usage in receiver code paths.
    • Lint/test fixes and naming improvements.
    • Prevent unkeyed composite literal usages.
  • Tests updated/fixed to reflect new storage behavior.

Motivation

  • Improve durability for audit/logging use-cases where telemetry must not be lost.
  • Allow the Collector to act as a reliable ingestion point by persisting incoming data when configured.

Behavioral notes

  • Persistent storage is opt-in via receiver configuration (storage extension). Default behavior remains unchanged unless storage is enabled in config.
  • When enabled, OTLP receiver persists incoming items to configured storage backend and reads them for export, reducing the risk of data loss on collector restart or exporter unavailability.

Testing performed

  • Unit tests updated and fixed.
  • New/updated tests to ensure persistence behavior and that logs are not lost.
  • Lint and build checks applied.

How to validate

  • Enable storage extension and configure OTLP receiver to use it.
  • Simulate exporter downtime or restart the collector; verify persisted telemetry is replayed on recovery.
  • Verify behavior without storage remains same as before (no persistence).

Compatibility / Migration

  • Backwards compatible: if no storage is configured, receiver behaves as before.
  • Configuration changes required only to opt-in to persistence.

Request for reviewers

  • Verify correctness of storage wiring and lifecycle handling.
  • Validate that no data duplication occurs during replay.
  • Review error paths and shutdown behavior to ensure clean shutdown without data loss.

Related

  • Addresses reliability requirements for audit logging and long-term retention scenarios.

@MJarmo MJarmo force-pushed the AddStorageToOTLPREciver branch 2 times, most recently from 60daf98 to 40f7bc3 Compare September 29, 2025 10:05
@MJarmo MJarmo force-pushed the AddStorageToOTLPREciver branch from 27e99c1 to 318de4f Compare October 7, 2025 09:26
@hilmarf hilmarf moved this to In progress in OTel-Audit-Logging Oct 14, 2025
@MJarmo MJarmo changed the title Add storage to otlpr eciver Add storage to otlp reciver Oct 16, 2025
@MJarmo MJarmo force-pushed the AddStorageToOTLPREciver branch from 06ffb75 to b5eaa4b Compare October 16, 2025 13:47
Michał Jarmolkiewicz and others added 9 commits October 16, 2025 15:47
Signed-off-by: MJarmo <[email protected]>
Signed-off-by: MJarmo <[email protected]>
Signed-off-by: MJarmo <[email protected]>
Signed-off-by: MJarmo <[email protected]>
Signed-off-by: MJarmo <[email protected]>
Signed-off-by: MJarmo <[email protected]>
Signed-off-by: MJarmo <[email protected]>
Signed-off-by: MJarmo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

1 participant