Code

TBS: Replace badger with pebble#15235

Code

Merged

carsonip merged 190 commits intomainelastic/apm-server:mainfrom tbs-pebble-rebaseelastic/apm-server:tbs-pebble-rebaseCopy head branch name to clipboardJan 29, 2025

+2,048-2,325

Commits

Commits on Dec 16, 2024

• WIP

  carsonip

  committed
  7478521

  Copy full SHA for 7478521

• Add pebble

  carsonip

  committed
  29b3a63

  Copy full SHA for 29b3a63

• Use pebble

  carsonip

  committed
  452c0ca

  Copy full SHA for 452c0ca

• Implement sampling decision

  carsonip

  committed
  e30ee49

  Copy full SHA for e30ee49

• Fix locking, use NoSync

  carsonip

  committed
  3467415

  Copy full SHA for 3467415

• Tune batch flush size, add delete

  carsonip

  committed
  886643f

  Copy full SHA for 886643f

• Sharded

  carsonip

  committed
  7d2618d

  Copy full SHA for 7d2618d

• Increase flush threshold to 5MB

  carsonip

  committed
  79e32c0

  Copy full SHA for 79e32c0

Commits on Dec 19, 2024

• Merge branch 'main' into tbs-pebble

  carsonip

  committed
  75efa89

  Copy full SHA for 75efa89

Commits on Jan 14, 2025

• Merge branch 'main' into tbs-pebble

  carsonip

  committed
  dc5bc9a

  Copy full SHA for dc5bc9a

• Replace badger

  carsonip

  committed
  d2b1080

  Copy full SHA for d2b1080

• Remove gc loop and drop loop

  carsonip

  committed
  640ee0c

  Copy full SHA for 640ee0c

• Fix test

  carsonip

  committed
  8596317

  Copy full SHA for 8596317

• Restore storage monitoring

  carsonip

  committed
  3a6f727

  Copy full SHA for 3a6f727

• Update processor test

  carsonip

  committed
  fd9abbc

  Copy full SHA for fd9abbc

• Explain AlreadyTailSampled failure

  carsonip

  committed
  896dedb

  Copy full SHA for 896dedb

• Update size estimation

  carsonip

  committed
  2c70ad5

  Copy full SHA for 2c70ad5

• Add FIXME

  carsonip

  committed
  cc38dbb

  Copy full SHA for cc38dbb

• Flush by number of pending writes for fair perf comparison

  carsonip

  committed
  8841d41

  Copy full SHA for 8841d41

• Try disable WAL

  carsonip

  committed
  adac2a2

  Copy full SHA for adac2a2

• Try in-memory mode

  carsonip

  committed
  a71fab0

  Copy full SHA for a71fab0

• Revert "Try in-memory mode"

  Show description for 15387c8
  carsonip

  committed
  15387c8

  Copy full SHA for 15387c8

• Revert "Try disable WAL"

  Show description for d73809d
  carsonip

  committed
  d73809d

  Copy full SHA for d73809d

• Use fork for pebble batch config

  carsonip

  committed
  b294877

  Copy full SHA for b294877

• Add db commit threshold bytes

  carsonip

  committed
  e2d2cdf

  Copy full SHA for e2d2cdf

• Use BatchOption

  carsonip

  committed
  60ab9ca

  Copy full SHA for 60ab9ca

• Add FIXME

  carsonip

  committed
  ef1b422

  Copy full SHA for ef1b422

• Fix compile error

  carsonip

  committed
  4b1cbdd

  Copy full SHA for 4b1cbdd

• Add logger to pebble

  carsonip

  committed
  c0e679c

  Copy full SHA for c0e679c

• Sort imports, remove logger.go

  carsonip

  committed
  c883c35

  Copy full SHA for c883c35

• Disable pebble level compression

  carsonip

  committed
  08c9205

  Copy full SHA for 08c9205

• Add table bloom filter

  carsonip

  committed
  1c89735

  Copy full SHA for 1c89735

• Add FIXME

  carsonip

  committed
  c35dcb9

  Copy full SHA for c35dcb9

Commits on Jan 15, 2025

• Add prefix to sampling decision key to separate from events to improve perf

  carsonip

  committed
  ea7b481

  Copy full SHA for ea7b481

• Use a sync.Map for sampling decision to establish baseline perf

  carsonip

  committed
  5dde3d0

  Copy full SHA for 5dde3d0

• Try enabling snappy to trade cpu for reduced disk IO / usage

  carsonip

  committed
  b8f50f3

  Copy full SHA for b8f50f3

• Try 16KB block size for better compression

  carsonip

  committed
  554e93d

  Copy full SHA for 554e93d

• Try FormatMajorVersion

  carsonip

  committed
  e1694a1

  Copy full SHA for e1694a1

• Revert "Use a sync.Map for sampling decision to establish baseline perf"

  Show description for a1e9010
  carsonip

  committed
  a1e9010

  Copy full SHA for a1e9010

Commits on Jan 20, 2025

• Use a separate DB for decision

  carsonip

  committed
  92b49ec

  Copy full SHA for 92b49ec

• First attempt to fix test and benchmarks

  carsonip

  committed
  2643e50

  Copy full SHA for 2643e50

• Use configured codec

  carsonip

  committed
  e7ac296

  Copy full SHA for e7ac296

• Use prefix bloom filter

  carsonip

  committed
  c128e75

  Copy full SHA for c128e75

• Prefix bloom can only be tested when db flushes to disk

  carsonip

  committed
  86975b8

  Copy full SHA for 86975b8

• Add decision db flush

  carsonip

  committed
  add0f9e

  Copy full SHA for add0f9e

• Set decision db block size to 2KB

  carsonip

  committed
  f4049af

  Copy full SHA for f4049af

Commits on Jan 21, 2025

• Change sampling decision flush threshold

  carsonip

  committed
  0c8d662

  Copy full SHA for 0c8d662

• Disable decision batch

  carsonip

  committed
  6ba50f5

  Copy full SHA for 6ba50f5

• Disable batch for events

  carsonip

  committed
  779edce

  Copy full SHA for 779edce

• Remove all batch code

  carsonip

  committed
  01f2276

  Copy full SHA for 01f2276

• Add fixme

  carsonip

  committed
  b1adee6

  Copy full SHA for b1adee6

• Try bypassing sharded rw

  carsonip

  committed
  985951f

  Copy full SHA for 985951f

• Revert "Try bypassing sharded rw"

  Show description for 24d9a9c
  carsonip

  committed
  24d9a9c

  Copy full SHA for 24d9a9c

• WIP

  carsonip

  committed
  80c6cfe

  Copy full SHA for 80c6cfe

• Use partition

  carsonip

  committed
  681d12e

  Copy full SHA for 681d12e

• Implement TTL using partition

  carsonip

  committed
  b7854e1

  Copy full SHA for b7854e1

• Compact the DBs

  carsonip

  committed
  5236c7c

  Copy full SHA for 5236c7c

• Clean up

  carsonip

  committed
  44ddc87

  Copy full SHA for 44ddc87

• Fix TTL Loop

  carsonip

  committed
  0411dfa

  Copy full SHA for 0411dfa

• Fix close error

  carsonip

  committed
  616bc0b

  Copy full SHA for 616bc0b

• Remove ManagedReadWriter

  carsonip

  committed
  e4c62b7

  Copy full SHA for e4c62b7

• Fix modulo of negative number

  carsonip

  committed
  c085876

  Copy full SHA for c085876

• Rename to prefix read writer

  carsonip

  committed
  3961e81

  Copy full SHA for 3961e81

• Split interface

  carsonip

  committed
  9443b73

  Copy full SHA for 9443b73

• Add SplitReadWriter

  carsonip

  committed
  f3582cc

  Copy full SHA for f3582cc

• Rename s->sm

  carsonip

  committed
  a0556b7

  Copy full SHA for a0556b7

• Remove TTL in WriterOps

  carsonip

  committed
  39fe340

  Copy full SHA for 39fe340

• Rename

  carsonip

  committed
  825be6c

  Copy full SHA for 825be6c

• Remove pointer receiver

  carsonip

  committed
  a67a315

  Copy full SHA for a67a315

• Fix PrefixReadWriter flush

  carsonip

  committed
  13b4031

  Copy full SHA for 13b4031

• make fmt

  carsonip

  committed
  740c4f9

  Copy full SHA for 740c4f9

• Use upstream pebble, go mod tidy

  carsonip

  committed
  ec1a087

  Copy full SHA for ec1a087

• make notice

  carsonip

  committed
  b2bf6b0

  Copy full SHA for b2bf6b0

• Delete storage whitebox test

  carsonip

  committed
  1c31a69

  Copy full SHA for 1c31a69

• Rename test

  carsonip

  committed
  1fef6a7

  Copy full SHA for 1fef6a7

• Delete TTL in writerOpts

  carsonip

  committed
  19a6e7d

  Copy full SHA for 19a6e7d

• Try bypassing ShardedReadWriter

  carsonip

  committed
  adcfc22

  Copy full SHA for adcfc22

• Revert "Try bypassing ShardedReadWriter"

  Show description for 3675eef
  carsonip

  committed
  3675eef

  Copy full SHA for 3675eef

• Add FIXME

  carsonip

  committed
  145d392

  Copy full SHA for 145d392

• Fix ttl loop param

  carsonip

  committed
  cd86c64

  Copy full SHA for cd86c64

• Add FIXME

  carsonip

  committed
  2e8e00b

  Copy full SHA for 2e8e00b

Commits on Jan 22, 2025

• Do not use sharded rw

  carsonip

  committed
  fc833fd

  Copy full SHA for fc833fd

• feat: update to badger v2

  kruskall

  committed
  3d1b41b

  Copy full SHA for 3d1b41b

Commits on Jan 23, 2025

• Reduce memtable size by half

  carsonip

  committed
  55e2de4

  Copy full SHA for 55e2de4

• Add FIXME

  carsonip

  committed
  ee69d3c

  Copy full SHA for ee69d3c

• Add FIXME

  carsonip

  committed
  2cc8de4

  Copy full SHA for 2cc8de4

• Move rw

  carsonip

  committed
  44cb0f0

  Copy full SHA for 44cb0f0

• Add partitioner

  carsonip

  committed
  82e102b

  Copy full SHA for 82e102b

• Use partitioner

  carsonip

  committed
  9edb4fc

  Copy full SHA for 9edb4fc

• Add comments

  carsonip

  committed
  d08369f

  Copy full SHA for d08369f

• Naming

  carsonip

  committed
  7cf2002

  Copy full SHA for 7cf2002

• Add prefix test

  carsonip

  committed
  d6a94bd

  Copy full SHA for d6a94bd

• Remove sharding completely

  carsonip

  committed
  69fff0d

  Copy full SHA for 69fff0d

• Rename to PartitionReadWriter

  carsonip

  committed
  505b8c1

  Copy full SHA for 505b8c1

• Add header

  carsonip

  committed
  05a6df4

  Copy full SHA for 05a6df4

• go mod tidy

  carsonip

  committed
  2c850cf

  Copy full SHA for 2c850cf

• Add storage limit handling

  carsonip

  committed
  e646428

  Copy full SHA for e646428

• Stop exposing storage

  carsonip

  committed
  1b993e1

  Copy full SHA for 1b993e1

• Remove some badger mentions

  carsonip

  committed
  8cf67b5

  Copy full SHA for 8cf67b5

• Pebble bug

  carsonip

  committed
  1e09fa3

  Copy full SHA for 1e09fa3

• Flush on sm.Close

  carsonip

  committed
  9b75264

  Copy full SHA for 9b75264

• Remove unused code

  carsonip

  committed
  4c409d3

  Copy full SHA for 4c409d3

• Remove badger mention

  carsonip

  committed
  e99da03

  Copy full SHA for e99da03

• Remove writerOpts and wrappedRW

  carsonip

  committed
  3abeccf

  Copy full SHA for 3abeccf

• Add example

  carsonip

  committed
  d39da8a

  Copy full SHA for d39da8a

• Add FIXME

  carsonip

  committed
  a41e3c9

  Copy full SHA for a41e3c9

• Clean up comments

  carsonip

  committed
  e4a0a0f

  Copy full SHA for e4a0a0f

• Add FIXME

  carsonip

  committed
  19bfccb

  Copy full SHA for 19bfccb

• Remove storage gc interval

  carsonip

  committed
  a1e444c

  Copy full SHA for a1e444c

• Rename to rotate partitions

  carsonip

  committed
  0b02728

  Copy full SHA for 0b02728

• Use a stable format

  carsonip

  committed
  2fe27f7

  Copy full SHA for 2fe27f7

• Add log

  carsonip

  committed
  917584e

  Copy full SHA for 917584e

• Improve comment

  carsonip

  committed
  9c0df8c

  Copy full SHA for 9c0df8c

• Improve log

  carsonip

  committed
  24297af

  Copy full SHA for 24297af

• Add FIXME

  carsonip

  committed
  80533b3

  Copy full SHA for 80533b3

• No need for merger

  carsonip

  committed
  8c71c8d

  Copy full SHA for 8c71c8d

• Add FIXME

  carsonip

  committed
  737f57b

  Copy full SHA for 737f57b

• Resume from current partition on restart

  carsonip

  committed
  68ed084

  Copy full SHA for 68ed084

• Use json

  carsonip

  committed
  dac6f1a

  Copy full SHA for dac6f1a

• Better storage limit reached error message

  carsonip

  committed
  8400483

  Copy full SHA for 8400483

• Test StorageLimitReadWriter

  carsonip

  committed
  b9725a2

  Copy full SHA for b9725a2

• Refactor

  carsonip

  committed
  e411486

  Copy full SHA for e411486

• Enable parallel writers

  carsonip

  committed
  64f0d78

  Copy full SHA for 64f0d78

• Remove storage limit threshold

  carsonip

  committed
  d5e10a4

  Copy full SHA for d5e10a4

• Change partitioner meta key

  carsonip

  committed
  9ff4008

  Copy full SHA for 9ff4008

Commits on Jan 24, 2025

• Improve comments

  carsonip

  committed
  9285fbc

  Copy full SHA for 9285fbc

• Remove StorageDir config

  carsonip

  committed
  74109a1

  Copy full SHA for 74109a1

• Remove errgroup

  carsonip

  committed
  9b14a25

  Copy full SHA for 9b14a25

• Change partitioner interface

  carsonip

  committed
  c2d62ed

  Copy full SHA for c2d62ed

• Remove writeEntry

  carsonip

  committed
  ed25987

  Copy full SHA for ed25987

• Add sanity check to total partitions

  carsonip

  committed
  7074ccf

  Copy full SHA for 7074ccf

• Fix partitioner interface

  carsonip

  committed
  32213d6

  Copy full SHA for 32213d6

• Bypass pebble Flush issue for now

  carsonip

  committed
  25d31ed

  Copy full SHA for 25d31ed

• Remove bounds when using SeekPrefixGE

  carsonip

  committed
  b990792

  Copy full SHA for b990792

• Remove RW Flush

  carsonip

  committed
  3bceb43

  Copy full SHA for 3bceb43

• Use iter.Seq, remove PartitionIterator

  carsonip

  committed
  767128f

  Copy full SHA for 767128f

• Remove superfluous err checks

  carsonip

  committed
  ce25f01

  Copy full SHA for ce25f01

• Add missing err check

  carsonip

  committed
  7d84395

  Copy full SHA for 7d84395

• Rename lb->b

  carsonip

  committed
  153db27

  Copy full SHA for 153db27

• Add comment to ReadPartitions and WritePartition

  carsonip

  committed
  a5fea7b

  Copy full SHA for a5fea7b

• Fix partitioner interface failing test

  carsonip

  committed
  cbce23d

  Copy full SHA for cbce23d

• Refactor Size

  carsonip

  committed
  8f367be

  Copy full SHA for 8f367be

• Add comment explaining partition buffer

  carsonip

  committed
  bb8d553

  Copy full SHA for bb8d553

• Merge branch 'main' into tbs-pebble-rebase

  carsonip

  committed
  f9a01da

  Copy full SHA for f9a01da

• make fmt update

  carsonip

  committed
  8ba4af0

  Copy full SHA for 8ba4af0

• make update

  carsonip

  committed
  5d97b1a

  Copy full SHA for 5d97b1a

• Workaround pebble Flush hang

  carsonip

  committed
  8264339

  Copy full SHA for 8264339

• Add missing pebble cleanup in test

  carsonip

  committed
  700ca26

  Copy full SHA for 700ca26

• Fix invalid pebble comparer

  carsonip

  committed
  449da8e

  Copy full SHA for 449da8e

• Change trace ID separator

  carsonip

  committed
  5d1ed19

  Copy full SHA for 5d1ed19

• make fmt

  carsonip

  committed
  e9fc733

  Copy full SHA for e9fc733

Commits on Jan 25, 2025

• Ensure correctness of DeleteRange

  carsonip

  committed
  b83e172

  Copy full SHA for b83e172

Commits on Jan 27, 2025

• Merge branch 'main' into tbs-pebble-rebase

  carsonip

  committed
  0e1763a

  Copy full SHA for 0e1763a

• Simplify DeleteRange calls and remove requirement to have separator

  carsonip

  committed
  0e4bfcf

  Copy full SHA for 0e4bfcf

• Remove BypassReadWriter

  carsonip

  committed
  00b3c15

  Copy full SHA for 00b3c15

• Update IsTraceSampled comment

  carsonip

  committed
  36c29d9

  Copy full SHA for 36c29d9

• Use RWMutex instead of atomics

  carsonip

  committed
  bb9ee60

  Copy full SHA for bb9ee60

• Add comment

  carsonip

  committed
  c84d04d

  Copy full SHA for c84d04d

• Update comments

  carsonip

  committed
  ed02357

  Copy full SHA for ed02357

• Add TODO

  carsonip

  committed
  72c247f

  Copy full SHA for 72c247f

• Turn FIXME into TODO

  carsonip

  committed
  2224b41

  Copy full SHA for 2224b41

• Update storage manager read test

  carsonip

  committed
  d25aa45

  Copy full SHA for d25aa45

• Add DiskUsage bench test

  carsonip

  committed
  7006e17

  Copy full SHA for 7006e17

• Cache disk usage

  carsonip

  committed
  1e066d8

  Copy full SHA for 1e066d8

• Skip check storage limit if there is no limit

  carsonip

  committed
  2535e6c

  Copy full SHA for 2535e6c

• Test storage manager run

  carsonip

  committed
  c27732f

  Copy full SHA for c27732f

• Better errors

  carsonip

  committed
  058d654

  Copy full SHA for 058d654

Commits on Jan 28, 2025

• Merge branch 'main' into tbs-pebble-rebase

  carsonip

  committed
  9da37fa

  Copy full SHA for 9da37fa

• Add doc.go

  carsonip

  committed
  f691e00

  Copy full SHA for f691e00

• Rename partitioner methods

  carsonip

  committed
  de566b0

  Copy full SHA for de566b0

• Return newInactive in Rotate

  carsonip

  committed
  e57a2d3

  Copy full SHA for e57a2d3

• activeIDs lock internally

  carsonip

  committed
  f68f765

  Copy full SHA for f68f765

• Change currentIDs to CurrentIDFunc

  carsonip

  committed
  34213e1

  Copy full SHA for 34213e1

• Update error for consistency

  carsonip

  committed
  7de54dd

  Copy full SHA for 7de54dd

• Merge branch 'main' into tbs-pebble-rebase

  carsonip

  committed
  899a2a0

  Copy full SHA for 899a2a0

• make fmt

  carsonip

  committed
  4fa8670

  Copy full SHA for 4fa8670

• Add concurrency test to partitioner

  carsonip

  committed
  0adcd8c

  Copy full SHA for 0adcd8c

Commits on Jan 29, 2025

• Reuse sampling decision byte slices

  carsonip

  committed
  35e5d0a

  Copy full SHA for 35e5d0a

• Add partitionerMeta struct

  carsonip

  committed
  3d3602b

  Copy full SHA for 3d3602b

• Move back TestStorageMonitoring

  carsonip

  committed
  e0857a5

  Copy full SHA for e0857a5

• Move meter provider to storage manager

  carsonip

  committed
  6f40324

  Copy full SHA for 6f40324

• Add comment to prefix rw

  carsonip

  committed
  f4822df

  Copy full SHA for f4822df

• Correct meter name

  carsonip

  committed
  84f1a78

  Copy full SHA for 84f1a78

• Add RW comment

  carsonip

  committed
  81cd610

  Copy full SHA for 81cd610

• Remove unused var

  carsonip

  committed
  d8d0b55

  Copy full SHA for d8d0b55

• Fix meter name again

  carsonip

  committed
  545a0e2

  Copy full SHA for 545a0e2

• Add comment

  carsonip

  committed
  31577e1

  Copy full SHA for 31577e1

• Use config.Storage not only for testing

  carsonip

  committed
  afd5994

  Copy full SHA for afd5994

• Close db once in main

  carsonip

  committed
  287fc4c

  Copy full SHA for 287fc4c

• Add changelog

  carsonip

  committed
  5cb24d7

  Copy full SHA for 5cb24d7