Skip to content

Conversation

@rnewson
Copy link
Member

@rnewson rnewson commented Oct 23, 2025

Overview

Use purge more efficiently by building up a queue and then issuing a batch at once. There is a minimum batch size (for efficiency) and a maximum (to prevent heavily conflicted documents forcing a very large purge request).

Testing recommendations

eunit

Related Issues or Pull Requests

#5655

Checklist

  • Code is written and works correctly
  • Changes are covered by tests
  • Any new configurable parameters are documented in rel/overlay/etc/default.ini
  • Documentation changes were made in the src/docs folder
  • Documentation changes were backported (separated PR) to affected branches

@rnewson rnewson force-pushed the auto-purge-batching branch from 646a0e9 to 8a825f9 Compare October 23, 2025 16:54
@rnewson rnewson requested a review from nickva October 23, 2025 16:54
@rnewson rnewson force-pushed the auto-purge-batching branch from 8a825f9 to 07542fd Compare October 23, 2025 20:28
Copy link
Contributor

@nickva nickva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely done! +1

See the discussion about maybe increasing the default min size a bit if you want, and about the duplicate doc ids in a batch.

Otherwise, it seems to work nicely. I pulled all rate limiting stops and on a local 3 node cluster could purge faster than we can insert docs:

Config

    'couch_scanner_plugins' : {
        'doc_write_rate_limit': 10000,
        'db_rate_limit': 1000,
        'doc_rate_limit': 10000,
        'shard_rate_limit': 1000,
    },
    'couch_auto_purge_plugin': {
         'min_batch_size': 2,
         'max_batch_size': 10
    },

50k docs, Q=8, 100% deletions (-x 1.0), -t use the plugin to purge:

./conflicts.py -a adm:pass -q 8 -n 50000 -x 1.0 -z -t
*** old dbs prefixed with bench_db
 * deleting old db bench_db_1761279606
 * Fill:  40000ld dbs
 * Fill:  50000ench_db_1761279829
*** filling 50000 total docs and 50000 deleted docs, took 108 sec, rate = 464/sec
 *** triggering auto-purge ***
 * Fill:  30000
*** db_info:  {'del_count': 28140, 'doc_count': 0, 'sizes': {'file': 95336140, 'external': 249988, 'active': 33806191}}
 * Fill:  50000
*** filling 50000 total docs and 50000 deleted docs, took 96 sec, rate = 519/sec
 *** triggering auto-purge ***
*** db_info:  {'del_count': 24719, 'doc_count': 0, 'sizes': {'file': 96995451, 'external': 217362, 'active': 31123018}}
*** db_info:  {'del_count': 13388, 'doc_count': 0, 'sizes': {'file': 102004344, 'external': 135480, 'active': 23621708}}
*** db_info:  {'del_count': 5432, 'doc_count': 0, 'sizes': {'file': 106640938, 'external': 43546, 'active': 15371275}}
*** db_info:  {'del_count': 0, 'doc_count': 0, 'sizes': {'file': 108787192, 'external': 0, 'active': 11549485}}
*** plugin purged 50000 deleted docs in 48 sec, rate = 1038/sec

@rnewson rnewson force-pushed the auto-purge-batching branch 4 times, most recently from 7e9fad6 to 513d6cd Compare October 24, 2025 16:10
@rnewson rnewson force-pushed the auto-purge-batching branch from 513d6cd to e1313df Compare October 24, 2025 20:08
@rnewson rnewson merged commit 784574b into main Oct 24, 2025
51 checks passed
@rnewson rnewson deleted the auto-purge-batching branch October 24, 2025 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants