Skip to content

mempalace status silently truncates at 10,000 drawers (incorrect count displayed) #850

@vnguyen-lexipol

Description

@vnguyen-lexipol

Description

mempalace status reports the wrong drawer count on palaces with >10,000 drawers. No error, no warning — it silently displays 10000 drawers regardless of the actual count.

Root cause

miner.py:status() uses col.get(limit=10000, include=["metadatas"]) to fetch all drawer metadata, then displays len(metas) as the count. ChromaDB caps the result at 10,000 rows.

# miner.py line ~408
r = col.get(limit=10000, include=["metadatas"])
metas = r["metadatas"]
# ...
print(f"  MemPalace Status — {len(metas)} drawers")  # always ≤ 10000

Impact

  • Displayed count is wrong. My palace has 122,686 drawers; status showed 10,000.
  • Per-wing/room breakdown is incomplete. Only the first 10k drawers (by ChromaDB's internal ordering) are counted — some wings are entirely missing from the output.
  • No warning that results are truncated. Users assume the count is accurate.

Fix

Paginate with offset and use col.count() for the total:

total = col.count()
metas = []
batch_size = 10000
offset = 0
while offset < total:
    r = col.get(limit=batch_size, offset=offset, include=["metadatas"])
    batch = r["metadatas"]
    if not batch:
        break
    metas.extend(batch)
    offset += len(batch)

# ...
print(f"  MemPalace Status — {total} drawers")

Tested on a 122,686-drawer palace — correctly displays the full count and complete wing/room breakdown.

Related

Environment

  • mempalace 3.0.14
  • chromadb 0.6.3
  • Python 3.13
  • macOS

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/cliCLI commandsbugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions