Persist the unfinished frontier so --resume resumes (issue #36)

[tamnd]

Fixes the main defect in #36. The CDP error in that issue's title was already fixed in eb683dd the day after it was filed; this is the aside underneath it, which turned out to be the more serious of the two.

What the reporter said

kage clone paulgraham.com --refresh multiple times helped to scrap more, but we should have a memory of what failed and an option like --continue to get the missing ones.

--resume is already on by default, so asking for --continue is the tell. They had to reach for --refresh because resume does not resume.

What was happening

state.json held only the visited set:

type state struct {
	Visited []string `json:"visited"`
}

The pending frontier lives in the cloner's channels and was never written out, and load seeded both visited and seen from what it read.

Follow a resumed run from there. The only URL enqueued at startup is the seed, plus anything from sitemap.xml. On a resumed run the seed was already written, so isVisited(key) is true, enqueuePage returns false, and nothing is queued. The frontier is otherwise rebuilt purely by re-rendering pages and following their links, which resume guarantees never happens.

So the run printed its summary and exited successfully having done no work at all. On a 5,000 page site interrupted at page 500, restarting got you nothing, and the exit code and the output both said everything was fine.

Only sites with a sitemap.xml appeared to resume, since those URLs are seeded independently on every run. That is presumably why this survived eleven releases. --refresh worked because cloner.go:131 deliberately skips loading the state, so the seed enqueues normally and everything is re-rendered from scratch. That is exactly what the reporter observed and reasonably misread as "helped to scrap more".

The README promises the opposite:

Hit Ctrl-C and it saves its place on the way out; run it again and it picks up where it stopped.

Reproduced on main against a local 8 page chain site:

$ kage clone 127.0.0.1:8732 -p 3        # stop partway
  pages 3

$ kage clone 127.0.0.1:8732             # resume
  pages 0                               # and it never gets any further

The fix

Save the unfinished frontier next to the visited set and re-queue it at startup.

Each entry carries its depth, because --max-depth is measured from the seed and a resumed run has no way to recompute it. A state file written by an older kage has no pending key, loads fine, and resumes with nothing outstanding, which is exactly what that run recorded.

A page is pending from the moment it is offered until it is written, which falls out nicely:

• an interrupted render carries over, which is the bug above
• a page held back by --max-pages carries over, so -p 20 to look at a site and then a plain run to finish it is now a workflow rather than a dead end
• a page that failed carries over and the next run retries it. That is the "memory of what failed" the issue asked for. Until now the only way to pick up a failure was --refresh, which re-renders everything.

A page robots.txt disallows is marked done rather than left pending, since a later run would only fetch robots.txt and skip it again.

The run also says what it is leaving behind, so a short run stops looking like a finished one:

resume: 137 pages already done
resume: picking up 412 pages
...
resume: 89 pages still to do, rerun to continue

Verified

Same site, with the fix:

$ kage clone 127.0.0.1:8732 -p 3
  pages 3
  resume: 1 page still to do, rerun to continue

$ cat 127.0.0.1/_kage/state.json
{ "visited": ["index.html", "p1.html/index.html", "p2.html/index.html"],
  "pending": [{"url": "http://127.0.0.1:8732/p3.html", "depth": 3}] }

$ kage clone 127.0.0.1:8732 -p 3   # pages 3
$ kage clone 127.0.0.1:8732        # pages 2, all 8 written
$ kage clone 127.0.0.1:8732        # pages 0, genuinely nothing left

New tests:

• TestCloneResumeFinishesTheCrawl drives real Chrome over a five page chain, stops the first run partway, and asserts the resumed run walks the rest and that a third run then has nothing to do. This fails on main.
• TestCloneResumeRetriesFailures drops the connection on one page during the first run, brings it back for the second, and asserts the page is retried. The connection is dropped rather than answered with a 5xx because Chrome renders an error page happily.
• TestFrontierPersistsUnfinishedWork, TestFrontierMarkDoneDropsWork and TestFrontierLoadsStateWrittenBeforePending cover the state file without Chrome.

TestCloneResumeSkipsVisited still passes unchanged: after a complete crawl there is nothing pending, so a resumed run correctly does nothing.

go build ./..., go vet ./... and the full go test ./... pass with Chrome present.

Also worth knowing

The rest of #36 is not addressed here and should be split off:

• The Object reference chain is too long error in the title was fixed in eb683dd on 2026-06-17, the day after the report. The reporter was never told, and the issue is still open. Worth one comment and a question about whether 161 occurrences on one site still reproduce, since that number is not random.
• "Errors in toast messages and bad format, unusable as is" is unactionable as written. If they meant the saved pages rather than the terminal output, Keep the doctype so saved pages are not quirks mode (issue #16) #78 may have covered it.