#54 lists "the docs site says the same things as the README" as an open item, with the note "they have drifted before". They had drifted again, and #66 closed it by hand.
The drift was not subtle and nobody had noticed: -serve-sts appeared in four docs pages and nowhere in the README — the flag that decides whether the KubeMicroVM operator starts at all, missing from the page that leads with running the operator.
It was found by diffing the runtime flags each surface names, not by reading:
rflags = {m.group(1) for m in re.finditer(r'`?-([a-z]+(?:-[a-z]+)+)`?', readme)}
for f in glob.glob('docs/*.md'):
for m in re.finditer(r'`-([a-z]+(?:-[a-z]+)+)`', open(f).read()):
if m.group(1) not in rflags:
print(f, m.group(1))That took a minute and will keep working. Reading will not: the item has now been closed by hand at least twice, which is the definition of a check that wants to be a job.
Ask
A CI check that fails when a surface names a runtime flag another does not. The flag set is the highest-value signal because a flag is a promise about how the binary behaves, but the same shape covers the image tags and the coverage numbers each surface quotes (29/29 operations, 100 conformance checks) — all of which have a single right answer and three places to say it.
scripts/smoke.sh is the precedent: the README walkthrough cannot rot because CI executes it. This is the same idea applied to the parts of the docs that are claims rather than commands.
#54 lists "the docs site says the same things as the README" as an open item, with the note "they have drifted before". They had drifted again, and #66 closed it by hand.
The drift was not subtle and nobody had noticed:
-serve-stsappeared in four docs pages and nowhere in the README — the flag that decides whether the KubeMicroVM operator starts at all, missing from the page that leads with running the operator.It was found by diffing the runtime flags each surface names, not by reading:
That took a minute and will keep working. Reading will not: the item has now been closed by hand at least twice, which is the definition of a check that wants to be a job.
Ask
A CI check that fails when a surface names a runtime flag another does not. The flag set is the highest-value signal because a flag is a promise about how the binary behaves, but the same shape covers the image tags and the coverage numbers each surface quotes (
29/29 operations,100 conformance checks) — all of which have a single right answer and three places to say it.scripts/smoke.shis the precedent: the README walkthrough cannot rot because CI executes it. This is the same idea applied to the parts of the docs that are claims rather than commands.