Repository navigation
Conversation
…overs the output The output directory was not ignored. report.json and dashboard.html both embed the full text of every post in a corpus, and .gitignore listed only *.jsonl and dashboard.html, so jev-out/report.json was one git add away from being published. Ignore the directory wholesale. CI runs the suite on Node 20 (the engines floor) and 22 (current LTS), plus a smoke job that validates every rubric pack. Nothing in CI needs a key: a green tick that depends on a repository secret is one contributors cannot reproduce and forks cannot run at all.
The gate that decides whether a finding is confirmed or dismissed as time
drift was corrupted whenever any post was dropped for a missing rating.
const control = subset.map((i) => new Date(i.date).getTime());
const pairs = subset.map(...).filter(([a]) => a != null);
partialSpearman(a, b, control.slice(0, pairs.length));
control.slice(0, pairs.length) is the FIRST n dates, not the dates OF the n
surviving posts. Drop one post and every date after it lands on the wrong
one. The array is the right length, so nothing downstream complains and the
partial correlation comes back a plausible number that is simply wrong.
This is the load-bearing path. survivesControl is what stops a dimension that
drifts over time from being reported as a result, and it was reading noise.
Fixed by carrying [value, outcome, date] through one filter, which makes the
misalignment unrepresentable rather than merely absent. Both the report layer
and the CLI table had the same bug.
Verified on a 178-post corpus: zero posts are dropped there, so every
correlation is byte-identical and no published number moves. The bug was
latent on that data and would have fired on any corpus with a partial rating.
Also adds corpusFacts() and cadenceOf() to the report, which the dashboard
needs: posting volume by year, follower trajectory, headline medians, and a
DERIVED description of whether volume rose or fell. Derived rather than
asserted, because the dashboard that motivated it hard-coded a heading saying
the author had stopped posting, which was true of exactly one corpus.
Five defects, all of which made the corpus look better or more complete than
it was.
Zero engagements was treated as missing. The guard was
impressions && engagements ? engagements / impressions : null
so a post that nobody engaged with produced conversion_rate: null and was
excluded from every correlation. That is not a missing measurement, it is the
clearest possible signal that the writing did not work, and dropping it biases
every finding upward by systematically removing the worst posts. Only a zero
DENOMINATOR now yields null.
Same-date collisions kept media. Posts are joined to analytics by date. On a
colliding day the code correctly refused to attach metrics, then attached
media type and count from the day's pooled rows anyway, so a video and a
text-only post published the same day both came back as video. All per-post
fields are now null on a collision. Followers stay, and the comment says why:
it is a per-day level, so two posts on one day genuinely share it.
Unparseable dates vanished. Four separate paths dropped rows with a bare
continue. A locale-shifted export dropped 100% of media rows and the only
symptom was every post reporting text_only. Drops are now counted per source
and reported with sample values.
The follower back-walk was unvalidated. It assumed the export's headline total
corresponds to the last row of the daily sheet, with hardcoded offsets for
both. A mismatch would have biased every reach_rate by a constant. Now checked
three ways, and on failure the follower fields are nulled rather than trusted.
Posts older than the follower window received the earliest known count. A
guess presented as a measurement. They now receive none.
Also: the permutation seed is overridable and reported instead of a hardcoded
default no caller could reach, so a run is auditable and seed sensitivity can
be tested. The default is unchanged and pinned by a literal assertion.
…te one
The skill told agents to hand-build an HTML dashboard from a design brief,
every time, from scratch. The README described it as a capability. Neither
was backed by code.
buildDashboard(report, { ratings, pack, platform }) returns one self-contained
document: inline CSS, inline JS, data embedded, no CDN and no fetch, so it
opens over file:// on a machine with no network. 178 posts render to ~600 KB.
Wired as `jev-writer dashboard`, and into `run`.
The presentation rules are the reason it exists. An earlier version reported
rank correlations and p-values and was correctly called unreadable by its
first reader. So: no statistics vocabulary outside one collapsed methods
section, every judgment as a coloured band rather than a number, bands
arriving direction-aware from the report so low-is-good still paints green,
and charts only where a chart says something.
This was ported from a private analysis repo, where every sentence was written
against one person's data. Generalising it was most of the work:
- the volume heading asserted that the author had stopped posting. Now
derived from cadenceOf(), with four branches, so an author whose output
grew is told that instead.
- the methods text hardcoded "30 questions", "Sixteen", "fourteen". Now
counted from the pack.
- "LinkedIn" appeared in three strings. Now a parameter.
- question families were a hardcoded list of 30 dimension keys. Now declared
by the pack, with a catch-all so an unknown pack renders on day one
instead of silently dropping half its questions.
- it re-read posts.jsonl to join fields the report already carried.
The LinkedIn pack declares those five families here. Presentation only:
grouping never touches how a question is asked, rated or correlated.
AGENTS.md calls report.mjs the most important file to understand. It had 442
lines, 13 exports and zero tests. The dashboard had none either.
27 tests to 67. What the new ones pin:
- the power refusal. Given a perfect, noiseless correlation on 12 posts, the
report must produce no findings and no writing prompt. This is the single
behaviour the project's claim rests on, and nothing was checking it.
- direction of merit. The same raw value bands as excellent or poor
depending on the declared direction, and neutral dimensions never band as
either. Getting this wrong paints the worst posts green.
- the quintile artifact. Bands are quintiles of the author's own
distribution, so counting posts in the bottom two bands gives 40% for
every dimension by construction. That shipped once as a weakness measure.
The test asserts a weak spot is a gap between two real medians instead.
- band labels carry no statistics vocabulary, and labelFor never emits the
"the MOST is slop" shape that an earlier generated-label scheme produced.
- the dashboard is genuinely self-contained: no external script, no
stylesheet, no CDN host anywhere in the output.
- post text is escaped rather than injected.
- the volume narrative is derived. A corpus whose output grew must not be
told its posting fell away.
One fixture was wrong on the first pass: both dimensions were linear in the
loop index, so identical gap shares were the correct answer and the artifact
test proved nothing. Decorrelated.
The README documented
import { score, bool } from 'jev-writer/rubrics/pack.mjs';
which never resolved. With no exports map Node falls back to legacy subpath
resolution, so that specifier points at <pkg>/rubrics/pack.mjs while the file
lives at <pkg>/src/rubrics/pack.mjs. The documented import would have thrown
ERR_MODULE_NOT_FOUND.
An exports map fixes the path and does a second job: it makes the listed
subpaths the stable contract and hard-blocks everything else, so
src/providers/gateway.mjs stops being reachable by accident. src/index.mjs
re-exports the same surface for anyone who wants one import.
The lockfile still carried "name": "content-signal" from before the rename,
which meant npm ci could reject it as out of sync. Regenerated, and CI drops
the install fallback it needed while that was true.
Also: files no longer lists an empty docs directory, keywords widened toward
the vocabulary this actually gets searched with (llm-as-a-judge, evals,
false-discovery-rate), and publishConfig set for when it goes to npm.
Adds the two sections a stranger needs first and the repo did not have: what
you can do with it (six concrete uses, each naming the command that does it,
and an honest statement of who it is not for) and a features table.
Corrects three things the README claimed that the code did not do:
- npx jev-writer fails. The package is unpublished, so every quickstart line
404'd for every reader, including the two in the skill. Replaced with
npx github:Kaos599/jev-writer, verified working, plus the clone path.
- output lands in jev-out/, not jev-writer-out/. An agent following the old
instructions would cat a file that does not exist.
- the rubric import path was wrong. Now matches the exports map.
New sections for the dashboard and the generated writing prompt, both of which
now exist. Getting started is a four-step walkthrough ending in what each
output file is and which ones must never be committed.
The worked example under the false-discovery section is this project's own
result and is not flattering: thirty dimensions, 43 posts with reach data, ten
relationships at p < 0.05, exactly one surviving correction, and that one
tagged exploratory so the tool declined to promote it. Nine of ten were inside
the noise budget that 48 tests buys. If the README is going to claim the
statistics are adversarial toward their own conclusions, it should show them
being adversarial toward this repo's.
Plus a changelog.
The skill's step 5 was fifty lines instructing an agent to hand-write a
dashboard from a design brief, every time, and its frontmatter advertised the
result as a capability. Replaced with: run the command, then read the page
honestly. The design brief survives as the rationale for why the shipped page
looks the way it does, and as the spec for anyone building a custom one.
Corrections, each verified against source rather than assumed:
- every npx jev-writer invocation 404'd, because the package is unpublished.
The skill's step 3 was the first command an agent ran.
- jev-writer-out/ does not exist. The directory is jev-out/. An agent
following the old instructions would cat a file that was never written.
- the claim that validatePack enforces a direction of merit was false. It
checks tiers and never looks at direction. Enforcement is at runtime in
two places, and the failure mode is worth knowing: a structurally valid
pack can rate an entire corpus and then produce no report at all.
- interpreting-results.md described a report shape that had drifted from
what buildReport returns.
- platform-exports.md now says plainly that only LinkedIn ships an adapter.
It read as a list of supported platforms; it is a list of candidates.
- ZDR is requested on the gateway transport only. The other two had no
equivalent and the doc implied otherwise.
AGENTS.md records both statistical bugs fixed today in mechanism terms, with
the generalised rules: never reintroduce parallel arrays filtered in step, and
test Number.isFinite rather than truthiness on anything that can be zero.
Also fixes my own arithmetic in the README. The pack has 30 questions but 6
are Choice and are never correlated, so it is 24 rated dimensions against 2
outcomes, not 30.
CRITICAL. The dashboard reported "0.0% reach" when there was no engagement data at all. null * 100 is 0 in JavaScript, so the percentage formatter turned "we cannot measure this" into "your posts got zero reach", stated with the same confidence as a real number. Reachable whenever no analytics export is supplied, which is a case the adapter already warns about. A fabricated finding is the one output this project exists to prevent, and it was on the hero card. Now renders "No data". HIGH. cadenceOf picked the peak year with a strict >, so the FIRST of two equal years won and the later one was averaged into the "other years" figure. An author whose latest year exactly matched their best year was told they were publishing less. That is the precise false accusation the function was written to avoid, reintroduced through a comparison operator. HIGH. Benjamini-Hochberg corrected over the dimensions that could actually be tested, while the power headline announced dimensions x outcomes. Where outcome coverage is thin the two diverge, and a reader told 48 tests were corrected could have been looking at a correction over six. The family size is now computed once, from the same guards, and the number announced is the number corrected over. Both the report layer and the CLI table. HIGH. buildDashboard accepted a report.json written before corpusFacts existed and then threw an opaque TypeError on the first property access. It now says which file is stale and which command to re-run. MEDIUM. The README said `run` performed five steps. It performs four; the CLI's own help text was already correct and the README disagreed with it. MEDIUM. Code features are ranked against the rubric in the analyze table but not in report.json, so the two correct over different families and can disagree. That was presented as one uniform behaviour. Now stated plainly, with which artifact to read for which question. LOW. The statistics-vocabulary test anchored on markup that does not exist and passed only through a fallback. A wording change to one heading would have made it slice the whole document as "the main surface" and pass vacuously. It now fails loudly if the anchor moves. Three regression tests added, 67 to 70.
…dening feat: ship the dashboard, and fix two bugs that were reporting wrong numbers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes everything merged into
devtomain.Contents are the ten commits reviewed in #1: the dashboard command, two live statistical bugs fixed, seven defects from an adversarial review, CI, the test suite grown from 27 to 70, a public
exportssurface, governance files, and a documentation pass that removed every claim the code did not support.The two that changed behaviour rather than shape:
control.slice(0, pairs.length)misaligned publication dates with values whenever a post was dropped for a missing rating, silently corrupting the gate that dismisses a finding as time drift.And the one the review caught that mattered most: the dashboard rendered "0.0% reach" where the honest answer was "no data", because
null * 100is0.CI passed all three jobs on #1:
npm teston Node 20 and 22, and the CLI plus rubric-pack smoke job.