Blog: Your Competitor Set Is Wrong, and So Are Their Numbers - #323
Open
jeremypenguin-max wants to merge 1 commit into
Open
Blog: Your Competitor Set Is Wrong, and So Are Their Numbers#323jeremypenguin-max wants to merge 1 commit into
jeremypenguin-max wants to merge 1 commit into
Conversation
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R5c56XeUKSKJSaVjPU2Jpc
ywwenjin163
pushed a commit
to ywwenjin163/open-seo
that referenced
this pull request
Sep 12, 2026
…ones (every-app#323) DataForSEO dropped the Claude Sonnet 4.0 family from its llm_responses catalog, so model_name=claude-sonnet-4-0 was rejected with 'Invalid Field: model_name' while still billing the failed task. Point Claude at claude-sonnet-4-5 and validate every model_name against DataForSEO's accepted catalog before dispatching the paid call.
ywwenjin163
pushed a commit
to ywwenjin163/open-seo
that referenced
this pull request
Sep 12, 2026
…ges-table polish (#367) * Site audit P0: issue engine, incremental persistence, block detection Implements the P0 feature set from docs/site-audit-pm-research.md: - Issue engine: 24 issue types (shared registry with severity, explanation, how-to-fix). Per-page reporters run inside crawl steps; cross-page checks (duplicate titles/descriptions/content, broken internal links, redirect chains/loops, orphan pages) run at finalize as SQL over the persisted crawl. - New audit_links + audit_issues tables, audit_pages columns (depth, content hash, header signals, fetch class, sitemap flag); audit tables moved to src/db/audit.schema.ts. - Incremental persistence: pages/links/issues written to D1 inside each crawl-batch step with deterministic row ids + upserts (retry idempotent); slim step state; robots.txt checkpointed as step state for deterministic replay; merged progress steps keep a 10k-page crawl within the Workflows step budget. - Crawler: manual redirect handling with inline follow of normalization-equivalent redirects (slash-canonical sites), response header capture (X-Robots-Tag, Link rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered links, honest "we were blocked" classification (403/429/cf-mitigated/challenge). - UI: Issues tab (default) with severity grouping, per-type explanations, drill-down, CSV/JSON/Sheets export, blocked banner. - MCP: run_site_audit, get_audit_status, get_audit_issues (severity- sorted, how_to_fix per issue), get_audit_pages. - Lighthouse strategies reduced to auto/none (legacy all/manual map on read); auto stays 10 URLs x 2 = 20 checks. - Self-healing: getStatus reconciles audits whose workflow instance errored/terminated without reaching mark-failed. Deploy notes: run db:migrate:prod (additive migration 0022); terminate running audits before deploying - the workflow step structure changed and in-flight instances cannot replay under the new code (a finalize guard fails them loudly instead of completing empty). * feat(onboarding): hide agent chat step; subscribe after intro steps (every-app#312) * feat(onboarding): hide agent chat step; subscribe after intro steps Remove the hosted-only strategy-chat diversion from the onboarding sequence. After the three intro questions, hosted users now hit the subscribe paywall directly, then return to the GSC and MCP connect steps. The chat route and components stay in place but unlinked, to be revisited later. Preserve the post-payment 'You're in!' interstitial by carrying checkout=success through validateSearch. * fix(onboarding): set checkout=success from subscribe route, not speculatively The previous redirect baked checkout=success into the onboarding return URL at the point needsSubscription is true — i.e. before the user had paid. It only worked because the subscribe route gates its redirect on actual access. Move the marker to the subscribe route's redirect-to-app path, where checkoutCompleted reflects a real returned-from-Stripe payment, so the 'You're in!' screen can never show pre-payment. * website: change link * fix(rank-tracking): unarchive config when re-adding an archived domain (every-app#313) * Unify dual-backend DB layer (D1 default + Postgres opt-in) (every-app#238) * D1 → Postgres data migration (ETL + runbook) (every-app#274) * Fix Postgres-only rank-tracking & site-audit workflow failures (every-app#317) * rank-tracking: raise per-project config limit from 20 to 100 (every-app#318) The cap was only a soft guard against runaway scheduled DataForSEO workload, not a hard product constraint. Bump it to 100 so projects tracking many domain/location combos aren't blocked. Co-authored-by: Claude <noreply@anthropic.com> * fix(db): add missing indexes and drop redundant ones (every-app#319) Postgres advisor flagged seq-scans and redundant indexes across both backends (D1 + Postgres): - add projects(organization_id) — org-scoped project listings seq-scanned - add account(account_id, provider_id) — better-auth sign-in lookup - add verification(expires_at) — expired-token cleanup range scan - drop saved_keyword_tag_assignments_keyword_idx — covered by unique (saved_keyword_id, tag_id) prefix - drop rank_snapshots_run_idx — covered by unique (run_id, tracking_keyword_id, device) prefix Mirrored in both schema dialects + parity-test required-index guard. * refactor(keywords): unify keyword-metric fetching behind one helper (every-app#320) * Fix production errors: onboarding crash hardening + DataForSEO spend/noise cleanup (every-app#282) * fix(ai-search): use valid Claude model_name and fail fast on unknown ones (every-app#323) DataForSEO dropped the Claude Sonnet 4.0 family from its llm_responses catalog, so model_name=claude-sonnet-4-0 was rejected with 'Invalid Field: model_name' while still billing the failed task. Point Claude at claude-sonnet-4-5 and validate every model_name against DataForSEO's accepted catalog before dispatching the paid call. * fix(mcp): 405 the standalone GET SSE stream to stop /mcp OOM (every-app#325) The stateless MCP server returns JSON on POST (enableJsonResponse) and pushes no server-initiated messages, so the optional standalone GET SSE stream serves no purpose. Left enabled, each GET holds an SSE stream open indefinitely (25s keepalive, no eventStore) and pins a fresh per-request McpServer (~5MB of tools + Zod schemas); a few dozen concurrent connected clients exceed the 128MB isolate limit. This was 100% of the /mcp exceededMemory OOMs (GET only; POST never OOMed). Return 405 (spec-compliant 'no standalone stream') before building the server, so GET allocates nothing. Also removes the bulk of the elevated GET canceled / responseStreamDisconnected outcomes. * Re-add free plan as the floor; remove subscribe gate (every-app#321) * Pin production to Postgres via committed Hyperdrive binding (#329) * Add Cloudflare Turnstile captcha on email signup (#326) * Triage production log errors: audit crash, Autumn webhook FK, PostHog capture, auth rate-limit IP, log noise (#327) * Add badseo.dev: a test site of deliberate SEO mistakes An open-source Cloudflare Worker that serves ~27 pages, each breaking one common technical-SEO rule (missing title, redirect loop, orphan page, thin content, and so on). It doubles as the end-to-end fixture for the OpenSEO site audit: every page declares the audit issues it should trigger, and scripts/run-audit.ts drives the real audit engine against a running copy to check that it does (36/36 checks, 25/25 issue types). Styled to match the OpenSEO marketing site (web/). Maintained-by-OpenSEO badge links back to openseo.so. * badseo.dev: logo in pill, footer/hover polish, SEO-optimized titles - Use the OpenSEO pine-tree logo (downscaled, base64-embedded, served at /openseo-logo.png) in a light chip inside the badge, replacing the ◎ glyph. - Footer band now fills to the bottom of the page (dropped the mismatched body padding strip) with room for the floating badge. - Index rows: remove the stray full-row underline and the stark white hover box; hover is now a soft cream tint with the name underlined. - Drop the "Maintained by OpenSEO" hero eyebrow; new H1 "A website demonstrating common technical SEO problems" and a cleaner subtitle. - Optimize homepage + catalog <title>/meta around real keywords from OpenSEO keyword research (technical seo issues KD25/vol170; technical seo checklist KD16/vol390), keeping meta lengths within limits. * Site audit P0 (1/3): issue engine, incremental persistence, block detection Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md: - Issue engine: shared registry of issue types (severity, explanation, how-to-fix). Per-page reporters run inside crawl steps; cross-page checks (duplicate titles/descriptions/content, broken internal links, redirect chains/loops, orphan pages) run at finalize as SQL over the persisted crawl. - New audit_links + audit_issues tables, audit_pages columns (depth, content hash, header signals, fetch class, sitemap flag); audit tables moved to src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG). - Incremental persistence: pages/links/issues written inside each crawl-batch step with deterministic row ids + upserts (retry idempotent); slim step state; robots.txt checkpointed as step state; merged progress steps keep a 10k-page crawl within the Workflows step budget. - Crawler: manual redirect handling with inline follow of normalization- equivalent redirects, response header capture (X-Robots-Tag, Link rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered links, honest 'we were blocked' classification (403/429/cf-mitigated/ challenge). - MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages; limitTier resolved via shared AuditService.resolveAuditLimitTier. - Lighthouse strategies reduced to auto/none (legacy all/manual map on read). - Self-healing: getStatus reconciles audits whose workflow instance errored/ terminated without reaching mark-failed. The Issues UI and the badseo.dev e2e fixture site stack on top of this PR. Deploy notes: run db:migrate:prod (additive); terminate running audits before deploying — the workflow step structure changed and in-flight instances cannot replay under the new code (a finalize guard fails them loudly instead of completing empty). * Site audit P0 (2/3): Issues tab UI - Issues tab (new default) with severity grouping, per-type explanations and how-to-fix, drill-down to affected pages, CSV/JSON/Sheets export, and the 'we were blocked' banner when the crawl was challenged. - Tabs always render (Issues/Pages, Performance when Lighthouse ran); audit route search schema gains the issues tab and defaults to it. Stacks on claude/audit-p0-server (issue engine + persistence). * badseo.dev: render the badge logo as a white tree, no chip The silver source logo was invisible on the dark pill, so it sat in a white chip. Render it white via a CSS filter instead, so the tree fills the pill with no backing background. * badseo.dev: add build (typecheck) step before deploy - Add 'build'/'typecheck' scripts (tsc --noEmit); 'deploy' now runs the build before wrangler deploy. - Scope the tsconfig typecheck to the Worker source (src/); the e2e harness in scripts/ imports the main app and is run with tsx from the repo root. - Document the deploy flow and first-time custom-domain setup in the README. * badseo.dev: add trailing-slash redirect-cycle fixture + regression test Reproduces the 508 "Loop Detected" class of bug from every-app#61: a CMS-style page whose canonical URL ends in a trailing slash, with the non-slash form 301-redirecting to it. A crawler that strips trailing slashes turns the canonical /foo/ back into /foo, follows the 301 to /foo/, strips it again, and loops. - New fixture at /redirect/trailing-slash: the non-slash form (intercepted in index.ts on the raw path) 301s to the slash form, which is served as the canonical 200. - Harness asserts the page is crawled exactly once as a 200 with NO redirect loop, plus a dedicated "Trailing-slash cycle -> 200, no loop" guard. Verified the guard bites: temporarily disabling crawlPage's slash-canonical inline-follow makes both checks fail (redirect-loop, status 301); with it in place the harness is 38/38, 25/25 issue types. * Add webapp-testing skill (installed via /reload-skills) Vendors the anthropics/skills webapp-testing toolkit: real files under .agents/skills/webapp-testing, a symlink from .claude/skills/, and skills-lock.json pinning the source + hash. Matches how the other project skills are tracked. * Site audit: redesign issues tab as grouped table + calmer page header - Issues: single bordered table with severity sections (Critical/Warning/Info headers carry the counts), dot indicators instead of filled pills, plain right-aligned page counts, all rows collapsed by default; expanded rows get a severity-colored left rule - Removed the dead severity-count chips (they looked like filters but were inert spans) - Header: audited hostname is now the H1 with the status badge inline - Blocked banner: compact tinted panel instead of a full-size alert - Stats: hairline strip instead of four separate cards; issues stat shows a severity breakdown, Lighthouse tile hidden when no tests ran, dropped the orange issues-count coloring * audit: fix trailing-slash redirect cycle at the root (preserve slashes) Replaces the crawlPage inline-follow workaround with the root-cause fix, so we don't carry two fixes for the same bug (every-app#61). - normalizeUrl: stop stripping trailing slashes. A trailing slash is the canonical form on most CMSes, which 301 the non-slash version to it. Stripping rewrote the canonical URL into its own redirect source and looped (508). Now /path and /path/ are distinct and the redirect resolves normally. - crawlPage: remove the isSelfAfterNormalization inline-follow (+ now-unused resolveRawUrl). With slashes preserved it's dead code; a trailing-slash redirect is recorded as an ordinary hop. - add canonicalUrlKey (www/http/https-tolerant) and use it for the Lighthouse homepage match, which had the same redirect-mismatch vulnerability. - tests: preserve-trailing-slash + canonicalUrlKey unit tests; badseo harness guard is now fix-agnostic (canonical resolves to 200, no loop/error). Verified: 36 audit unit tests pass, tsc clean, badseo e2e 38/38. Reintroducing stripping makes the trailing-slash guard fail (redirect-loop), confirming the regression guard bites. * Audit: add no-outgoing-links + meta-description-too-short checks, catch empty H1s Two checks Ahrefs covers that we didn't, plus a fix: <h1></h1> now counts as missing. badseo.dev gains fixtures for all three (41 checks, 27/27 issue types covered). * Audit pages table: honest redirect/non-HTML rows, wrapped titles - 3xx rows show their redirect target (dim →) instead of a red 'missing' title, and dash out H1/Words/Images since nothing was analyzed - red 'missing' only when the engine actually flagged missing-title, so 200 non-HTML files (security.txt) read as blank, not broken - URL cells include the host when it differs from the audited site's, so apex→www redirect sources no longer render identically to their target - titles wrap to two lines (line-clamp) in a wider column instead of truncating at 220px; PagesTable moved to its own file (lint max-lines) * Audit pages table: canonical-host display, URL default sort, full title wrap - host prefix now compares against the site's predominant 2xx host, not the typed start URL — auditing apex 12port.com no longer prefixes every www row with the host - default sort by URL so the table opens as a site inventory instead of leading with redirects on error-free sites - titles wrap fully instead of clamping at two lines; long titles are the thing being audited, so their tails shouldn't be hidden * ci: exclude vendored skills from prettier; format test file --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Topical article in the same vein as "Wrong Half of Your Audience" (#185): the why behind the Competitive Analysis library (#237), written for a founder who has a competitor list and a quarter to plan. Links into all four strategies, so the library gets a blog-side entry point.
What is in it
web/content/blogs/competitor-set-is-wrong.md, ~2,100 wordsweb/public/blog/competitor-set-is-wrong/, plus one reused from the library folderThe four corrections
find_serp_competitorsinstead of naming it.Every number came out of OpenSEO. Same runs as #237: the national restoration brand with 7,587 pages against 49,729 keywords, and the five mold keywords each reporting an identical 74,000 volume and 22,496 traffic estimate. Screenshots are real app screenshots cropped to the data, account chrome removed.
Quotes. Topic-led and multi-guest: Ann Smarty, Christopher Gimmer, and the AI visibility panel. Verbatim from interview transcripts. Not about any one guest or their company.
Checked before opening. Ran the repo's
openseo-review-web-contentanddeslopstandards. No em dashes.## Table of Contentspresent,/docs/mcplinked, one copy-paste MCP prompt. Every internal route and external URL verified resolving today. Prettier clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01R5c56XeUKSKJSaVjPU2Jpc