Define and enforce a privacy-conscious profile indexability policy for sitemap.ts

[chonilius]

Problem

src/app/robots.ts and src/app/sitemap.ts exist, implying the app is meant to be publicly indexable — makes sense for /reputation/[handle] (public contributor profiles) and /issues (public bounty board) to drive discovery/SEO for a marketplace product. But /reputation/[handle] renders real usernames, avatar URLs, GitHub organizations, and lifetime earnings figures pulled via fetchReputationByUsername. If sitemap.ts enumerates every known handle (likely, to be useful for SEO) by calling the backend for a full user list, this creates a public, crawlable, permanently-indexed-by-search-engines directory of exactly who's earning what on the platform tied to their real GitHub identity — a privacy/doxxing-adjacent concern for contributors who may not have anticipated their MergeFi earnings being one Google search away from their real name, and a strategic problem if sitemap.ts calls the live backend at build/request time with no pagination or rate limiting, since a large user base could make sitemap generation slow or produce a sitemap exceeding the 50,000-URL/50MB per-file limit search engines enforce.

Why this is hard

• Requires reading sitemap.ts to determine its actual current data source (mock data only? live backend list?) and how it paginates (or doesn't) — undocumented currently.
• This is as much a product/privacy policy decision as an engineering one: should earnings be public at all, should indexability be opt-in per contributor, should the sitemap only include handles who've explicitly enabled a "public profile" flag (does AuthUser/backend even support such a flag today — check the backend or flag as a needed cross-repo addition)? You need to make and justify a reasonable default given no explicit product spec exists, then implement it.
• If opting toward "indexable by default," at minimum the lifetime-earnings figure and any other sensitive-feeling fields need a considered decision on whether they belong in SEO-crawlable server-rendered HTML at all, versus client-fetched-and-gated — this affects src/app/reputation/[handle]/page.tsx's rendering strategy, not just the sitemap.
• Sitemap generation for a potentially large user base needs to respect the sitemap size/count limits with a sitemap index file if the user count could exceed single-file limits, and must not hammer the backend on every crawler request — needs appropriate caching/revalidation given Next.js 16's route handler caching model for sitemap.ts.

Scope

• Determine and document a privacy-conscious default for profile indexability (with a plan for an explicit opt-out or opt-in flag, coordinating the needed backend field if it doesn't exist).
• Update sitemap.ts to respect that policy, paginate/chunk correctly for scale, and cache appropriately rather than hitting the backend on every crawl.
• Review robots.ts to ensure it doesn't inadvertently block or allow indexing inconsistently with the sitemap policy.
• Add a noindex fallback path for any profile page that shouldn't be crawlable per the policy, enforced via response headers/metadata, not just omission from the sitemap (omission alone doesn't prevent direct-URL indexing).

Acceptance criteria

• The chosen indexability policy is documented and implemented consistently across sitemap.ts, robots.ts, and per-page metadata.
• Sitemap generation is proven to scale (tested against a large mock user list) without exceeding size limits or excessive backend load.

[rudra496]

Happy to take this one. I went through src/app/robots.ts and src/app/sitemap.ts and the current sitemap only enumerates static routes — so the exposure today is the per-handle profile pages being indexable at all rather than a crawled earnings directory, but the gap is real: nothing stops a future handle-listing sitemap, and the profile pages server-render lifetime earnings into crawlable HTML with no per-user control.

Plan:

1. Policy first: default profiles to non-indexable (noindex + omitted from the sitemap) until a user explicitly opts in — for an earnings-tied platform I think opt-in is the only defensible default, and it means the sitemap never has to justify enumerating people who didn't ask to be listed.
2. sitemap.ts: keep it to public, non-personal routes (/, /issues, static pages), fetch the optional public-profile handle list from the backend behind a stable cache (revalidate), chunk into a sitemap index if handles ever exceed the 50k/50MB limits, and degrade to static-only if the backend is unavailable so crawlers never get a 5xx.
3. robots.ts: align with the policy — no contradictory disallows, and explicitly disallow nothing that sitemap includes.
4. /reputation/[handle]: serve robots: noindex metadata by default via the page's generateMetadata, with an opt-in flag (backend field, flagged as a cross-repo addition in the PR if it doesn't exist yet) flipping it to index + sitemap inclusion.
5. Tests for the sitemap contents, the metadata on both flag states, and the large-list chunking path.

I've got an a11y batch PR in review on this repo (#338) and can start immediately.