feat(crawlers): 댓글과 답글을 정본 본문에 담는다 - #13
Merged
Merged
Conversation
added 5 commits
August 10, 2026 08:13
news.hada.io started rejecting Chrome/124 on 2026-08-09 — that day 10 of 21 geeknews rows landed without metrics (0 gaps the day before). Verified 124 is blocked and 128+ passes, so the shared feed UA moves to 139.
Only hackernews stored discussion text; geeknews/reddit/x/linkedin kept a comment count and dropped the content, which breaks the AGENTS.md contract that content_markdown is consumption-ready. Adds skim_core.comments (platform-neutral Comment + section renderer) and wires six crawlers to it. Two cost nothing extra: geeknews reuses the topic HTML already fetched for metrics, and x reuses the TweetDetail response whose non-author replies were being filtered out. Standalone tweets are re-fetched only when they have 3+ replies, capped at 20 per run. threads is left out: its timeline response carries no third-party replies and a separate post-detail doc_id would have to be captured from the web app.
The timeline GraphQL response carries no third-party replies, so threads was the one platform left storing only a reply count. The post page turns out to ship the replies in its SSR payload and needs no login, so no new persisted query coordinate has to be maintained. Two traps this had to work around: requesting threads.net redirects to a shell with the payload stripped (threads.com must be hit directly), and the author's own self-reply chain sits in the same edges array as the replies, which would duplicate body text already stored. Fetches only posts with 3+ replies, capped at 20 per run.
The threshold of 3 was copied from x, where a re-fetch costs 3-9s and has to be rationed. A threads reply fetch is a single GET (1-4s measured), and the per-run cap already bounds the cost, so the threshold only decides where that budget goes. Gating on `comments` at 3 dropped 12.6% of posts (519 of 4132 have 1-2 replies), and that field is unreliable in both directions: it counts deleted and private replies, and a post reporting 1 reply yielded 2. Only skip posts reporting zero.
Two caps were discarding replies that had already been fetched. The per-run cap of 20 posts skipped whole posts, and MAX_REPLIES=15 truncated documents that carried more - one post shipped 24 replies and 9 were dropped. A crawl now stores 21 replies on a single post where 15 was the ceiling before. render_comment_section takes max_comments=None for "keep everything". Pagination is deliberately not added. Going past what the document ships requires BarcelonaPostPageRefetchableDirectQuery in batches of 4, and that request needs session cookies and an x-fb-lsd token, so it is attributable to the logged-in account. The current path needs no login at all, and keeping that property was chosen over collecting more replies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
댓글/답글을
content_markdown에 담는다. 이전에는 hackernews만 토론 본문을 저장했고,나머지는 댓글 수만 남기고 내용을 버려서 "DB는 소비 준비가 끝난 상태"라는 계약을 어기고 있었다.
7개 플랫폼을 붙였고, 그중 둘은 추가 요청 없이 얻는다.
작업 중 발견한 GeekNews 403 회귀도 함께 고쳤다.
플랫폼별
## GeekNews Comments## X Replies## Reddit Comments## LinkedIn Comments## YouTube Comments## Product Hunt Comments## Threads Replies포맷은
skim_core.comments가 단독으로 책임진다. 각 크롤러는 응답을Comment로 정규화만 한다.함께 고친 것: GeekNews 403
news.hada.io가 2026-08-09부터 Chrome/124를 차단하기 시작했다.FEED_HEADERS가 정확히 124여서그날 21건 중 10건이 지표 없이 저장됐다(8/8까지 결측 0). 128 이상이 통과하는 걸 확인해 139로 올렸다.
threads: doc_id 없이 답글을 얻는 방법
타임라인 GraphQL 응답에는 타인 답글이 오지 않는다. 대신 게시물 문서의 SSR 페이로드가 답글까지 담고
있고 로그인도 필요 없어서, persisted query 좌표를 새로 떠안지 않았다.
두 가지 함정:
threads.net으로 요청하면 리다이렉트 뒤 페이로드가 빠진 255KB 셸이 온다.threads.com으로 직접 받으면 843KB에 답글이 들어온다.edges배열에 담긴다. 그대로 두면 24편짜리 연작이 통째로 "답글"로 중복 저장된다.페이지네이션은 의도적으로 넣지 않았다. 문서가 주는 것보다 더 가져오려면
BarcelonaPostPageRefetchableDirectQuery를 4건씩 반복 호출해야 하는데, 그 요청은 세션 쿠키와x-fb-lsd토큰을 요구해 로그인 계정으로 식별된다. 답글 수집량보다 계정 안전을 택했다.조사 중 정정한 판단 3건
data/sessions/가 없어 비로그인 상태였다.conversation_id필터를 추가했다.direct_reply_count가 삭제·비공개 답글까지 세는 부정확한 값이라 1로 낮췄다.알려진 한계
comments가 0보다 커도 답글 섹션이 안 붙을 수 있다. 삭제·비공개 답글까지 세는 값이라 실제 노출 답글이 없는 게시물이 있다(브라우저로 열어도 안 보인다). 추출 실패로 오판하지 않도록 AGENTS.md에 적어뒀다.크롤 시간 영향
reddit, linkedin, producthunt, threads는 게시글당 요청이 1건씩 는다. 급하면
--no-content로 댓글수집만 끌 수 있다(geeknews와 x 스레드 경로는 원래 추가 요청이 없어 영향 없음).
검증