diff --git a/AGENTS.md b/AGENTS.md index a114de9..3bcd97e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -127,7 +127,6 @@ CLI (uv run skim ...) → skim_cli.cli → skim_core.crawlers.REGISTRY lookup | producthunt | `## Product Hunt Comments` | 제품당 1건 (PH 제품 페이지) | | threads | `## Threads Replies` | 답글 1개 이상인 게시물 전부, 게시물당 1건 | | lobsters | `## Lobsters Comments` | 게시물당 1건 (초당 1요청). 같은 응답의 `description_plain`이 본문 폴백 | -| bluesky | `## Bluesky Replies` | 답글 1개 이상인 게시물, 게시물당 1건 (무인증 postThread) | - **댓글 조회는 `comments`가 0보다 클 때만 한다.** 0건인 글을 조회하면 "유효 댓글 없음"과 "HTTP 실패"가 둘 다 `None`이라 구분되지 않는다. reddit은 그 때문에 조용한 서브레딧에서 @@ -162,7 +161,7 @@ CLI (uv run skim ...) → skim_cli.cli → skim_core.crawlers.REGISTRY lookup | 유형 | 위치 | 옵션 기준 | 플랫폼 | |------|------|-----------|--------| -| Feed | `packages/skim-core/src/skim_core/crawlers/feed/` | `since` | hackernews, lobsters, geeknews, youtube, producthunt, arxiv, huggingface, everyto, blogs, ailabs, bluesky | +| Feed | `packages/skim-core/src/skim_core/crawlers/feed/` | `since` | hackernews, lobsters, geeknews, youtube, producthunt, arxiv, huggingface, everyto, blogs, ailabs | | API | `packages/skim-core/src/skim_core/crawlers/api/` | `count` | threads, x, linkedin, reddit | #### 좁은 창에서 0건이 나오는 소스 diff --git a/README.ko.md b/README.ko.md index 93e226e..5c0eae9 100644 --- a/README.ko.md +++ b/README.ko.md @@ -44,7 +44,6 @@ Skim은 여러 public feed와 세션 기반 social source에서 post를 수집 | Feed | Every.to | RSS feeds | | Feed | Blogs | `PERSONAL_BLOGS`의 RSS feeds | | Feed | AI Labs | OpenAI, Anthropic, LangChain, DeepMind, Google Research, Hugging Face, Mistral | -| Feed | Bluesky | 공개 XRPC (로그인 불필요) | | API | Threads | Instagram Private API | | API | X | `twitter-api-client` 기반 GraphQL | | API | LinkedIn | Voyager GraphQL | diff --git a/README.md b/README.md index 7419ca3..9620cc0 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,6 @@ Skim collects posts from multiple public feeds and session-based social sources, | Feed | Every.to | RSS feeds | | Feed | Blogs | RSS feeds in `PERSONAL_BLOGS` | | Feed | AI Labs | OpenAI, Anthropic, LangChain, DeepMind, Google Research, Hugging Face, Mistral | -| Feed | Bluesky | Public XRPC (no login) | | API | Threads | Instagram Private API | | API | X | GraphQL via `twitter-api-client` | | API | LinkedIn | Voyager GraphQL | diff --git a/docs/TODO.ko.md b/docs/TODO.ko.md index 28a3155..e402c69 100644 --- a/docs/TODO.ko.md +++ b/docs/TODO.ko.md @@ -8,7 +8,6 @@ Skim에 넣을 source 후보와 promotion checklist입니다. 구현 계획은 ` - Communities: Hacker News (newest + Show + Ask), Lobsters, GeekNews, Product Hunt - Social/API: Threads, X, LinkedIn, Reddit -- Social/공개: Bluesky (`BLUESKY_ACCOUNTS`, 로그인 불필요) - Articles: Every.to, `PERSONAL_BLOGS`의 블로그와 뉴스레터 - Video: `YOUTUBE_CHANNELS`의 YouTube channels - Papers: Hugging Face Daily Papers, arXiv (cs.AI, cs.CL, cs.LG, cs.CV) @@ -56,6 +55,11 @@ Skim에 넣을 source 후보와 promotion checklist입니다. 구현 계획은 ` ## 제외된 소스 +- `bluesky` - 2026-08-10에 추가했다가 같은 날 제거했다. 크롤러는 동작했지만 + (프로덕션 실측 HTTP 200, 엔트리 20개) `BLUESKY_ACCOUNTS`에 게시 빈도가 낮은 계정 + 하나뿐이라 데일리 `--days 1` 창에서 매번 0건이었다. 실제 실행 조건에서 산출을 + 확인하지 않고 넣었고, 요청받은 플랫폼도 아니었다. 되살리려면 매일 올라오는 계정 + 목록이 필요한데 그건 큐레이션 결정이다. - `every.to/Guides` - `/guides/feed`가 HTTP 500이고 대체 피드도 sitemap도 없다 (2026-08-09 확인). `/guides` 페이지 자체는 살아 있어서, 전용 인덱스 파서를 만들 값어치가 생기면 `scrape` 소스로 복귀할 수 있다. 마지막 수집 2026-06-02. ## Promotion Checklist diff --git a/docs/TODO.md b/docs/TODO.md index 9a3e38e..ea02198 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -8,7 +8,6 @@ Candidate sources and promotion checklist for Skim. Keep implementation plans un - Communities: Hacker News (newest + Show + Ask), Lobsters, GeekNews, Product Hunt - Social/API: Threads, X, LinkedIn, Reddit -- Social/public: Bluesky (`BLUESKY_ACCOUNTS`, no login required) - Articles: Every.to, blogs and newsletters in `PERSONAL_BLOGS` - Video: YouTube channels in `YOUTUBE_CHANNELS` - Papers: Hugging Face Daily Papers, arXiv (cs.AI, cs.CL, cs.LG, cs.CV) @@ -54,20 +53,13 @@ Probed and failed, so nobody re-checks them by hand (2026-08-10): date field on any entry. `fetch_feed` drops undated entries, so registering it collects nothing while looking healthy. Needs a custom parser that reads dates off the article page. -## Underfilled Sources - -Registered and working, but the daily `--days 1` window collects nothing from them. - -- `bluesky` — the crawler is fine (verified against production 2026-08-10: HTTP 200, - 20 entries), but `BLUESKY_ACCOUNTS` holds one account, `bsky.app`, whose posting - interval runs days to a month. The newest post was 4 days old at check time, so a - daily run yields 0 almost every day. A wider lookback does not fix this — it would - just re-collect the same posts. The fix is a list of accounts that actually post - daily, which is a curation decision. Until then `doctor` will keep flagging bluesky - as a 0-count regression. - ## Retired Sources +- `bluesky` — added 2026-08-10 and removed the same day. The crawler worked (HTTP 200, + 20 entries against production), but `BLUESKY_ACCOUNTS` held one low-traffic account, + so the daily `--days 1` window collected nothing. It was added without checking output + under the real run condition, and nobody had asked for the platform. Bringing it back + requires a list of accounts that actually post daily, which is a curation decision. - `every.to/Guides` — `/guides/feed` returns HTTP 500 with no alternate feed or sitemap (checked 2026-08-09). The `/guides` page itself is alive, so it can return as a `scrape` source if it becomes worth a custom index parser. Last collected 2026-06-02. ## Promotion Checklist diff --git a/packages/skim-core/src/skim_core/crawlers/__init__.py b/packages/skim-core/src/skim_core/crawlers/__init__.py index 7e9beaf..baa6d7c 100644 --- a/packages/skim-core/src/skim_core/crawlers/__init__.py +++ b/packages/skim-core/src/skim_core/crawlers/__init__.py @@ -13,7 +13,6 @@ from .feed.ailabs import AILabsCrawler from .feed.arxiv import ArxivCrawler from .feed.blogs import BlogsCrawler -from .feed.bluesky import BlueskyCrawler from .feed.everyto import EveryToCrawler from .feed.geeknews import GeekNewsCrawler from .feed.hackernews import HackerNewsCrawler @@ -30,7 +29,6 @@ "hackernews": HackerNewsCrawler, "lobsters": LobstersCrawler, "geeknews": GeekNewsCrawler, - "bluesky": BlueskyCrawler, "youtube": YouTubeCrawler, "producthunt": ProductHuntCrawler, "arxiv": ArxivCrawler, diff --git a/packages/skim-core/src/skim_core/crawlers/feed/bluesky.py b/packages/skim-core/src/skim_core/crawlers/feed/bluesky.py deleted file mode 100644 index cbaf55d..0000000 --- a/packages/skim-core/src/skim_core/crawlers/feed/bluesky.py +++ /dev/null @@ -1,191 +0,0 @@ -""" -@file bluesky.py -@description Bluesky 크롤러 (무인증 공개 XRPC) - -threads/x/linkedin과 달리 로그인이 필요 없다. `public.api.bsky.app`이 무인증으로 -authorFeed와 postThread를 내주므로 계정이 노출되지 않는다. 대신 계정 팔로우가 -소스 목록을 소유하지 않으므로 볼 계정을 `BLUESKY_ACCOUNTS`에서 관리한다 -(searchPosts는 403이라 키워드 수집은 못 한다). -""" - -import time -from datetime import datetime, timedelta, timezone -from typing import Any, List, Optional - -import typer - -from ...comments import Comment, append_comment_section, render_comment_section -from ...feed_config import ( - BLUESKY_ACCOUNTS, - BLUESKY_AUTHOR_FEED_URL, - BLUESKY_POST_THREAD_URL, -) -from ...feed_utils import make_retrying_session -from ...models import Post - -MAX_COMMENTS = 15 -MIN_REPLIES_FOR_FETCH = 1 -REQUEST_INTERVAL_SECONDS = 0.5 - -_SESSION = make_retrying_session({"Accept": "application/json"}) - - -def _parse_created(value: str) -> Optional[datetime]: - try: - return datetime.fromisoformat((value or "").replace("Z", "+00:00")) - except (ValueError, AttributeError): - return None - - -def post_web_url(uri: str, handle: str) -> str: - """at:// URI를 사람이 여는 주소로 바꾼다. - - at://did:plc:xxx/app.bsky.feed.post/3mse... -> https://bsky.app/profile/{handle}/post/3mse... - """ - rkey = (uri or "").rsplit("/", 1)[-1] - if not rkey or not handle: - return "" - return f"https://bsky.app/profile/{handle}/post/{rkey}" - - -class BlueskyCrawler: - """Bluesky 공개 API 크롤러.""" - - platform = "bluesky" - - async def crawl(self, **options: Any) -> List[Post]: - since: datetime = options.get( - "since", datetime.now(timezone.utc) - timedelta(days=1) - ) - count = options.get("count") - no_content: bool = options.get("no_content", False) - - accounts = options.get("accounts") or BLUESKY_ACCOUNTS - posts: List[Post] = [] - for index, actor in enumerate(accounts): - if index: - time.sleep(REQUEST_INTERVAL_SECONDS) - try: - posts.extend(self.fetch_author_feed(actor, since)) - except Exception as exc: # noqa: BLE001 - 한 계정 실패가 나머지를 막지 않는다 - typer.echo(f" [!] Bluesky {actor} 수집 실패: {exc}") - - posts.sort(key=lambda p: p.timestamp, reverse=True) - if count is not None: - posts = posts[:count] - - if not no_content: - self.attach_replies(posts) - return posts - - def fetch_author_feed(self, actor: str, since: datetime) -> List[Post]: - """한 계정의 최근 게시물. 리포스트는 제외한다.""" - resp = _SESSION.get( - BLUESKY_AUTHOR_FEED_URL, - params={"actor": actor, "limit": 50, "filter": "posts_no_replies"}, - timeout=20, - ) - resp.raise_for_status() - feed = resp.json().get("feed") or [] - - results: List[Post] = [] - for entry in feed: - # reason이 있으면 남의 글을 올린 리포스트다. 원저자 타임라인에서 받는다. - if entry.get("reason"): - continue - post = self._entry_to_post(entry, actor) - if post is None: - continue - created = _parse_created(post.timestamp) - if created and created < since: - continue - results.append(post) - return results - - def attach_replies(self, posts: List[Post]) -> None: - """답글을 정본 본문 뒤에 잇는다. 게시물당 요청 1건.""" - failures = 0 - for index, post in enumerate(posts): - if (post.comments or 0) < MIN_REPLIES_FOR_FETCH: - continue - if index: - time.sleep(REQUEST_INTERVAL_SECONDS) - uri = getattr(post, "at_uri", "") - if not uri: - continue - try: - section = self.fetch_reply_section(uri) - except Exception as exc: # noqa: BLE001 - 답글 실패가 게시물 저장을 막지 않는다 - failures += 1 - typer.echo(f" [!] Bluesky 답글 수집 실패: {exc}") - continue - if section: - post.content_markdown = append_comment_section( - post.content_markdown or post.content, section - ) - - if failures: - typer.echo(f" [!] Bluesky 답글 수집 실패 {failures}건 (본문만 저장)") - - def fetch_reply_section(self, uri: str) -> Optional[str]: - """postThread에서 직계 답글만 뽑는다. 작성자 self-thread는 건너뛴다.""" - resp = _SESSION.get( - BLUESKY_POST_THREAD_URL, - params={"uri": uri, "depth": 1}, - timeout=20, - ) - if resp.status_code != 200: - return None - thread = resp.json().get("thread") or {} - root_handle = ((thread.get("post") or {}).get("author") or {}).get( - "handle" - ) or "" - - collected: List[Comment] = [] - for reply in thread.get("replies") or []: - reply_post = reply.get("post") or {} - author = (reply_post.get("author") or {}).get("handle") or "unknown" - # 작성자 self-reply 연작은 이미 본문에 담기는 흐름이라 뺀다. - if author and author == root_handle: - continue - text = ((reply_post.get("record") or {}).get("text") or "").strip() - if not text: - continue - created = (reply_post.get("record") or {}).get("createdAt") or "" - collected.append( - Comment( - author=f"@{author}", - text=text, - score=reply_post.get("likeCount"), - created=created[:16].replace("T", " ") or None, - ) - ) - - return render_comment_section( - "Bluesky Replies", collected, max_comments=MAX_COMMENTS, score_unit="like" - ) - - def _entry_to_post(self, entry: dict, source: str) -> Optional[Post]: - raw = entry.get("post") or {} - record = raw.get("record") or {} - text = (record.get("text") or "").strip() - if not text: - return None - author = raw.get("author") or {} - handle = author.get("handle") or "" - created = _parse_created(record.get("createdAt", "")) - - return Post( - platform=self.platform, - author=author.get("displayName") or handle or "unknown", - content=text, - content_markdown=text, - timestamp=created.astimezone(timezone.utc).isoformat() if created else "", - url=post_web_url(raw.get("uri", ""), handle), - likes=raw.get("likeCount"), - comments=raw.get("replyCount"), - reposts=raw.get("repostCount"), - source=source, - external_id=raw.get("uri", "") or None, - at_uri=raw.get("uri", ""), - ) diff --git a/packages/skim-core/src/skim_core/feed_config.py b/packages/skim-core/src/skim_core/feed_config.py index 926884c..970f837 100644 --- a/packages/skim-core/src/skim_core/feed_config.py +++ b/packages/skim-core/src/skim_core/feed_config.py @@ -145,14 +145,6 @@ def youtube_videos_url(canonical_id: str) -> str: LOBSTERS_RSS = "https://lobste.rs/rss" LOBSTERS_ITEM_JSON = "https://lobste.rs/s/{short_id}.json" -# Bluesky - 무인증 공개 XRPC. 계정 목록이 곧 소스 목록이라 여기서 관리한다 -# (searchPosts는 403이라 키워드 수집은 못 한다). -BLUESKY_ACCOUNTS = [ - "bsky.app", -] -BLUESKY_AUTHOR_FEED_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed" -BLUESKY_POST_THREAD_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.getPostThread" - # arXiv - Atom API. 카테고리별로 따로 조회해 합친다. # 크롤러가 합친 뒤 --count로 자르므로 카테고리를 늘려도 수집량은 그대로다. # 넓어지는 건 커버리지지 볼륨이 아니다. cs.AI 단독이면 cs.CL/cs.LG에만 올라온 diff --git a/tests/test_new_sources.py b/tests/test_new_sources.py index 3620573..944241f 100644 --- a/tests/test_new_sources.py +++ b/tests/test_new_sources.py @@ -1,4 +1,4 @@ -"""소스 확장 회귀 테스트: arxiv 다중 카테고리, HN 다중 피드, lobsters, bluesky.""" +"""소스 확장 회귀 테스트: arxiv 다중 카테고리, HN 다중 피드, lobsters.""" import asyncio import unittest @@ -6,9 +6,7 @@ from unittest.mock import MagicMock, patch from skim_core.crawlers import REGISTRY -from skim_core.crawlers.feed import bluesky from skim_core.crawlers.feed.arxiv import ArxivCrawler -from skim_core.crawlers.feed.bluesky import BlueskyCrawler, post_web_url from skim_core.crawlers.feed.hackernews import HackerNewsCrawler from skim_core.crawlers.feed.lobsters import ( LobstersCrawler, @@ -41,8 +39,7 @@ def test_undated_feed_is_not_registered(self): self.assertNotIn("yozm.wishket.com", " ".join(PERSONAL_BLOGS.values())) def test_new_platforms_are_in_the_registry(self): - for name in ("lobsters", "bluesky"): - self.assertIn(name, REGISTRY) + self.assertIn("lobsters", REGISTRY) def _arxiv_entry(title, link, published): @@ -236,64 +233,5 @@ def test_item_fetch_failure_keeps_the_post(self): self.assertEqual(post.content_markdown, "original") -def _bsky_post( - text, handle="a.bsky.social", uri="at://did:plc:x/app.bsky.feed.post/abc" -): - return { - "uri": uri, - "author": {"handle": handle, "displayName": "A"}, - "record": {"text": text, "createdAt": "2026-08-09T10:00:00.000Z"}, - "likeCount": 3, - "replyCount": 2, - "repostCount": 1, - } - - -class BlueskyTests(unittest.TestCase): - def test_at_uri_becomes_a_web_url(self): - self.assertEqual( - post_web_url("at://did:plc:x/app.bsky.feed.post/3mse", "bsky.app"), - "https://bsky.app/profile/bsky.app/post/3mse", - ) - self.assertEqual(post_web_url("", "bsky.app"), "") - - def test_reposts_are_skipped(self): - crawler = BlueskyCrawler() - response = MagicMock(status_code=200) - response.json.return_value = { - "feed": [ - {"post": _bsky_post("mine")}, - {"post": _bsky_post("someone else"), "reason": {"$type": "repost"}}, - ] - } - response.raise_for_status.return_value = None - - with patch.object(bluesky._SESSION, "get", return_value=response): - posts = crawler.fetch_author_feed( - "a.bsky.social", datetime(2026, 1, 1, tzinfo=timezone.utc) - ) - - self.assertEqual([p.content for p in posts], ["mine"]) - - def test_author_self_replies_are_not_treated_as_discussion(self): - crawler = BlueskyCrawler() - response = MagicMock(status_code=200) - response.json.return_value = { - "thread": { - "post": _bsky_post("root", handle="me.bsky.social"), - "replies": [ - {"post": _bsky_post("my own follow-up", handle="me.bsky.social")}, - {"post": _bsky_post("someone replied", handle="other.bsky.social")}, - ], - } - } - - with patch.object(bluesky._SESSION, "get", return_value=response): - section = crawler.fetch_reply_section("at://x") - - self.assertIn("someone replied", section) - self.assertNotIn("my own follow-up", section) - - if __name__ == "__main__": unittest.main()