Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,6 @@ CLI (uv run skim ...) → skim_cli.cli → skim_core.crawlers.REGISTRY lookup
| producthunt | `## Product Hunt Comments` | 제품당 1건 (PH 제품 페이지) |
| threads | `## Threads Replies` | 답글 1개 이상인 게시물 전부, 게시물당 1건 |
| lobsters | `## Lobsters Comments` | 게시물당 1건 (초당 1요청). 같은 응답의 `description_plain`이 본문 폴백 |
| bluesky | `## Bluesky Replies` | 답글 1개 이상인 게시물, 게시물당 1건 (무인증 postThread) |

- **댓글 조회는 `comments`가 0보다 클 때만 한다.** 0건인 글을 조회하면 "유효 댓글 없음"과
"HTTP 실패"가 둘 다 `None`이라 구분되지 않는다. reddit은 그 때문에 조용한 서브레딧에서
Expand Down Expand Up @@ -162,7 +161,7 @@ CLI (uv run skim ...) → skim_cli.cli → skim_core.crawlers.REGISTRY lookup

| 유형 | 위치 | 옵션 기준 | 플랫폼 |
|------|------|-----------|--------|
| Feed | `packages/skim-core/src/skim_core/crawlers/feed/` | `since` | hackernews, lobsters, geeknews, youtube, producthunt, arxiv, huggingface, everyto, blogs, ailabs, bluesky |
| Feed | `packages/skim-core/src/skim_core/crawlers/feed/` | `since` | hackernews, lobsters, geeknews, youtube, producthunt, arxiv, huggingface, everyto, blogs, ailabs |
| API | `packages/skim-core/src/skim_core/crawlers/api/` | `count` | threads, x, linkedin, reddit |

#### 좁은 창에서 0건이 나오는 소스
Expand Down
1 change: 0 additions & 1 deletion README.ko.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@ Skim은 여러 public feed와 세션 기반 social source에서 post를 수집
| Feed | Every.to | RSS feeds |
| Feed | Blogs | `PERSONAL_BLOGS`의 RSS feeds |
| Feed | AI Labs | OpenAI, Anthropic, LangChain, DeepMind, Google Research, Hugging Face, Mistral |
| Feed | Bluesky | 공개 XRPC (로그인 불필요) |
| API | Threads | Instagram Private API |
| API | X | `twitter-api-client` 기반 GraphQL |
| API | LinkedIn | Voyager GraphQL |
Expand Down
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@ Skim collects posts from multiple public feeds and session-based social sources,
| Feed | Every.to | RSS feeds |
| Feed | Blogs | RSS feeds in `PERSONAL_BLOGS` |
| Feed | AI Labs | OpenAI, Anthropic, LangChain, DeepMind, Google Research, Hugging Face, Mistral |
| Feed | Bluesky | Public XRPC (no login) |
| API | Threads | Instagram Private API |
| API | X | GraphQL via `twitter-api-client` |
| API | LinkedIn | Voyager GraphQL |
Expand Down
6 changes: 5 additions & 1 deletion docs/TODO.ko.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ Skim에 넣을 source 후보와 promotion checklist입니다. 구현 계획은 `

- Communities: Hacker News (newest + Show + Ask), Lobsters, GeekNews, Product Hunt
- Social/API: Threads, X, LinkedIn, Reddit
- Social/공개: Bluesky (`BLUESKY_ACCOUNTS`, 로그인 불필요)
- Articles: Every.to, `PERSONAL_BLOGS`의 블로그와 뉴스레터
- Video: `YOUTUBE_CHANNELS`의 YouTube channels
- Papers: Hugging Face Daily Papers, arXiv (cs.AI, cs.CL, cs.LG, cs.CV)
Expand Down Expand Up @@ -56,6 +55,11 @@ Skim에 넣을 source 후보와 promotion checklist입니다. 구현 계획은 `

## 제외된 소스

- `bluesky` - 2026-08-10에 추가했다가 같은 날 제거했다. 크롤러는 동작했지만
(프로덕션 실측 HTTP 200, 엔트리 20개) `BLUESKY_ACCOUNTS`에 게시 빈도가 낮은 계정
하나뿐이라 데일리 `--days 1` 창에서 매번 0건이었다. 실제 실행 조건에서 산출을
확인하지 않고 넣었고, 요청받은 플랫폼도 아니었다. 되살리려면 매일 올라오는 계정
목록이 필요한데 그건 큐레이션 결정이다.
- `every.to/Guides` - `/guides/feed`가 HTTP 500이고 대체 피드도 sitemap도 없다 (2026-08-09 확인). `/guides` 페이지 자체는 살아 있어서, 전용 인덱스 파서를 만들 값어치가 생기면 `scrape` 소스로 복귀할 수 있다. 마지막 수집 2026-06-02.

## Promotion Checklist
Expand Down
18 changes: 5 additions & 13 deletions docs/TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ Candidate sources and promotion checklist for Skim. Keep implementation plans un

- Communities: Hacker News (newest + Show + Ask), Lobsters, GeekNews, Product Hunt
- Social/API: Threads, X, LinkedIn, Reddit
- Social/public: Bluesky (`BLUESKY_ACCOUNTS`, no login required)
- Articles: Every.to, blogs and newsletters in `PERSONAL_BLOGS`
- Video: YouTube channels in `YOUTUBE_CHANNELS`
- Papers: Hugging Face Daily Papers, arXiv (cs.AI, cs.CL, cs.LG, cs.CV)
Expand Down Expand Up @@ -54,20 +53,13 @@ Probed and failed, so nobody re-checks them by hand (2026-08-10):
date field on any entry. `fetch_feed` drops undated entries, so registering it collects
nothing while looking healthy. Needs a custom parser that reads dates off the article page.

## Underfilled Sources

Registered and working, but the daily `--days 1` window collects nothing from them.

- `bluesky` — the crawler is fine (verified against production 2026-08-10: HTTP 200,
20 entries), but `BLUESKY_ACCOUNTS` holds one account, `bsky.app`, whose posting
interval runs days to a month. The newest post was 4 days old at check time, so a
daily run yields 0 almost every day. A wider lookback does not fix this — it would
just re-collect the same posts. The fix is a list of accounts that actually post
daily, which is a curation decision. Until then `doctor` will keep flagging bluesky
as a 0-count regression.

## Retired Sources

- `bluesky` — added 2026-08-10 and removed the same day. The crawler worked (HTTP 200,
20 entries against production), but `BLUESKY_ACCOUNTS` held one low-traffic account,
so the daily `--days 1` window collected nothing. It was added without checking output
under the real run condition, and nobody had asked for the platform. Bringing it back
requires a list of accounts that actually post daily, which is a curation decision.
- `every.to/Guides` — `/guides/feed` returns HTTP 500 with no alternate feed or sitemap (checked 2026-08-09). The `/guides` page itself is alive, so it can return as a `scrape` source if it becomes worth a custom index parser. Last collected 2026-06-02.

## Promotion Checklist
Expand Down
2 changes: 0 additions & 2 deletions packages/skim-core/src/skim_core/crawlers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
from .feed.ailabs import AILabsCrawler
from .feed.arxiv import ArxivCrawler
from .feed.blogs import BlogsCrawler
from .feed.bluesky import BlueskyCrawler
from .feed.everyto import EveryToCrawler
from .feed.geeknews import GeekNewsCrawler
from .feed.hackernews import HackerNewsCrawler
Expand All @@ -30,7 +29,6 @@
"hackernews": HackerNewsCrawler,
"lobsters": LobstersCrawler,
"geeknews": GeekNewsCrawler,
"bluesky": BlueskyCrawler,
"youtube": YouTubeCrawler,
"producthunt": ProductHuntCrawler,
"arxiv": ArxivCrawler,
Expand Down
191 changes: 0 additions & 191 deletions packages/skim-core/src/skim_core/crawlers/feed/bluesky.py

This file was deleted.

8 changes: 0 additions & 8 deletions packages/skim-core/src/skim_core/feed_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -145,14 +145,6 @@ def youtube_videos_url(canonical_id: str) -> str:
LOBSTERS_RSS = "https://lobste.rs/rss"
LOBSTERS_ITEM_JSON = "https://lobste.rs/s/{short_id}.json"

# Bluesky - 무인증 공개 XRPC. 계정 목록이 곧 소스 목록이라 여기서 관리한다
# (searchPosts는 403이라 키워드 수집은 못 한다).
BLUESKY_ACCOUNTS = [
"bsky.app",
]
BLUESKY_AUTHOR_FEED_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed"
BLUESKY_POST_THREAD_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.getPostThread"

# arXiv - Atom API. 카테고리별로 따로 조회해 합친다.
# 크롤러가 합친 뒤 --count로 자르므로 카테고리를 늘려도 수집량은 그대로다.
# 넓어지는 건 커버리지지 볼륨이 아니다. cs.AI 단독이면 cs.CL/cs.LG에만 올라온
Expand Down
Loading
Loading