Repository navigation
Crawl list link rot #390
Description
Automated monthly check of the bundled crawl lists (app/src/main/assets/crawl_urls.json, app/src/main/assets/crawl_urls/*.json).
Checked 3264 entries: 2587 ok, 92 dead, 585 inconclusive.
Dead means HTTP 404/410 or NXDOMAIN, seen twice. Verify in a real browser before removing an entry; a dead deep link can often be replaced by the same site's working section page in the same category.
Dead entries
en (29)
es (30)
fr (23)
ru (10)
| Category | URL | Result |
|---|---|---|
| COOKING | https://lifehacker.ru/topics/eda/recepty/ | HTTP 404 |
| ENVIRONMENT | https://wwf.ru/ | NXDOMAIN |
| FASHION | https://www.vogue.ru/fashion | HTTP 410 |
| FOOD | https://www.timeout.ru/msk/feature/restaurants | HTTP 404 |
| HOME_IMPROVEMENT | https://www.leroymerlin.ru/ | NXDOMAIN |
| MUSIC | https://www.colta.ru/music | HTTP 404 |
| OUTDOOR_RECREATION | https://www.tourister.ru/world/europe/russia | HTTP 404 |
| RELATIONSHIPS_DATING | https://www.marieclaire.ru/psychology/relationship/ | HTTP 404 |
| RELATIONSHIPS_DATING | https://www.thevoicemag.ru/psychology/relations/ | HTTP 404 |
| TRAVEL | https://www.tourister.ru/ | HTTP 404 |
Inconclusive (not acted on)
CI runners use datacenter IPs that many sites bot-block, so these counts are informational only.
| Result | Count |
|---|---|
| HTTP 403 | 451 |
| timeout | 29 |
| HTTP 401 | 16 |
| HTTP 429 | 16 |
| HTTP 405 | 12 |
| HTTP 402 | 9 |
| HTTP 406 | 8 |
| gaierror | 8 |
| HTTP 307 | 6 |
| SSLCertVerificationError | 4 |
| SSLEOFError | 4 |
| HTTP 503 | 4 |
| HTTP 500 | 3 |
| HTTP 502 | 3 |
| HTTP 302 | 3 |
| HTTP 400 | 2 |
| HTTP 103 | 1 |
| HTTP 522 | 1 |
| ConnectionResetError | 1 |
| HTTP 301 | 1 |
| HTTP 498 | 1 |
| UnicodeEncodeError | 1 |
| HTTP 525 | 1 |
Run: https://github.com/digital-grease/fauxx/actions/runs/41650137681
Activity
krustfurt commentedon Nov 13, 2026
Yeah get latest urls. Go to #393