Skip to content

feat: add VietnamWorks job board scraper with offline fallback [fj4WqyCCw3C5ShR1RfB7MoBPTpkRrBFYP1uT35g3MvT] - #123

Open
waterWang wants to merge 3 commits into
mergeos-bounties:masterfrom
waterWang:feat/vietnamworks-scraper
Open

feat: add VietnamWorks job board scraper with offline fallback [fj4WqyCCw3C5ShR1RfB7MoBPTpkRrBFYP1uT35g3MvT]#123
waterWang wants to merge 3 commits into
mergeos-bounties:masterfrom
waterWang:feat/vietnamworks-scraper

Conversation

@waterWang

Copy link
Copy Markdown

Summary

Adds a VietnamWorks (vietnamworks.com) job board adapter — the largest job board in Vietnam, owned by Navigos Group.

Changes

  • New scraper src/nerajob/scrapers/vietnamworks.py — implements BaseScraper.search(query, location, limit) with:
    • Live API mode via https://api.vietnamworks.com/job-search/v1.0/jobs (public endpoint, no API key required)
    • Rate limiting: 1 request per 5 seconds to avoid abusive traffic
    • Offline fallback: 5 deterministic sample postings when NERAJOB_VIETNAMWORKS_OFFLINE=1 is set or live API is unavailable
    • Robust error handling: graceful degradation to offline samples on any HTTP/network error
  • Registration in src/nerajob/scrapers/registry.py
  • Tests tests/test_vietnamworks.py:
    • test_vietnamworks_registered — verifies scraper is in available_scrapers()
    • test_vietnamworks_offline — verifies offline mode returns correct data
    • test_vietnamworks_online_mocked — verifies live API path with mocked HTTP response, including query filtering

Acceptance Criteria

  • New scraper implements BaseScraper.search(query, location, limit)
  • Registered in src/nerajob/scrapers/registry.py
  • Prefer official/public API; document rate limits + ToS notes in PR
  • Tests with mocked HTTP (CI must not depend on live network)
  • nerajob scan --source vietnamworks -q python -n 5 works when live (or degrades gracefully)
  • No secrets committed

Legal/ToS Note

VietnamWorks is a public job board. The adapter uses conservative rate limiting (1 req/5s) and a documented User-Agent. No API key or authentication is required for the public search endpoint. Users should review VietnamWorks' terms of service before heavy usage.

Fixes #17

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[50 MRG] Scraper: Vietnam tech jobs board adapter (TopCV or VietnamWorks public pages — ToS-safe)

1 participant