chore: bump crawl4ai for CVE fixes

Clears the strict worker pip-audit: crawl4ai 0.8.6 (GHSA-qxjp-w3pj-48m7,
GHSA-365w-hqf6-vxfg, GHSA-f989-c77f-r2cq, GHSA-4qqr-vv2q-cmr5,
GHSA-7cx2-g3h9-382p, GHSA-6qhc-x826-342c), all fixed by 0.8.9. Worker-only;
lxml stays capped, so the PYSEC-2026-87 ignore remains valid.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Noppanat Wadlom <noppanat.wad@gmail.com>

Author: kaiitunnz

pyproject.toml

@@ -83,7 +83,7 @@ runtime-agent = [
     "arxiv>=2.2.0",
     "beautifulsoup4>=4.14.3",
     "chunkr-ai>=0.3.7",
-    "crawl4ai>=0.8.0",
+    "crawl4ai>=0.8.9",
     "ddgs>=9.10.0",
     "google-genai>=1.47.0",
     "hydra-core>=1.3.2",

src/worker/requirements/requirements.txt

@@ -12,7 +12,7 @@ boto3==1.41.5
 chunkr-ai==0.3.7
 colorama==0.4.6
 colorlog==6.9.0
-crawl4ai==0.8.6
+crawl4ai==0.8.9
 datasets==4.3.0
 ddgs==9.10.0
 diffusers==0.38.0