[robbrad]

Summary

Consolidates the fixes from a fresh full-suite triage run into a single PR (folds in #2159, #2160, #2161, #2162 - now closed as superseded).

• AberdeenshireCouncil (Since update no sensors work #2158, priority): self-inflicted regression from the Pembrokeshire council all data becomes unavailable every 12hrs #2073 retry fix (0.168.0). parse_data() ignores the page it's given and makes its own request to a working endpoint, but the generic framework fetch to an unused root URL ran first regardless - that URL returns a persistent 500 on the council's own end, and the retry logic correctly (if unfortunately) turned that into a hard failure, breaking every user's sensors. Fixed by overriding get_data() to skip the pointless fetch - works immediately on upgrade, no config changes needed. Verified live: root URL 500s, real data endpoint 200s with valid data, at the same moment.
• EastDevonDC: 403 was purely a missing User-Agent header - default python-requests/x.x UA gets blocked, this project's own polite scraper UA string (get_scraper_user_agent()) works fine. Verified live end-to-end.
• HorshamDistrictCouncil: switched to build_retry_session() for consistency with the rest of the codebase. Does not fix the council's current failure (verified live it still fails after 5 retries - a sustained block, not a transient blip) - this is defensive hardening only.
• ISSUE_RESOLUTION_PROGRESS.md: documents a fresh full-suite triage (352 councils, 26 failed / 326 passed in 5m21s after reconfiguring the local Selenium grid for real 8-way concurrency instead of the 1-session cap it had), the Aberdeenshire regression writeup, and updated findings for Brown Bin / Garden Waste #2118 (Flintshire) and Slough– ccc-recommended-settings timeout (Selenium) #2079 (Slough).

Broader finding (not fixed here, flagging for awareness)

Found 59 councils with the same shape as Aberdeenshire's bug (parse_data() ignores the fetched page and makes its own request, but skip_get_url isn't set). Cross-referenced against real failures - only Aberdeenshire is currently affected, the rest have url fields pointing somewhere healthy. Worth a slower systematic audit sometime, not urgent.

Test plan

• poetry run pytest uk_bin_collection/tests custom_components/uk_bin_collection/tests --ignore=uk_bin_collection/tests/step_defs/ - 221 passed
• black --check clean
• Live verification for each fix (see commit messages)

Closes

Fixes #2158

Summary by CodeRabbit

• Bug Fixes

  • Improved bin collection retrieval for Aberdeenshire Council.
  • Improved address lookup and calendar requests for East Devon District Council.
  • Added more reliable request retry handling for Horsham District Council.
  • Resolved regressions affecting collection data retrieval and improved handling of known council-specific issues.

• Documentation

  • Updated issue-resolution progress and triage records with current fixes, remaining issues, and regression findings.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 3f537d33-77c3-4d81-af52-1c48fd97942d

📥 Commits

Reviewing files that changed from the base of the PR and between 46d94c1 and 38cdb74.

📒 Files selected for processing (4)

• ISSUE_RESOLUTION_PROGRESS.md
• uk_bin_collection/uk_bin_collection/councils/AberdeenshireCouncil.py
• uk_bin_collection/uk_bin_collection/councils/EastDevonDC.py
• uk_bin_collection/uk_bin_collection/councils/HorshamDistrictCouncil.py

---

📝 Walkthrough

Walkthrough

Three council scrapers receive request-handling updates: Aberdeenshire bypasses the root fetch, East Devon sends scraper headers, and Horsham uses a retry-enabled session. Issue-resolution documentation records related triage and fixes.

Changes

Council scraper reliability fixes

Layer / File(s)	Summary
Aberdeenshire root request handling uk_bin_collection/.../AberdeenshireCouncil.py, ISSUE_RESOLUTION_PROGRESS.md	Aberdeenshire returns an empty root-fetch result so parsing performs the specific endpoint request; issue-triage and fix references are updated.
East Devon request headers uk_bin_collection/.../EastDevonDC.py	Addressfinder and calendar requests now use a shared scraper User-Agent header, with error formatting updates.
Horsham retry-enabled session uk_bin_collection/.../HorshamDistrictCouncil.py	Parsing now uses the project retry session for GET and POST requests; bin record values remain unchanged.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• robbrad/UKBinCollectionData#2110: Introduced the session and header flow that the Horsham change now adapts to retry-enabled requests.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/post-merge-triage-fixes

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.89%. Comparing base (4d6f787) to head (38cdb74).
⚠️ Report is 11 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2163   +/-   ##
=======================================
  Coverage   86.89%   86.89%           
=======================================
  Files           9        9           
  Lines        1160     1160           
=======================================
  Hits         1008     1008           
  Misses        152      152           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.