Repository navigation
Conversation
The council withdrew the per-address lookup from its self-service portal (#1881): Check_your_Bin_Day redirects to the homepage, so the Selenium uprn + postcode scraper could only ever time out. The district's collection calendars are published as plain HTML on the main council site instead, as two area calendars ("Calendar A - The North" and "Calendar B - The South"), each a list of date ranges and the bins collected during that range. Parse those with a plain requests GET. BREAKING CHANGE: a week-long range says which bins go out that week but not on which day, so the scraper now needs the user's area and their normal collection weekday rather than a uprn and postcode. Both are passed in the existing house_number field, e.g. "North, Monday" or "Clay Cross, Friday", following the OrkneyIslandsCouncil precedent for a non-address value in that field; nothing in collect_data.py or the Home Assistant config flow needs new plumbing. Existing users must reconfigure. input.json drops uprn, postcode and web_driver in the same commit. The parser handles the page's real irregularities: ordinal suffixes, entries whose start date carries neither month nor year, the month heading formats ("August 2026", "December '26"), ranges from one to twelve days, ranges crossing the year boundary, "No collections" weeks, and bank-holiday weeks, where the council shifts every household a day later. Settlement lists are scraped rather than hardcoded, and Sutton and Temple Normanton, which appear on both calendars, raise an error asking for an explicit area. Targets #1881 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 43 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2280 +/- ##
==========================================
+ Coverage 80.97% 82.07% +1.09%
==========================================
Files 16 34 +18
Lines 1619 2616 +997
==========================================
+ Hits 1311 2147 +836
- Misses 308 469 +161 ☔ View full report in Codecov by Harness. |
|
Cherry-picked into #2277 unchanged (commit Nothing wrong with the work — Re-verified on the consolidated branch: the live test and the 20 offline tests all pass, parity is clean, and input.json is consistent with the new area+weekday inputs. |
1 participant
North East Derbyshire no longer takes a UPRN or a postcode, and no longer uses
Selenium. It now takes the user's collection area and normal collection weekday in
the house number field, e.g.
North, MondayorClay Cross, Friday.Existing Home Assistant users will have to reconfigure the integration after the release
that carries this (the config flow is built from
input.json, whichINPUT_JSON_URLpinsto a release tag, so nothing changes for them until then). CLI users must swap
-u <uprn> -p <postcode>for-n "<area>, <weekday>".Why
https://myselfservice.ne-derbyshire.gov.uk/service/Check_your_Bin_Dayno longer offers aper-address lookup — it redirects to the council homepage — so the Selenium scraper could
only ever time out waiting for the postcode form. Verified live from a UK residential
connection on 2026-10-15. This is the long-standing failure tracked in #1881.
The council publishes the same information as plain HTML on its main site instead:
https://www.ne-derbyshire.gov.uk/bins-and-recycling/bin-collection-datesreturns 200 to aplain
requestsGET withget_scraper_user_agent()(it is Cloudflare-fronted but does notchallenge a plain GET). It carries two area calendars, "Calendar A - The North" and
"Calendar B - The South", each a list of date ranges and the bins collected during that
range:
Why the inputs had to change, and why they go in
house_numberA week-long range tells you which bins go out that week but not which day your bin goes
out, so the scraper needs two things from the user: the area and their normal collection
weekday.
collect_data.pyonly plumbs through a fixed set of arguments, so the options were:house_numberfield (chosen).OrkneyIslandsCouncilalready uses
house_numberfor a non-address value, so there is precedent, no change tocollect_data.py, and no change to the HA config flow — the field the user already seessimply carries a different kind of value, described in
wiki_note. The scraper validatesit and raises a message listing the accepted areas and weekdays.
house_numberfor the weekday,postcodefor the real postcode, deriving the area.Checked and rejected: the page contains no postcode-to-settlement mapping. The only
postcode anywhere on it is the council's own office address (S42 6NG), and there is no
other published mapping from postcode to Calendar A/B.
collect_data.pyplus the HA config flow. More correct in theabstract, but it touches the integration, and
INPUT_JSON_URLis pinned to a release tag,so it would reach users no sooner than option 1 while adding a field to every council's
flow. Not needed, since option 1 works.
What the parser handles
The page's markup is irregular, and all of this is covered by the unit tests:
<li>elements, South dates in<p>elements.<h4>August 2026</h4>and as<p>December '26</p>/<p>March 27</p>.1st,2nd,3rd,21st,30th).Monday 3rd to Friday 7th August),taking both from the end date and the month heading.
Wednesday 30th December 2026 to Friday 1st January 2027).Monday 4th January to Friday 15th January 2027is twoweeks and yields two collections per weekday).
No collectionsweeks, which are skipped.household a day later ("when there is a Bank Holiday Monday, the collection day is normally
one day later than usual"): a
Tuesday 30th March to Saturday 3rd April 2027range collectsa Monday household on the Tuesday and a Friday household on the Saturday.
B lack Bintypo, since bin names are matched with non-letters removed.Settlement lists are scraped from the "Which calendar do I use?" section rather than
hardcoded, and an explicit
North/Southalways wins. Sutton and Temple Normanton appear onboth lists (their smaller 'rural' route properties are on Calendar A and the rest on
Calendar B), so passing either name alone raises an error asking for an explicit area.
Known limitation, stated in
wiki_note: the Christmas catch-up collections(
Tuesday 29th December 2026: ... if your bin should have been emptied on the 25th of December) are not attributed to the households they apply to, so that one collection isunder-reported rather than mis-reported.
Test plan
uk_bin_collection/tests/test_north_east_derbyshire.py— new, 20 offline tests driving theparser from a fixture reproducing the live markup shapes captured on 2026-10-15. Covers a
North address and a South one, settlement lookup, a two-day range, a twelve-day range, the
year boundary, a bank-holiday week, ordinal parsing, the typo, past-date filtering,
No collectionsweeks, a settlement on both calendars, an unknown area, an unreadableweekday, an exhausted calendar, and the HTTP fetch with the right User-Agent. All pass.
poetry run pytest uk_bin_collection/tests custom_components/uk_bin_collection/tests --ignore=uk_bin_collection/tests/step_defs/ -q→ 359 passed, no failures.
poetry run pytest uk_bin_collection/tests/step_defs/ -k NorthEastDerbyshire --local_browser True -q→ 1 passed in 2.4s (one plain GET, no Selenium).
collect_data.py NorthEastDerbyshireDistrictCouncil <url> -n "Clay Cross, Friday" -s→ valid JSON, 142 entries starting 16/10/2026, cross-checked against the published South
calendar including the bank-holiday weeks and the December/January boundary.
python -c "import json; json.load(...)"oninput.json→ clean.web_driverconsistency: the entry has noweb_driverand the class no longer callscreate_webdriver. (The CI script itself reads from the GitHub API and was rate-limitedlocally, so this was checked with the offline equivalent.)
poetry run blackon both changed files → unchanged.input.jsonis updated in the same commit:uprn,postcodeandweb_driverremoved,house_numberset toNorth, Monday,urlpointed at the calendar page withskip_get_url, andwiki_noterewritten to say exactly what to enter and that existingusers must reconfigure.
Targets #1881 — left open rather than auto-closed so the reporter can confirm against their
own address and calendar.
🤖 Generated with Claude Code