fix(doncaster)!: rewrite onto the council's OnCreate waste app

[robbrad]

⚠️ Breaking change: Doncaster now needs a postcode and house number, not a UPRN

Every existing Doncaster user must reconfigure the integration after the release that carries this. The council's replacement app has no UPRN route at all (evidence below), so there was no way to keep the old input.

Why the rewrite

DoncasterCouncil.py called https://www.doncaster.gov.uk/Compass/PremiseDetail/GetCollectionsForCalendar with a UPRN and epoch Start/End params. Verified live from a UK residential connection on 2026-10-15:

• That endpoint now 302s to /Themes/DMBC.Theme2/Static/Error.html?aspxerrorpath=..., an ASP.NET error page served with HTTP 200. The scraper's if response.status_code != 200 guard therefore passed and json.loads died with JSONDecodeError: Expecting value: line 1 column 1 (char 0) — exactly the Home Assistant failure in Doncaster: setup fails with JSON decoding error; council lookup now uses OnCreate #2256.
• https://www.doncaster.gov.uk/Compass/Entity/Launch/D3/ renders "Nothing To Display".
• The council's bins page links "Check when your collection days are" to https://doncaster-wasterecycling.oncreate.app/w/webpage/bin-query, a Netcall Liberty Create app. A plain GET returns a 35 KB JS shell with an empty <title> and no form inputs, so it needs Selenium.

Why the UPRN could not be kept

I checked specifically for a UPRN route before accepting a breaking change:

Attempt	Result
The bin-query form	Postcode is a mandatory field; the other box is "Property Number or Name", a substring search. There is no UPRN field.
address-collections?uprn=100050768956	404
address-collections?id=58989 without a token	403
The id in the results links	A Liberty Create record id, not a UPRN, and it arrives alongside a per-session webpage_token HMAC and auth value that cannot be derived client-side.

So postcode + property is the only way in, and uprn is replaced by postcode + house_number, matching SevenoaksDistrictCouncil, which drives the same platform.

How the new scraper works

1. Opens bin-query, fills Postcode and Property Number or Name, submits.
2. Picks the right row out of the results table and opens its collection page.
3. Reads the dates out of the calendar.

Three things about this platform are worth knowing, and the code is shaped around all three:

• The results listing is paginated at 25 rows (num_data_rows: 25, pagination_bottom: "dynamic" in the widget's own settings), so the property box has to be filled rather than relying on a postcode-only search. Because that box is a plain substring match against the council's record, the scraper falls back to a postcode-only search when the property text finds nothing, then matches locally.
• Matching is on whole words, with the leading words preferred. The site's substring search answers a request for house number "3" with both number 3 and number 13 on the same street, and a request for "23" turns up Rear Of 23 ... alongside 23 ..., so neither a naive substring test nor "first result wins" is safe.
• The calendar is FullCalendar and only draws one month, but Liberty Create hands the widget its whole event list up front, in the container's data-params. Parsing that attribute returns every published date with no month-by-month clicking. The rendered grid (td.fc-daygrid-day[data-date] plus .fc-event-title) is kept as a fallback.

The form fields are located through the for attribute of their visible labels, because Liberty Create generates the input ids from internal page references (payload[PAG0000624GBSLB1][PWG0002347GBSLB1]...) that change whenever the council rebuilds the app.

No User-Agent is forced on the driver (create_webdriver(..., None, ...)). This app has no bot management of its own, and a hardcoded string would claim an OS and Chrome version that the browser's own JavaScript then contradicts.

input.json

Field	Before	After
uprn	100050768956	removed
postcode	—	DN6 9EU
house_number	—	The Laurels Nursing Home
url	the dead Compass launcher	the OnCreate bin-query page
web_driver	—	http://selenium:4444 (the class calls create_webdriver)
wiki_note	FindMyAddress / UPRN	postcode + house number or name, and that a UPRN is no longer accepted

The test address is non-residential, but it took some finding

Doncaster's app only holds data for properties on a domestic collection round, and almost everything institutional in the borough is on trade waste and returns "There is no collection information available for this address". Confirmed against the live site for schools, academies, pre-schools and day nurseries, branch libraries, a community centre, a scout and guide association, a cricket club, a social club and institute, a miners' welfare recreation ground, an air cadet centre, a children's centre, an assembly hall, a leisure centre, the Civic Offices, and a supermarket — all empty.

Finding one that works took a sweep of 174 postcodes (prioritising village parishes, where community buildings are likeliest to be on a domestic round) plus 125 individual named-property checks. Two registered nursing homes do return collections:

• The Laurels Nursing Home, High Street, Norton, DN6 9EU — 15 collections, Refuse and Recycling weekly alternating. Used as the fixture.
• Headingley Park Nursing Home, Headingley Way, Edlington, DN12 1SB — 12 collections, but no Recycling, so a thinner test.

The Laurels is a CQC-registered care business whose address is published in the CQC register and the usual care-home directories, so it is a business premises rather than anyone's home, and the property name says so in the fixture itself. The second commit also drops the residential street names from the unit test fixtures — those are just as public as input.json — in favour of this address and hypothetical numbers on the same street.

If a nursing home is still the wrong call, the honest alternative is that there may be no suitable address at all: nothing institutional in the borough that I could find is on a domestic round.

Test plan

Run from a UK residential connection with Chrome 154 installed, not from a sandbox.

Verified:

• pytest uk_bin_collection/tests/step_defs/ -k Doncaster --local_browser True -q — 1 passed, 8.9 s. Re-run after the fixture change.
• The CLI end to end: collect_data.py DoncasterCouncil <url> -s -p "DN6 9EU" -n "The Laurels Nursing Home" returns 15 collections (Refuse, Recycling), 06/10/2026 to 12/01/2027, matching the live page in a browser. This exercises the house-name lookup path, not just a number.
• pytest uk_bin_collection/tests custom_components/uk_bin_collection/tests --ignore=uk_bin_collection/tests/step_defs/ -q — 364 passed, 0 failed (339 before this change, plus 25 new). The playbook's known sandbox-only failures (test_check_postcode_valid, test_check_postcode_invalid, test_create_webdriver_local) pass here because this machine has network.
• black --check clean on both changed Python files.
• Offline parity (playbook §5): nothing missing from input.json, no orphan keys, 352 entries, valid JSON.
• web_driver-vs-create_webdriver consistency: Doncaster is consistent. That check flags 14 pre-existing mismatches on master (Bexley, Bromley, Chelmsford, East Riding, Epping Forest, Kingston upon Thames, North Warwickshire, Peterborough, Rugby, South Holland, Stirling, Windsor and Maidenhead, Wirral, Wokingham), all untouched here.
• New unit tests: 25 cases in uk_bin_collection/tests/test_doncaster_council.py covering several waste types with dates, a subscribed service whose dates have run out (seasonal garden waste), duplicate and unreadable events, the "no collection information" error, a page with no calendar at all, the grid fallback, address matching, and the driver being closed on both success and timeout.

Not verified:

• Only the one test address was exercised live. The postcode-only fallback path (used when the property box matches nothing) is covered by unit tests, not against the live site.
• tests/check_selenium_url_in_input.json.py reads input.json from the GitHub API, so locally it checks nothing; I ran an offline equivalent instead.
• The Home Assistant config flow was not exercised in a running HA instance. The field mapping was read from config_flow.py (house_number → required number, web_driver → optional Selenium URL).
• Whether CI's own runners can reach oncreate.app — playbook §4 notes CI IPs are blocked by many council sites.

Credit

Diagnosed by @Smokinloads in #2256, who had already identified the migration to OnCreate, the HTTP 200 error page and the exact failure mode before any of this work started. This PR is their diagnosis implemented.

Targets #2256 rather than Fixes — the live test passes for the project's test address, but it would be good to have the reporter confirm it works for their own property, since they are the one who has to reconfigure.

Not done here

Not merged and not released — that is the maintainer's call. docs/MAINTAINER_PLAYBOOK.md is deliberately untouched to avoid conflicting with the concurrent Fylde retirement PR, and wiki/Councils.md is left to the Deploy - Wiki workflow.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Updated Doncaster collection lookups to use the council’s current website and search by postcode plus house name or number.
  • Improved collection results by reading calendar events across available dates, with a fallback to the displayed calendar.
  • Added clearer errors when an address or collection information cannot be found.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 75b2b60f-a35c-4f4c-a119-8a12b3760da1

📥 Commits

Reviewing files that changed from the base of the PR and between d62b4f1 and d5bf98b.

📒 Files selected for processing: 3

• uk_bin_collection/tests/input.json
• uk_bin_collection/tests/test_doncaster_council.py
• uk_bin_collection/uk_bin_collection/councils/DoncasterCouncil.py

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 0 remain after this review.

📝 Walkthrough

Doncaster lookups now use the Liberty Create waste app instead of the retired Compass endpoint. The scraper searches by postcode and house name or number, selects the matching address, and parses collection dates from its calendar.

Changes: Doncaster collection lookup

Layer / File(s)	Summary
Search and select an address uk_bin_collection/uk_bin_collection/councils/DoncasterCouncil.py, uk_bin_collection/tests/test_doncaster_council.py	The scraper searches by postcode and property, scores address matches, and retries without the property filter when needed. Tests cover match scoring and search results.
Extract calendar events uk_bin_collection/uk_bin_collection/councils/DoncasterCouncil.py, uk_bin_collection/tests/test_doncaster_council.py	The parser reads widget event dates or falls back to the rendered calendar grid. It removes duplicate events, sorts results, and handles missing collection information and unrecognized calendars.
Validate and run the lookup uk_bin_collection/uk_bin_collection/councils/DoncasterCouncil.py, uk_bin_collection/tests/input.json, uk_bin_collection/tests/test_doncaster_council.py	parse_data validates the postcode and property input, runs the browser lookup, and quits the driver. The sample input and tests cover lookup errors and driver cleanup.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant parse_data
  participant SeleniumWebDriver
  participant LibertyCreate
  participant CalendarParser
  parse_data->>SeleniumWebDriver: Submit postcode and property search
  SeleniumWebDriver->>LibertyCreate: Request matching addresses
  LibertyCreate-->>SeleniumWebDriver: Return address results
  SeleniumWebDriver->>LibertyCreate: Open the selected address
  LibertyCreate-->>SeleniumWebDriver: Return the address calendar
  parse_data->>CalendarParser: Parse calendar events
  CalendarParser-->>parse_data: Return dated collections

Loading

Merge Risk: ⚪ Minimal · up to d5bf9

No confirmed issue blocks merging. The live app’s behavior for a search with no matching addresses remains unverified.

Pre-merge checks: 5

Check name	Status	Explanation
Docstring Coverage		Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 32 functions across 2 files. (1 skipped: 1…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: rewriting the Doncaster scraper to use the council's OnCreate waste app.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 94.73684% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.47%. Comparing base (85a38d3) to head (d5bf98b).
⚠️ Report is 7 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...ion/uk_bin_collection/councils/DoncasterCouncil.py	94.73%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2282      +/-   ##
==========================================
+ Coverage   80.97%   82.47%   +1.49%     
==========================================
  Files          16       34      +18     
  Lines        1619     2596     +977     
==========================================
+ Hits         1311     2141     +830     
- Misses        308      455     +147     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[robbrad]

Cherry-picked into #2277 unchanged (9f06517e + d5bf98bc), so closing this one.

Nothing wrong with the work — bump.yml runs on every push to master, so merging each council separately would have cut a release each. #2277 now carries this whole pass as one merge and one release, and its body states the squash title must be fix!: so your breaking change still drives the version bump and reaches the changelog.

Thanks for checking the fixture properly and swapping it. The Laurels Nursing Home is a registered business at a named premises rather than a private household, which is what input.json needs. Re-verified independently: live test passes, 15 collections.

One observation, not a request: two of those 15 dates are in the past. That's harmless, because the integration's process_bin_data only keeps collection_date >= current_date, so Home Assistant users see the right next collection either way. Worth knowing that 11 other councils filter at the scraper instead, if it ever comes up for direct CLI or API consumers.