Standardising the format of input fields

[davida72]

In a previous post, I mentioned that I felt the onboarding process could be smoother. An area I think could be improved is the standardisation of input fields, such that council scripts expect them in the same format. At the moment, there are a number of special cases which I think could be removed.

To be clear, I'm not suggesting adding or removing input fields, just standardising the format. Inputs remain:

• Postcode
• UPRN
• Address (currently House Number)

The Home Assistant dialog box only asks for those that are needed, as I believe is the way it works currently.

My motivation is that I believe onboarding of the addon would be smoother, particularly if input fields could be auto-populated using API calls to endpoints such as Postcodes.io although I'll leave that topic for another thread.

Of the 316 councils, 203 use UPRN, which is already in a standardised format. 134 of them use postcode, which is also already in a standardised format. It's the other input fields that I think could be improved.

Examples

Islington - URL Subtitution

{
    "IslingtonCouncil": {
        "uprn": "5300094897",
        "url": "https://www.islington.gov.uk/your-area?Postcode=unused&Uprn=5300094897",
        "wiki_command_url_override": "https://www.islington.gov.uk/your-area?Postcode=unused&Uprn=XXXXXXXX",
        "wiki_name": "Islington Council",
        "wiki_note": "Replace XXXXXXXX with your UPRN.",
        "LAD24CD": "E09000019"
    }
}

This is a fairly common format where a URL needs to be substituted with the UPRN, but the script could simply append the UPRN to the URL itself. It only requires UPRN.

Charnwood - URL Substitution

{
    "CharnwoodBoroughCouncil": {
        "url": "https://my.charnwood.gov.uk/location?put=cbc10070067259&rememberme=0&redirect=%2F",
        "wiki_command_url_override": "https://my.charnwood.gov.uk/location?put=cbcXXXXXXXX&rememberme=0&redirect=%2F",
        "wiki_name": "Charnwood Borough Council",
        "wiki_note": "Replace XXXXXXXX with your UPRN, keeping \"cbc\" before it.",
        "LAD24CD": "E07000130"
    }
}

A variation on the URL substitution, where the UPRN could be inserted in the URL in the script.

Bromley - Custom URL

{
    "BromleyBoroughCouncil": {
        "url": "https://recyclingservices.bromley.gov.uk/waste/6087017",
        "web_driver": "http://selenium:4444",
        "wiki_command_url_override": "https://recyclingservices.bromley.gov.uk/waste/XXXXXXX",
        "wiki_name": "Bromley Borough Council",
        "wiki_note": "Follow the instructions [here](https://recyclingservices.bromley.gov.uk/waste) until the \"Your bin days\" page then copy the URL and replace the URL in the command.",
        "LAD24CD": "E09000006"
    }
}

This is a special case that I don't think needs to be. You can just navigate to their website and enter your postcode, e.g. BR1 1RY.

Castlepoint - Four Digit Code

{
    "CastlepointDistrictCouncil": {
        "skip_get_url": true,
        "uprn": "4525",
        "url": "https://apps.castlepoint.gov.uk/cpapps/index.cfm?fa=wastecalendar",
        "wiki_name": "Castlepoint District Council",
        "wiki_note": "For this council, 'uprn' is actually a 4-digit code for your street. Go [here](https://apps.castlepoint.gov.uk/cpapps/index.cfm?fa=wastecalendar) and inspect the source of the dropdown box to find the 4-digit number for your street.",
        "LAD24CD": "E07000069"
    }
}

This would work by using an address, albeit by switching to Selenium.

Midlothian - House Number

{
    "MidlothianCouncil": {
        "house_number": "52",
        "postcode": "EH19 2EB",
        "skip_get_url": true,
        "url": "https://www.midlothian.gov.uk/info/1054/bins_and_recycling/343/bin_collection_days",
        "wiki_name": "Midlothian Council",
        "wiki_note": "Pass the house name/number wrapped in double quotes along with the postcode parameter.",
        "LAD24CD": "S12000019"
    }
}

In this example, a house number is expected in the house number field.

Richmond - Street Name

{
    "LondonBoroughOfRichmondUponThames": {
        "house_number": "March Road",
        "skip_get_url": true,
        "url": "https://www.richmond.gov.uk/services/waste_and_recycling/collection_days/",
        "web_driver": "http://selenium:4444",
        "wiki_name": "London Borough Of Richmond Upon Thames",
        "wiki_note": "Pass the name of the street ONLY in the house number parameter, unfortunately post code's are not allowed.",
        "LAD24CD": "E09000027"
    }
}

In this example, a road name is expected in the house number field.

Brighton - Carefully Formatted Street Name

{
    "BrightonandHoveCityCouncil": {
        "house_number": "44 Carden Avenue, Brighton, BN1 8NE",
        "postcode": "BN1 8NE",
        "skip_get_url": true,
        "uprn": "22060199",
        "url": "https://cityclean.brighton-hove.gov.uk/link/collections",
        "web_driver": "http://selenium:4444",
        "wiki_name": "Brighton and Hove City Council",
        "wiki_note": "Use the full address as it appears on the drop-down on the site when you search by postcode.",
        "LAD24CD": "E06000043"
    }
}

In this example, a carefully formatted address is required in the house number field. In all three examples, if address was used instead of house_number then there would be consistency and scripts could fuzzy-search to the correct entry on the council website.

Google Calendar (Public) - iCal Lookup

{
    "GooglePublicCalendarCouncil": {
        "url": "https://calendar.google.com/calendar/ical/0d775884b4db6a7bae5204f06dae113c1a36e505b25991ebc27c6bd42edf5b5e%40group.calendar.google.com/public/basic.ics",
        "wiki_name": "Google Calendar (Public)",
        "wiki_note": "The URL should be the public ics file URL for the public Google calendar. See https://support.google.com/calendar/answer/37083?sjid=7202815583021446882-EU. Councils that currently need this are Trafford.",
        "supported_councils": [
            "TraffordCouncil",
            "ClackmannanshireCouncil",
            "HavantBoroughCouncil",
            "NorthWarwickshireBoroughCouncil",
            "NewryMourneAndDownDistrictCouncil",
            "EastDunbartonshireCouncil",
            "PendleBoroughCouncil",
            "TorfaenCountyBoroughCouncil",
            "EastHampshireCountyCouncil",
            "RibbleValleyCouncil",
            "BrentwoodBoroughCouncil",
            "IsleOfWightCouncil",
            "WestmorlAndFurnessCouncil",
            "DerryAndStrabaneDistrictCouncil",
            "NorwichCityCouncil"
        ],
        "supported_councils_LAD24CD": [
            "E06000046",
            "E07000068",
            "E07000085",
            "E07000090",
            "E07000124",
            "E07000218",
            "E08000009",
            "N09000005",
            "N09000010",
            "S12000005",
            "S12000045",
            "W06000020",
            "E07000122"
        ]
    }
}

Interesting! This presumably could be standardised by having an entry per council, ignoring all inputs, then the script using the appropriate URL.

Considerations

• If there are code-breaking changes then that leads to a poor experience for existing users, but I think it would be possible to make it backwards compatible.
• I haven't looked through all the scripts, so there may be some that absolutely require a special case, but I haven't found one yet.

Standardisation

By standardising the format of input fields and avoiding special cases, I think it lays a path for a smoother onboarding and simplification of documentation.

Happy to do some legwork in modifying council scripts to work with a standard formatting.

[davida72]

I did a quick parse of councils that look to be special case.

Edit: Ignore this. My method for finding special case councils was flawed. There are plenty of special case councils that take the UPRN field.

[davida72]

I've been thinking about this a fair amount.

Contrary to what I said in the initial post, I think house_number should remain the search field alongside uprn and postcode although

[davida72]

I've been thinking about this some more and, contrary to what I said in the original post, I think the three input fields should remain

- uprn
- postcode
- house_number

House number should just take house number or house name. I'd love a common function that reliably found a house name or number in an address - I'll see if one exists.

For the URL substitution councils, I'd prefer to see a solution like Melton that simply pulls the information with

url = f"https://my.melton.gov.uk/set-location?id={user_uprn}&redirect=collections&rememberloc="
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

Although there maybe some work to ensure that users existing configurations work with that system.

[robbrad]

Really appreciate the ideas you’re bringing to the table — always good to challenge how we work.

Just to add a bit of context: not all councils are like Melton. Some are even better (with open, unprotected APIs), but plenty aren’t — and not all of them accept UPRNs either. Councils invest in teams of developers, third-party services, and software specifically to protect their infrastructure (what a great way to spend my council tax 😅).

In a lot of cases, just visiting the site will generate a bunch of cookies, auth codes, and session tokens depending on things like the user’s device, browser, and other detection mechanisms.

Originally I wasn’t a fan of the Selenium approach either — it came out of necessity. The alternative methods were flaky, didn’t scale, and caused reliability issues. The speed to make a change and develop a selenium integration is a magnitude faster than trying to workout the backend system, even more so when a council changes. Selenium was chosen because it most closely mimics a real user browsing a council site.

At the moment, we’ve got coverage across 300+ councils. I don’t think it makes sense to risk that coverage or reliability just because getting Selenium working can be a bit fiddly. (It’s a fair point — it is fiddly — but so is supporting 300+ councils in a sustainable way.)

That said, I’m definitely up for making it easier to work with. I’m not wedded to Selenium specifically — I’m wedded to something that properly mimics user interaction (whether that’s Selenium, Playwright, or something else), while easy to maintain. We’ve already taken steps like the custom component that checks for Chrome installation or connects to a Dockerized Selenium server.

Open to ideas — as long as we keep the ease to maintain, reliability and coverage intact.

[robbrad]

Furthermore- selenium should not be the automatic choice to integrate either - if there is an api or url you can hit, without issue, this should be the primary choice

[davida72]

Nothing to disagree with here.

My point about Melton was simply that it's an example of a council script that takes a standard UPRN from the user and inserts it into the URL rather than

"wiki_command_url_override": "https://www.adur-worthing.gov.uk/bin-day/?brlu-selected-address=XXXXXXXX",
"wiki_note": "Replace XXXXXXXX with your UPRN. You will need to use [FindMyAddress]

which seems a bit clunky and unnecessary.

My point about standardisation of input formats was me thinking out loud. It's probably more helpful if I give specific examples of area of improvement and suggestions to address them.