Zero Config - Proof of Concept

[davida72]

Background

When I first found this project I ran into a bunch of problems getting it working properly in Home Assistant.

Admittedly there were points when I should have read the documentation better, but I didn't. I had all sorts of UPRN headaches before I realised I didn't need it, then all sorts of headaches setting up Selenium and finding a URL that worked on my network. After that it took a while before I realised that the documentation was incorrect as it was asking me for a house number but it actually needed an exact address match as it appears in the council's dropdown list.

I wanted to see if I could create an easier setup.

The Idea

Could I completely automate the setup process so that no configuration was required?

The Challenges

• How do I get address details needed?
• How do I establish the correct council script?
• How do I connect to a remote API server and Selenium instance?
• How do I handle special case searches?
• How much would it cost to run these services?

The Process

Home Assistant provides Latitude and Longitude. Those can be passed to a Ordnance Survey API which returns a bunch of useful information including nearest property, its address and UPRN. The postcode can then be passed on to the Postcodes API which returns information about the neighbourhood including LAD24CD code. At this point, we have a lot of useful information about the property.

Using the LAD24CD I refer to the input.json and establish a matching council script if it's available. I can also use input.json to populate other information, ultimately creating an API URL.

From there I call an instance of uk_bin_collection_api_server running on a hosted docker container, which in turn uses a Selenium node on the same machine.

And IT WORKS! Here's a link to a proof of concept. Use the map to click on a building, which sends Latitude and Longitude.

Most councils scripts that use a UPRN work first time, including ones which need Selenium. For those that need a specific search item, like house number or exact address, the scripts fall down because it's not in the expected format, but I've allowed the input fields to be editable so you can manually adjust. TL;DR special cases don't work.

I'm using three external services:

• OS Places API. I believe this is free up to £1,000 of API calls each month, which I doubt we'd exceed.
• Postcodes API. Free.
• Server for UKBinCollection API and Selenium. £3 per month. Three Selenium nodes can run in parallel.

Outstanding Issues

1. The special case searches. I believe a more elegant way of handling it in the scripts would be to have a common fuzzy address matching method. There are some good libraries that give reliable results and nobody would have to change any settings, but the council scripts would have to be adjusted.
2. Selenium overloading. Obviously everyone going through one Selenium instance is a chokepoint. I don't know how many people use the plugin but it feels workable with some changes. Three Selenium nodes; limit updates to every 24 hours; some queuing system.
3. Cost. £3 a month is okay, but I need to doublecheck the OS API costs.

Just a set of ideas that might make setup easier.

[davida72]

I'd also be interested to know whether it's possible to use every council script using

• postcode
• uprn
• canonical address from uprn

Feels like it would be good to have a consistent input format and avoid special cases. Happy to do some legwork to adjust special case scripts, assuming it's possible.

[robbrad]

Nice work @davida72

Running a managed service as you describe in the detail above has been on my mind for a long time(years in fact)

There are other options for online selenium AWS Device Farm - desktop browser testing being one.

I haven’t taken the project down this route for a few reasons.

1. The support required to keep the service level of a high enough bar that people would pay
2. We would then take some responsibility for scraping council data, data handling etc I’m not a fan of that - I feel like it needs to be the individual doing the scraping. Don’t know the legals on this.
3. The project is only here today due to the hundreds of people who have contributed and I’m not sure how I’d fairly split any of that money etc. if it was above running costs. Maybe charity?

With that said I’m not against exploring how we could make the process easier but I’d rather avoid making the project a black box.

[robbrad]

Maybe I miss understood the money ask. Are you saying we cover it?

Some more data for you - GitHub has 4 runners which run nightly with selenium. 312+ councils take 48 minutes to execute meaning roughly 1.6 councils a minute.

If we api controlled or throttled the execution one server could support 16,380 people making 1 request a week

[robbrad]

Do you have any idea how many people use the Home Assistant plugin?

Useful stats. Average script time of 40 seconds seems high. I haven't seen any that length and the non-Selenium scripts are much quicker. I wonder if there are some scripts timing out that push up the average?

The selenium scripts do take longer. I could prob mine some data to get avg time per council to drive the long ones down.

There are some, where they bring the average up

Do we want to scrape councils on behalf of people using infrastructure we manage? I’m not sure how defendable that is if a council takes action VS “I’m using a tool to scrape my own councils info.”

Also due to the selenium endpoint going to any website you throw at it you want to protect it. As it could be abused

I would suggest if people don’t want to host their own selenium then they use a managed service like aws device farm that they commission and pay for, then it’s their responsibility.

I don’t have any stats on number of users as that’s hard to track

[davida72]

I'm more interested in the possibilities of standardising the script inputs by using lat/lon taken from Home Assistant then other details from free API calls.

The hosted API server I'm ambivalent about. I'm personally not worried about the legal ramifications of scraping a council website on behalf of users, although maybe they'd notice the traffic coming from a single IP. I think the cost, if it could be kept to £3 a month in total for all users, is manageable. But overall it might not be worth the headache and responsibility.

On the first point, I'm minded to write a python script that takes lat/lon and returns UPRN, canonical address, postcode and LAD24CD. If Home Assistant used that to automatically establish details I think that would be a smoother onboarding process. Before incorporating it into the HA custom component the scripts would need to accommodate the standardised inputs.

Maybe the first small step would be to adjust a script to take a standardised input as well as whatever people have it set to currently so that it's backwards compatible.

If you haven't already, take a look at the json that comes back from the Ordnance Survey API. It's at the bottom of the page.

[robbrad]

The standard inputs remain as expected: house number, postcode, and UPRN — we already have functions to handle these. Adding an address lookup feature could be a nice enhancement, but it would require significant refactoring unless the lookup process returns the same fields as the existing pip library.

Some contributors have used alternative field names or formats for inputs, which has led to inconsistencies. Ideally, this should be corrected. However, since councils don’t follow a standard structure, we’ll need to analyze patterns across them and explore ways to standardize or fill in gaps where possible.

On the topic of scraping councils on behalf of others: I don’t support this approach. Most councils explicitly prohibit this in their terms of use. Yes, I understand that sounds contradictory given we provide tools that make scraping possible, but how people choose to use them — just like when checking their own bin dates — is their responsibility.

That said, I wont host or provide a server that scrapes websites I don’t own or control, especially without explicit permission from the relevant council. In the past:

1.	Councils have interacted directly through this repository,
2.	Reached out via LinkedIn,
3.	Or contacted me by email.

So if an issue arises, they know how to get in touch, and an IP being detected isn’t the only identifiable data

[davida72]

It's interesting that some councils have reached out in the past. All things considered, I agree with your decision not to host a service that others could use, so let's draw a line under that.

On the point of standardising inputs, I don't think I'm explaining myself well and I've made a mess of the conversation threading here, so I'll start a new topic specifically on that point later.

[robbrad]

Thanks @davida72 - and sorry I don’t think I have helped- I was on a plane to Seattle

[robbrad]

@davida72 one way to reduce the setup would be to remove docker

it might be possible to ship the python package with a chrome standalone binary and then anyone who uses the package will have the dependencies to run the selenium integrations

[robbrad]

If we move to playwright we can remove some of the selenium issues allowing people to run this locally with no need for a container

should be zero config.

I’ll need to test this approach

[davida72]

My ideal setup would be

• pulls address info automatically using API calls
• standardised script inputs
• a way of self-hosting something Selenium-like without a setup

Then I think it just works out of the box.