Don't rely on HEAD requests to check links

[thomasleese]

It turns out that some servers don't fully support the HEAD method, and instead of returning the same status code as they would for get, they return something different like 404 or 500.

We cannot, therefore, rely on the HEAD request to give us useful information about the link and instead make a GET request. This does pose the problem of downloading unnecessarily large files, so some investigation needs to be made into whether it's possible to fake a HEAD request by terminating the connection as the content starts to come through.

[thomasleese]

The majority of the fix has been implemented in #65 but it's still worth finding a way of avoiding downloading large files.