Add support for Pelias geocoder

[barbeau]

Mapzen offers an open-source geocoder named Pelias:
https://github.com/pelias/pelias

Pelias is used in Eraser Map, an Android app, which would be a good reference for adding Pelias to OTP Android (although from a glance it appears much of it is written using Kotlin).

I'd like to switch to this as our primary geocoding provider by default, even to potentially use this before the Android geocoder in our geocoding hierarchy.

So new geocoding hierarchy would probably look like:

• Pelias
  • Android native geocoder
    • Nominatim (based on user preference - default)
    • Google Places API (based on user preference)

...or:

• Pelias (based on user preference - default)
• Android native geocoder (based on user preference)
  • Nominatim (based on user preference - default)
  • Google Places API (based on user preference)

...with failover cascading to the next level when results from the previous level aren't available. We should probably have a fail safe setting to turn off Pelias geocoding in case it's not very good in some areas - I'll need to think a bit more which hierarchy makes the most sense.

Reasoning for switching to Pelias as a default is that we can use OTP Android as a sandbox to better understand what users are searching for in context of multimodal travel. This will hopefully lead to improved transit search. Mapzen is committed to open data and open-source for their systems, and we're potentially looking towards gathering more information from user's queries (with user permission) to assist in the process of building a better transit geocoder. Mapzen can also leverage the Transitland datastore for building better transit geocoding.

Transit geocoding is currently primitive at best - currently none of the geocoders we're using in OTP Android support transit data. In OneBusAway Android we have search only for stop and route IDs, which users complain about. Bottom line is we need a better way to support search for addresses, points of interest, and transit information, and I think in the long-term Pelias is our best chance to get there.

There are currently rate limits on a Mapzen API key (6 requests/sec or 30k per day) - I don't see us hitting these anytime soon for OTP Android alone, but from discussions with @dianashk we would be able to work with Mapzen to bump the limit if needed.

[barbeau]

It looks like Pelias returns GeoJSON features, along with some additional info - from https://github.com/pelias/pelias#developer-documentation--api-access:

    "geocoding": {
        "attribution": "https://search.mapzen.com/v1/attribution",
        "engine": {
            "author": "Mapzen",
            "name": "Pelias",
            "version": "1.0"
        },
        "query": {
            "boundary.circle.lat": 40.74358294846026,
            "boundary.circle.lon": -73.99047374725342,
            "boundary.circle.radius": 500,
            "point.lat": 40.74358294846026,
            "point.lon": -73.99047374725342,
            "private": false,
            "querySize": 1,
            "size": 1
        },
        "timestamp": 1460736907438,
        "version": "0.1"
    },
    "type": "FeatureCollection",
    "features": [
        {
            "geometry": {
                "coordinates": [
                    -73.99051,
                    40.74361
                ],
                "type": "Point"
            },
            "properties": {
                "borough": "Manhattan",
                "borough_gid": "whosonfirst:borough:421205771",
                "confidence": 0.9,
                "country": "United States",
                "country_a": "USA",
                "country_gid": "whosonfirst:country:85633793",
                "county": "New York County",
                "county_gid": "whosonfirst:county:102081863",
                "distance": 0.004,
                "gid": "geonames:venue:9851011",
                "id": "9851011",
                "label": "Arlington, Manhattan, NY, USA",
                "layer": "venue",
                "locality": "New York",
                "locality_gid": "whosonfirst:locality:85977539",
                "name": "Arlington",
                "neighbourhood": "Flatiron District",
                "neighbourhood_gid": "whosonfirst:neighbourhood:85869245",
                "region": "New York",
                "region_a": "NY",
                "region_gid": "whosonfirst:region:85688543",
                "source": "geonames"
            },
            "type": "Feature"
        }
    ],
    "bbox": [
        -73.99051,
        40.74361,
        -73.99051,
        40.74361
    ]
}

We're already using Jackson for JSON deserialization, so we can use this POJOs library to help us deserialize Pelias responses:
https://github.com/opendatalab-de/geojson-jackson

We're looking at switching to Pelias in our USF Maps app - related issue at CUTR-at-USF/usf-mobullity#176.

[dianashk]

Hey @barbeau, glad to see you're making progress on this! Let us know if you have any questions about the service. We can also connect you with our Mobile team to offer insight into our Android SDK. Looking forward to seeing the end product!

[barbeau]

@dianashk Ah, nice - https://github.com/pelias/pelias-android-sdk. I was actually looking to see if Mapzen had an Android library for Pelias, but I didn't see that on https://github.com/pelias/pelias or https://mapzen.com/documentation/search/get-started/ docs. That definitely helps on Android (an aside - USF Maps App (https://maps.usf.edu/) is responsive webapp based on OTP, with geocoder implemented server-side - if you have a Java server-side client library for Pelias let me know).

From a quick look, it appears that the only thing that might prevent us from using the Pelias Android SDK is that it requires Android API Level 15 (4.0.x Ice Cream Sandwich) or higher. In OBA and OTP Android we're still trying to support back to API Level 9 (2.3 Gingerbread), primarily for equity reasons to offer the service on older devices.

I assume the requirement of API Level 15 or higher is a hard limit that can't be changed, probably because of a dependency within the Pelias Android SDK (e.g., Retrofit?)?

[ecgreb]

According to the README Retrofit supports back to Android 2.3 so we are good there.

I just tried setting minSdkVersion 9 in the Pelias Android SDK and ran a few quick experiments:

• The library compiles however most of the unit tests fail. Probably due to some incompatibility with Robolectric but it seems like we could solve this with better config of the test environment.
• Running Android lint only produced one NewApi error which is easily fixed with a for loop. "Call requires API level 11 (current min is 9):android.widget.ArrayAdapter#addAll"
• PeliasSearchView extends the backwards compatible version of SearchView from the support library so we are good there as well.

The main issue would be finding a solution for Robolectric and then actually testing the library on devices running Gingerbread but sounds like it might be worth a try.

[barbeau]

Thanks @ecgreb! I appreciate you looking into this. (An aside - I actually have an SO post for the ArrayAdapter.addAll() issue that's a quick fix).

I'm wondering if it's just simpler for our group to throw together a POJOs project for Pelias and leverage Jackson, which we're already using in OTP and OBA Android. We could build upon this existing GeoJSON POJOs project.

Our on end, for supporting Gingerbread devices this may be preferable because a) fewer dependencies so APK is smaller (and app size is a big issue on these older devices) and b) runtime is likely more efficient and app data cache is likely smaller because we're instantiating one deserialization library instead of two. This method would also be usable on normal Java clients (another problem I need to solve).

I'll see if I can pull together a proof-of-concept for this setup.

Until your above comment on SearchView I didn't realize that you included an auto-complete widget in the Pelias SDK - very nice! I'll have to take a closer look at that.

Is there a demo Android app that shows the various Pelias Android SDK functionality?

[dianashk]

Check out EraserMap in the playstore. That's our privacy centric app that brings together all the various Mapzen services. You can check out autocomplete (and search) in there.

As for a server-side SDK I'm not sure exactly what you need. Could you elaborate? What language is that backend written in? Thanks.

[barbeau]

Thanks! Will check out EraserMap. For server-side SDK, I meant a Java client library.

[dianashk]

Ah got it. Unfortunately we don't currently have a Java client or know of any in the open source community. If you write one or come across something already available definite keep us posted!

[ecgreb]

@barbeau you can also find an autocomplete search view example in the Mapzen SDK demo app. Source code is here and the demo app APK can be downloded here.

We have discussed the idea of converting the standalone Pelias Android library into a pure Java client and moving all the Android specific components (like the search view extension) into the newer Mapzen Android SDK but even if we do that the Java client would likely still be based on Retrofit and Gson.

[barbeau]

Thanks @ecgreb!

I've thrown together an early version of a Java library here - https://github.com/CUTR-at-USF/pelias-client-library. I haven't pushed the Maven artifacts yet so you can't yet include it in another project as shown in the README, but you can take a look at the README and code to get an idea of what it looks like. Right now it supports simple search using API key and search text - I'll look at fleshing it out a bit as we start using the various functionality.

[barbeau]

@dianashk Is the main difference between the Mapzen /search and /autocomplete API endpoints that /search is synchronous (meaning that responses for requests will always be in-order, but may be slower) while /autocomplete is asynchronous (meaning that responses for requests will be returned as quickly as possible, which could be out-of-order)?

This seems to be implied in the docs, but isn't stated explicitly.

[dianashk]

@barbeau, the two endpoints actually differ quite a bit in the query logic as well. With /autocomplete we assume that the user is not done typing the last token in the string and allow for more variation in the results. With /search we try to be more strict in our matches as we assume what the user sent across is complete and exactly what they intended (while accounting a bit for spelling errors).

The async mechanism is something entirely handled by the client, since on the server all requests are coming in asynchronously and we have no way of know that they are connected. The /autocomplete client code neatly handles what could potentially be a messy situation when you send out a lot of async requests in a short period of time and the order in which they come back makes a big difference in user experience.

[barbeau]

@dianashk Ok, thanks, that makes sense!