Map Data Access through Tasking Manager

[kshitijrajsharma]

Hi everyone ! As part of HOT efforts for providing map data effectively , we are thinking on filling the gap between data access and creation . In this discussion I will like to discuss about the challenge/ problem we are thinking and we would like get your views/ideas around some research questions that we came up with.

Challenge / Problem :
We see a challenge in the difficulty of accessing and using the map data by our communities. Currently, HOT offers 1 main solution, through Export tool (and HDX) where data can be accessed. This works for HDX and its customers but doesn't have easier map data access for the data produced by tasking manager projects.
There might still be a gap in technical expertise , for people with GIS expertise it can be achieved . If one is not a GIS professional or lacks that expertise Is there any way or method from where we can easily access and visualize the data without downloading or having specific learning curve ?

General Problem :

• Lack of method/tools to directly access map data for general users
• Lack of visualization tools (Low level)

Current Scenario

Export tool Focuses on GIS users to overcome challenges of learning curve for overpass , Provides Data through HDX but on country level / Disaster prone areas , Does not focuses on tasking manager / communities around tasking manager , People still need GIS expertise to visually / play with the data. No visualization method has been adopted , People can’t directly look data

Now before jumping in to solutions directly , We want to first discuss about the problem . Here are the research questions that we need your input so that we can think about the solution / ideas together effectively

1. Do think this is actually a problem ?
2. Are their existing tools available to overcome this problem ?
3. If you are managing a project - How are you currently tackling this map data access and visualization problem ?
4. Do you think it would be easier if we think about the solution that can be integrated within tasking manager ?

There has been previous discussion as well around this topic , Pasting here for reference : #258

[emi420]

Great discussion @kshitijrajsharma !

1. Do think this is actually a problem?

Yes, definitely. All the tools that I know are hard to use for the novice user, that usually is the final beneficiary of the data. A first-hand example here are my fellow firefighters that don't know how to visualize relevant data about risk, mainly for forest and wildland-urban interface fires, but also for floods or vehicle accidents. I created a QGIS project for this but it was hard to use it and to keep it updated.

2. Are their existing tools available to overcome this problem?

AFAIK, not really. Mapbox Studio can be used in some way to create custom maps, but still is very limited, is not an easy-to-use tool, it's more oriented to technical users and its base map data isn't OSM only.

3. If you are managing a project - How are you currently tackling this map data access and visualization problem ?

One possible solution is to use Underpass. We can keep an up-to-date copy of the OSM database and also provide UI tools for visualization, including filtering the data. These tools are being build as React components, enabling developers to build easy-to-use visualization tools.

Demo link here (it's a dev testing website, it can fail anytime)

For example, you can create a map highlighting wetlands, schools, or any other feature:

As a personal project, I'm thinking into build a disaster map highlighting relevant features and aggregating data about fires (hotspots from the NASA FIRMS system), to create something like this:

4. Do you think it would be easier if we think about the solution that can be integrated within tasking manager ?

I think it will be good to think in solutions that can be integrated within the Tasking Manager and at the same time have the possibility of integration with other products.

[RAytoun]

This is definitely the idea that is needed to be able to "see" the data that does not get rendered on OpenStreetMap standard view.

[adiatmad]

1. Do think this is actually a problem ?
Sure thing! People who aren't too familiar with GIS stuff might feel a bit unsure about diving into using OSM data from Tasking Manager. IMHO when there are fewer folks who find OSM data user-friendly, they might hesitate to give their best shot at contributing.

2. Are their existing tools available to overcome this problem ?
I suppose there are tools out there, but it might take some people a little while to understand them. Consider Overpass Turbo as an illustration. Even though it's very impressive, it can take some time to fully understand. Oh, and there used to be this HOT Visualization Tool, but it has since been retired.

3. If you are managing a project - How are you currently tackling this map data access and visualization problem ?
Just get a GIS person on board to handle this and spread the word about what comes out of it.

4. Do you think it would be easier if we think about the solution that can be integrated within tasking manager ?
Totally! Think about an Export Tool that works hand in hand with the TM project. That would be fantastic for contributing and producing all of the OSM data in one location.

[anjarakrim]

I want to share my thought and experience in regard to data access in general in the HOT tasking manager. As a tasking creator, usually, after the mapping campaign is ended, some of our contributors (could be gov, student, or researcher) are asking how they can get the data.

• This is the first gap, making the data accessible to our contributors easily.
• For now, it's the creator's duty to set up another step in providing the data to the community, while it also takes time (second gap)
• Sometimes, even the tasking creator doesn't have enough skill to generate the data cause it will use many collaborative tools to be able to produce one download link/ 1 data set for the community (third gap)
• The last one is where to put and share the dataset sometimes, makes everyone confuse, and they need to open another link/platform that is not integrated at all with the HOT Tasking manager (last gap)

so... to answer this
Do think this is actually a problem ?
This is a problem for some. Since OSM data users are not limited to GIS experts, it confirms the need for a user-friendly tools for data access.
Are their existing tools available to overcome this problem ?
Existing tools we know exactly that it is constrained only for intermediate or gis expert
If you are managing a project - How are you currently tackling this map data access and visualization problem ?
Manually through the overpass, then our GIS specialist will take the next step for it.
Do you think it would be easier if we think about the solution that can be integrated within the tasking manager ?
I would love to see the integrated tools that help the community access the data, as well as see the visualization of data added during the mapping campaign in HOT Tasking Manager.

[RAytoun]

We already have a Humanitarian Layer on OpenStreetMap but it has never been developed.

If we can add a filter and method of selecting an icon or symbol to represent the feature chosen to be rendered then this would be brilliant.

An example would be to request all water_well=pump and be able to show which ones are working and which ones are not working or are only seasonal then you have this layer where it can be rendered without having to change to any other tools.

TagInfo does give an indication of distribution on a very small scale map without any other detail.

[Jorieke]

Great questions and discussion! Sharing my two cents:

Do think this is actually a problem ?
Yes, it is not easy for people to use or work with OpenStreetMap data. Even our GIS specialists sometimes struggle to retrieve the most recent OpenStreetMap data.

Are their existing tools available to overcome this problem ?
An easy to use tool I really like, but could see worked out a bit further is MyOSMatic: https://print.get-map.org/
Besides that, I think it is not very much known within the HOT community, but it is not that difficult to make some nice visual maps in overpass turbo. This is an example I saved here as example, it is from some time ago from the HOT Uganda project. https://bit.ly/2DpHZHo This is linking it as well to a Umap: http://umap.openstreetmap.fr/en/map/archived-functioningnon-functioning-wash-points_197813#11/3.1402/31.2451

If you are managing a project - How are you currently tackling this map data access and visualization problem ?
We are usually working with GIS specialists, but as I said they struggle as well.
HDX and other exports like geofabrik have the issue that it is not providing you with 'live' data. It can be weeks, sometimes months or years old. With Overpass, the Qgis QOSM plugin and the HOT export, you can only download subsets of data and/or you need to be knowledgeable for the OSM data model.
A lot of our GIS officers like to use the BBBike extract service: https://extract.bbbike.org/ also the OSM GPT has been flagged recently as an interesting tool to easily download data from OSM if you do not know much about OSM tagging: https://osm-gpt.rohitgautam.com.np/

Do you think it would be easier if we think about the solution that can be integrated within tasking manager ?
One one end I think yes, it would be easy to have a button on the TM that allows you to download data within the area of interest of the task. But this will only cover certain usages if active data creation is done. We use a lot more OSM data than just created through tasks. I however think having 1 entry point for all HOT tools would be very usefull, all is very much spread out currently.

[frozenrabi]

1. Do think this is actually a problem ?
I would not call this a problem but I would call this an issue that we were supposed to deal years ago. I had raised several conversations regarding this with various OSM enthusiasts and various HOT professionals. After years of contributing and volunteering on Openstreetmapping and not being able to use the OSM datasets just because of this export problem is very frustrating (I felt I was adding water to the sand). Even my colleagues and GIS Professionals drop me a message when the topic is OSM and they want to export OSM datasets of desirable AOI. It's been a long that HOT have been engaged on producing new mappers and making mappers to add features on map but isn't it too late to also equally explore and engage the GIS professionals, Cartographers and mappers to train and suggest way ahead for OSM data export. This is the main reason why the advanced mappers who had 100+ hours of OSM mapping experience are getting vanished. So I believe this sure is a issue that need to be addressed.

2. Are their existing tools available to overcome this problem ?
Yes, there was and there is but the thing is it need maintainace and upgrade. I was refering to HOT Export tool (https://export.hotosm.org/en/v3/). This tool was there since I was introduced to mapping back in 2019 and I'm afraid this tool too doesn't go unnoticed and disappear like OSM Galaxy. The other tool I generally use is Overpass turbo (https://overpass-turbo.eu/) and Quick OSM plugin in QGIS to overcome the issue of export.

3. If you are managing a project - How are you currently tackling this map data access and visualization problem ?
For the visualization, I generally use either Overpass turbo, JOSM Editor or Disaster Ninja. And for the access I use QGIS (for bulk datasets+ different features) and Overpass turbo for few specific features.

4. Do you think it would be easier if we think about the solution that can be integrated within tasking manager ?
This should not even be a question in the first place. It should be utmost priority to embed the export tool on every projects that are created in HOT Tasking manager. Just like the user are able to download task grids and AOI (expert mode), the users must also be able to download all the data from the same project in few clicks. While talking and advocating about open data for most of the times, we mappers feel betrayed because we are able to make our contribution easily while data exporting is always challenging and hectic. And 9 out of 10 trainings are related to mapping and validating rather than exporting and it's application. Similarly, upgrade of HOT export tool can also aid to this a lot. Being able to load AOI (Different formats), and able to extend the area of custom AOI or even draw the administrative boundary when user uses search box to find the place can be great addition to the new version.

[robsavoye]

There is a python module under development to make access to our raw OSM database much easier to build other tools on top of. Data extracts is now something multiple projects at HOT need to do, and we have our own high performance database schema which does require learning how to use it directly. It works with a local postgres as well as the REST API for Export Tool. Galaxy isn't dead, we just dropped the name and shifted the focus more towards data quality than general statistics. Many data extracts are similar, for example mapping amenities, so it also has canned queries for common data extracts. We use these data extracts in the field for FMTM, so initially they are focused on implementing the HOT data models for humanitarian data collection. Basically we are building infrastructure on top of Underpass, so we don't need to use Overpass anymore. This is already in use by multiple HOT projects, other contributors are welcome!

Sorry, this has nothing to do with TM, but there is a lot of exciting work going on now in the tech team.

[SColchester]

Spoke with a organization that creates projects often on Tasking Manager and they mentioned that being able to access the 'completed' data directly from Tasking Manager would be great. But specifically mentioned that it would also be great if it was available in a variery of geospatial formats e.g. geojson, KML, shapefile, OSM

[robsavoye]

We talked about this last week at the TM meeting with the new developers. As raw data extracts are now a python module. this would enable being about to download directly from Tasking Manager.

[petya-kangalova]

@kshitijrajsharma we had a discussion in the Tasking Manager meet up and talked about the key users and user cases : @ramyaragupathy

Who is this integration for? What is the main use?

• Closed Project - example: MSF would like to download the data for projects that have been completed. This can be done as a one off for all closed project. @kshitijrajsharma we were discussing if this will be possible to have a one off data dump for all closed projects.

• Active Project - ‘active’ - activity/ change in last 24h. Example: MapAction monitoring on a daily basis.

• Another question/use case for partners that came is whether we would be able to track the number of data downloads per project. @kshitijrajsharma let us know if that will be possible.

[robsavoye]

Downloading the data using the project AOI is easy. This module can be run standalone to download the data from Underpass. We use this in FMTM. Or you could use this module in TM someplace.
https://github.com/hotosm/osm-rawdata/blob/main/osm_rawdata/postgres.py

[petya-kangalova]

Sure, it is possible, but the request here is different. We are talking about the integration with the Tasking Manager, so people (without needing more advanced knowledge to use underpass) can download the data and question on tracking downloads.

[robsavoye]

That's the whole point with shared code modules. You don't need to know Underpass, just import the PostgresClient class into the TM Flask backend, and use it.

[RAytoun]

And the question then is ... What is Postgres Client or Flask backend? (without needing more advanced knowledge to use underpass)

…

On Wed, Nov 22, 2023 at 3:55 PM Rob Savoye ***@***.***> wrote: That's the whole point with shared code modules. You don't need to know Underpass, just import the PostgresClient class into the TM Flask backend, and use it. — Reply to this email directly, view it on GitHub <#6037 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACQDQBVRFD474QI2IJVJB5TYFYOABAVCNFSM6AAAAAA3ZXIFS6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMNBUGAZTK> . You are receiving this because you commented.Message ID: ***@***.***>

[robsavoye]

Underpass is our raw data base of the planet, updated every minute. PostgresClient is a python module that accesses the data. We use this for FMTM, since we're heavy users of data extracts when field mapping. My suggestion is to add a "Download Data" button to the page for a TM project, so nothing technical to learn.

[petya-kangalova]

Hi @robsavoye indeed, I think your feedback is to @kshitijrajsharma and the dev team on the implementation which is helpful. For @RAytoun and other users, what they need is exactly seeing this "Download OSM Data" per project.

[SColchester]

Couple extra thoughts since Collective session yesterday:

• Downloading by Campaign (i.e. a collection of projects) would also be valuable. For example - someone wanting all the roads for the Morocco Earthquake response will have a bad experience having to go to each of the five separate Morocco Earthquake projects to download. They will want the data across the whole disaster area in one go.
• Giving users the ability to download features other than those that are specified 'focus features' in the project or campaign area will be important. For example, an organization may create a project for mapping buildings only in a disaster response area because the roads are already very well mapped but the buildings are not. But this doesn't mean that the organization does not need road data, in fact the road data may be as important to them, so giving the user the ability to download multiple features will be valuable (maybe even a 'download all' option as well?)

[robsavoye]

FMTM is literally built around making and using data extracts, as we use ODK Collect to edit the OSM data in the field. I built a collection of python modules to do all the work, instead of implementing this in the FMTM backend only. This way multiple projects can use the same infrastructure code. For TM, I think you'd just download everything in the project AOI. If you want more control and filtering, that's what Export Tool is for. While we can't do this yet, as part of the E2E concept, it would be entirely possible to have TM "send" the project AOI to Export Tool, since we also plan to have TM "send" the project to FMTM. I'll talk about the E2E project during the Dec Meet Up.

[ramyaragupathy]

Please follow up on specific tasks:

• Let users download OSM data generated for each project #6143
• Export tool integration: Creating a new endpoint to generate a list of active projects #6144
• Export tool integration: Setting up a lambda function to track active projects and schedule export jobs #6145
• Export tool integration: infrastructure tasks #6146