You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/source/configuration.md
+27-8Lines changed: 27 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,15 +13,24 @@ In following chapters configuration values and keys are discussed for each part
13
13
All configuration values for MapSwipe Workers are stored in environment variables.
14
14
15
15
Required environment variables are:
16
+
### Firebase
16
17
- FIREBASE_API_KEY
17
18
- FIREBASE_DB
19
+
- FIREBASE_TOKEN
20
+
- GOOGLE_APPLICATION_CREDENTIALS
21
+
22
+
### Postgres DB
18
23
- POSTGRES_DB
19
24
- POSTGRES_HOST
20
25
- POSTGRES_PASSWORD
21
26
- POSTGRES_PORT
22
27
- POSTGRES_USER
23
28
24
-
Optional environment variables are:
29
+
### OSMCha
30
+
31
+
- OSMCHA_API_KEY
32
+
33
+
### Optional environment variables:
25
34
- SLACK_CHANNEL
26
35
- SLACK_TOKEN
27
36
- SENTRY_DSN
@@ -34,19 +43,16 @@ For satellite imagery access to at least one provider is needed. Define the API
34
43
- IMAGE_ESRI_API_KEY
35
44
- IMAGE_ESRI_BETA_API_KEY
36
45
37
-
In addition to get access to Firebase a Service Account Key is required.
38
-
The path the Service Account Key is defined in:
39
-
- GOOGLE_APPLICATION_CREDENTIALS
40
-
41
46
> Notes: When deploying using `docker` or `docker-compose``POSTGRES_HOST` should have the value `postgres` and the Service Account Key (`serviceAccountKey.json`) should be copied to `mapswipe_workers/serviceAccountKey.json` so that during the build of the image the file can by copied by Docker.
42
47
43
-
44
48
### Elaboration
45
49
46
50
**Firebase**: MapSwipe Workers use the Firebase Python SDK and the Firebase REST API. Both require the database name (`FIREBASE_DB`) and the API-Key from the Firebase instance. The Firebase Python SDK does also need a Service Account Key. The path to this file is set in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.
47
51
48
52
**Postgres**: MapSwipe Workers writes data to a Postgres database and generate files for the API based data in Postgres.
49
53
54
+
**OSMCha**: MapSwipe Workers enriches some Projects with data from OSM changelogs which are requested from OSMCha. Create an account, you will find you api key in your profile e.g. `Token 589adf125234a`
55
+
50
56
**Sentry (optional)**: MapSwipe workers use sentry to capture exceptions. You can find your project’s DSN in the “Client Keys” section of your “Project Settings” in Sentry. Check [Sentry's documentation](https://docs.sentry.io/error-reporting/configuration/?platform=python) for more information.
51
57
52
58
**Slack (optional)**: The MapSwipe workers send messages to slack when a project has been created successfully, the project creation failed or an exception gets raised. refer to [Python slackclient's documentation](https://github.com/slackapi/python-slackclient) how to get a Slack Token.
@@ -80,13 +86,26 @@ The Service Account Key (`serviceAccountKey.json`) should be saved to `postgres/
80
86
81
87
## Manager Dashboard
82
88
89
+
Please refer to the official [documentation](https://firebase.google.com/docs/web/learn-more#config-object) if you set up your own firebase.
90
+
Otherwise you can request guidance on the settings from the mapswipe team. The structure of your app.js should look like below.
Copy file name to clipboardExpand all lines: docs/source/data.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -31,7 +31,7 @@ Attributes:
31
31
32
32
33
33
## Aggregated Results
34
-
This gives you the unfiltered MapSwipe results. This is most suited if you want to apply some custom data processing with the MapSwipe data, e.g. select only specific tasks for machine learning. If you want to use MapSwipe data in the Tasking Manager you might look for the data explained below.
34
+
This gives you the unfiltered MapSwipe results. This is most suited if you want to apply some custom data processing with the MapSwipe data, e.g. select only specific tasks for machine learning. If you want to use MapSwipe data in the Tasking Manager you might look for the data described below.
35
35
36
36
Files:
37
37
-`aggregated_results_{project_id}.csv`, e.g. [agg\_results\_-M56eeMCZ5VeOHjJN4Bx.csv](https://apps.mapswipe.org/api/agg_results/agg_results_-M56eeMCZ5VeOHjJN4Bx.csv)
@@ -51,11 +51,11 @@ Files:
51
51
| 2_share | float | 2_count divived by total_count. This gives you the share of all users who marked as 2. |
52
52
| 3_share | float | 3_count divived by total_count. This gives you the share of all users who marked as 3. |
53
53
| agreement | float | This is defined as [Scott's Pi](https://en.wikipedia.org/wiki/Scott%27s_Pi) and gives you an understanding of inter-rater reliability. The value is 1.0 if all users agree, e.g. all users classify as "building". If users disagree this value will be lower. |
54
-
| geom | string | The geometry of this task as WKT geometry. |
54
+
| geom | string | The geometry of this task as WKT geometry. |
55
55
56
56
57
-
## HOT Tasking Manager Geometries
58
-
This gives you filtered MapSwipe data ready to be imported to the HOT Tasking Manager. Currently, the geometries in this dataset consist of maximum 15 MapSwipe Tasks, where at least 35% of all users indicated the presence of a building by classifying as "yes" or "maybe".
57
+
## HOT Tasking Manager Geometries
58
+
This gives you filtered MapSwipe data ready to be imported to the HOT Tasking Manager. Currently, the geometries in this dataset consist of maximum 15 MapSwipe Tasks, where at least 35% of all users indicated the presence of a building by classifying as "yes" or "maybe".
59
59
60
60
Files:
61
61
-`hot_tm_{project_id}.geojson`, e.g. [hot\_tm\_-M56eeMCZ5VeOHjJN4Bx.geojson](https://apps.mapswipe.org/api/hot_tm/hot_tm_-M56eeMCZ5VeOHjJN4Bx.geojson)
Copy file name to clipboardExpand all lines: docs/source/dev_setup.md
+24-40Lines changed: 24 additions & 40 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,9 @@
1
1
# Development Setup
2
2
3
-
In this document some tips and workflows for development are loosely collected. Those are independent of the production setup using Docker-Compose. A working Firebase Project (Including Firebase Functions and Database Rules) is presupposed. Get in touch with the MapSwipe team (e.g. in Slack) to get access to an existing Firebase Instance for development purposes.
3
+
In this document some tips and workflows for development are loosely collected.
4
+
Those are independent of the production setup using Docker-Compose.
5
+
A working Firebase Project (Including Firebase Functions and Database Rules) is presupposed.
6
+
Get in touch with the MapSwipe team (e.g. in Slack) to get access to an existing Firebase Instance for development purposes.
4
7
5
8
Check list:
6
9
1. Clone repo from GitHub.
@@ -15,7 +18,8 @@ Check list:
15
18
16
19
### Requirements
17
20
18
-
MapSwipe Workers requires GDAL/OGR (`gdal-bin`) and GDAL for Python (`libgdal-dev`, `python-gdal`) to be installed. Furthermore, we rely on Docker to set up Postgres.
21
+
MapSwipe Workers requires GDAL/OGR (`gdal-bin`) and GDAL for Python (`libgdal-dev`, `python-gdal`) to be installed.
22
+
Furthermore, we rely on Docker to set up Postgres.
The MapSwipe Workers requires a Service Account Key (`serviceAccountKey.json`) to access Firebase database. Request yours from the MapSwipe working group.
55
+
The MapSwipe Workers requires a Service Account Key (`serviceAccountKey.json`) to access Firebase database.
56
+
Request yours from the MapSwipe working group.
52
57
53
-
The path the Service Account Key is defined in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.
58
+
The path to the Service Account Key is defined in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.
54
59
55
-
You could also set up your own Firebase instance. However, this is not recommended. If you still want to do it, get your Service Account Key from Firebase from [Google Cloud Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts).
60
+
You could also set up your own Firebase instance. However, this is not recommended.
61
+
If you still want to do it, get your Service Account Key from Firebase from [Google Cloud Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts).
56
62
57
63
58
64
### Postgres
59
65
60
-
Setup a local Postgres instance for MapSwipe Workers using the Dockerfile provided for development purposes (`postgres/Dockerfile-dev`). The Dockerfile for production (`postgres/Dockerfile`) does need additional setup for build-in backup to Google Cloud Storage, which is not needed for local development. That is why a simplified Dockerfile for development is provided.
66
+
Setup a local Postgres instance for MapSwipe Workers using the Dockerfile provided for development purposes (`postgres/Dockerfile-dev`).
67
+
The Dockerfile for production (`postgres/Dockerfile`) does need additional setup for build-in backup to Google Cloud Storage, which is not needed for local development. That is why a simplified Dockerfile for development is provided.
61
68
Make sure that the specified port is not in use already. If so, adjust the port (also in the `.env` file).
62
69
63
70
```bash
@@ -90,7 +97,8 @@ mapswipe_workers --help
90
97
91
98
## Logging
92
99
93
-
Mapswipe workers logs are generated using the Python logging module of the standard library (See [Official docs](https://docs.python.org/3/library/logging.html) or this [Tutorial](https://realpython.com/python-logging/#the-logging-module). To use the logger object import the it from the `definitions` module:
100
+
Mapswipe workers logs are generated using the Python logging module of the standard library (See [Official docs](https://docs.python.org/3/library/logging.html) or this [Tutorial](https://realpython.com/python-logging/#the-logging-module).
101
+
To use the logger object import the it from the `definitions` module:
94
102
95
103
```python
96
104
from mapswipe_workers.definitions import logger
@@ -104,9 +112,11 @@ except Exception:
104
112
logger.exception('Additional information.')
105
113
```
106
114
107
-
Default logging level is Info. To change the logging level edit the logging configuration which is found in the module `definitions.py`. Logs are written to STDOUT and `~/.local/share/mapswipe_workers/mapswipe_workers.log`.
115
+
Default logging level is Info. To change the logging level edit the logging configuration which is found in the module `definitions.py`.
116
+
Logs are written to STDOUT and `~/.local/share/mapswipe_workers/mapswipe_workers.log`.
108
117
109
-
Per default logging of third-party packages is disabled. To change this edit the definition module (`mapswipe_workers/defintions.py`). Set the `disable_existing_loggers` parameter of the `logging.config.fileConfig()` function to `False`.
118
+
Per default logging of third-party packages is disabled. To change this edit the definition module (`mapswipe_workers/defintions.py`).
119
+
Set the `disable_existing_loggers` parameter of the `logging.config.fileConfig()` function to `False`.
110
120
111
121
112
122
## Firebase Functions
@@ -128,38 +138,12 @@ Firebase functions are used to increment/decrement or calculate various attribut
128
138
129
139
Those functions will be directly or indirectly triggered by incoming results from the MapSwipe App.
130
140
131
-
By using Firebase functions those attributes can be calculated in real-time and be accessed by users immediately. The use of those functions also reduces the data-transfer between the Firebase Realtime Database and MapSwipe Workers.
132
-
133
-
On how to setup development environment and how to deploy functions to the Firebase instance please refer to the official [Guide on Cloud Function for Firebase](https://firebase.google.com/docs/functions/get-started).
134
-
For more information refer to the official [Reference on Cloud Function for Firebase](https://firebase.google.com/docs/reference/functions/). For example function take a look at this [GitHub repository](https://github.com/firebase/functions-samples).
135
-
136
-
137
-
## Travis Setup
138
-
139
-
A Travis instance is used to build MapSwipe Workers and run tests.
140
-
There exists a Firebase instance only for Travis.
141
-
For the configuration of Travis following environment variables are used:
Those variables can be definied directly in the repository settings of Travis. For more inofmration refer to: https://docs.travis-ci.com/user/environment-variables/#defining-variables-in-repository-settings
155
-
156
-
Additionaly a Service Account Key in JSON format is encrypted and added to the GitHub repository using the travis CLI. Once Travis runs it will decrypt the Service Account Key. Read more on that process in the Travis docs: https://docs.travis-ci.com/user/encrypting-files/
157
-
158
-
Once a Travis build succeeds Travis executes an Ansible Playbook to deploy MapSwipe Workers to an already installed and configured server.
159
-
For this to work an SSH-Key (with access rights to the server) is also encrypted and added to the GitHub repository. Travis will decrypt the key and Ansible will use it to execute commands defined in the Playbook on the server.
160
-
161
-
All files encrypted for Travis (Service Account Key, SSH-Key) are stored in the `travis` directory.
141
+
By using Firebase functions those attributes can be calculated in real-time and be accessed by users immediately.
142
+
The use of those functions also reduces the data-transfer between the Firebase Realtime Database and MapSwipe Workers.
162
143
144
+
On how to setup the development environment and how to deploy functions to the Firebase instance please refer to the official [Guide on Cloud Function for Firebase](https://firebase.google.com/docs/functions/get-started).
145
+
For more information refer to the official [Reference on Cloud Function for Firebase](https://firebase.google.com/docs/reference/functions/).
146
+
For example function take a look at this [GitHub repository](https://github.com/firebase/functions-samples).
Copy file name to clipboardExpand all lines: docs/source/for_mapswipe_managers.md
+4-1Lines changed: 4 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,7 +22,10 @@ For BuildArea and ChangeDetection projects:
22
22
For Footprint projects:
23
23
* The GeoJSON file should contain only simple Polygons. We currently don't support complex Multipolygon geometries (e.g. [polygon with holes](https://developers.google.com/maps/documentation/javascript/examples/polygon-hole))
24
24
25
-
Once you submit, the task should appear relatively quickly in the manager dashboard. You will receive a message in Slack. But it's still not active and not visible to the MapSwipe app users. You need to set the project status to `active` through the manager dashboard. Just click on the respective button. If the new project does not appear in the app after about 1 hour, check Slack for an error message, and see Troubleshooting below.
25
+
Once you submit, the task should appear relatively quickly in the manager dashboard. You will receive a message in Slack.
26
+
But it's still not active and not visible to the MapSwipe app users.
27
+
You need to set the project status to `active` through the manager dashboard. Just click on the respective button.
28
+
If the new project does not appear in the app after about 1 hour, check Slack for an error message, and see Troubleshooting below.
0 commit comments