add support for uploading cookie

Co-authored-by: Máté Gyöngyösi <[email protected]>

Author: gedankenstuecke

make-images.ipynb

@@ -17,22 +17,28 @@
     "2. You can \"run\" each of these cells/regions by either pressing the \"play\" button in the menu above, or by pressing CTRL+ENTER on your keyboard.\n",
     "3. If you are in the \"editing\" mode where you type into a cell, pressing ESC will take you back to the mode where the arrow keys work. \n",
     "\n",
-    "### Uploading the OSM history file\n",
+    "## Getting an OSM history file\n",
     "\n",
     "Before we can do the rest of the steps in this notebook, you will have to upload an OSM history file that covers your region of interest. You can find the corresponding files on [Geofabrik's internal download server](https://osm-internal.download.geofabrik.de/?landing_page=true). Login with your OSM account to get to the `internal` files, and then find the page for the your region of interest. Ideally, find the smallest region that covers your area of interest to minimize the filesize you need to upload. \n",
     "\n",
     "**The right file**: When you've picked your region, you will likely see two similarly named files listed: `regionname-internal.osh.pbf` and `regionname-latest-internal.osm.pbf`. The file you will want to download is the `regionname-internal.osh.pbf` (i.e. the one that includes `osh` and **no `latest`**).\n",
     "\n",
     "Once you have downloaded your file onto your computer, you need to upload it into your container. There are a few options that you can consider, depending on how you are using this notebook:\n",
     "\n",
-    "#### Uploading a file you already have downloaded onto your computer\n",
+    "### 1. Uploading a file you already have downloaded onto your computer\n",
     "\n",
     "* If you've launched this notebook in _MyBinder_, you can then open up the filebrowser on the top of the left hand side (The _file_ icon, or by pressing CTRL+SHIFT+F). \n",
     "* If you've launched this notebook locally, you're likely in the _notebook_ interface, switch back to the _tree_ view and upload your file there (or put it into the shared folder which is accessible by both the container and your normal operating system).\n",
     "\n",
-    "#### Downloading a file directly from Geofabrik \n",
+    "### 2. Downloading a file directly from Geofabrik \n",
     "\n",
-    "This option is particularly useful if you use this notebook on MyBinder, as it allows you to directly download the OSM history file into your Binder, without having to first download it onto your computer and then re-uploading it to the Binder, speeding things up considerably. As the OSM history files are only accessible with an OSM account, you will unfortunately have to fill out the form below, including your OSM username & password. \n",
+    "This option is particularly useful if you use this notebook on MyBinder, as it allows you to directly download the OSM history file into your Binder, without having to first download it onto your computer and then re-uploading it to the Binder, speeding things up considerably. As the OSM history files are only accessible with an OSM account, you will have to be authenticated to use this method. \n",
+    "\n",
+    "There are two ways of achieving this to access files directly from _Geofabrik_. \n",
+    "\n",
+    "#### 2.1 Option a): Using your OSM username & password.\n",
+    "\n",
+    "You can use your OSM username & password to authenticate, by running the cell below and then entering the form fields. \n",
     "\n",
     "**Note:** Your username & password will only be used to get a cookie that allows downloading the files and won't be permanently stored (and are not accessible to anyone else). If you are using _MyBinder_, your virtual machine will automatically shut down after more than 10 minutes of inactivity, destroying all data within the virtual machine in the process. If you are running this in a Docker container on your own computer the container will also destroy itself after shutting it down. \n",
     "\n",
@@ -41,26 +47,22 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 6,
    "id": "7d8cfd4c-913e-4983-92ef-c40567421331",
-   "metadata": {
-    "jupyter": {
-     "source_hidden": true
-    }
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "0036ae1165254d019e392a3ebd9b7905",
+       "model_id": "79eb82aff050415b9c87d9073a876a99",
        "version_major": 2,
        "version_minor": 0
       },
       "text/plain": [
        "VBox(children=(Text(value='', description='OSM username:', placeholder='Steve'), Text(value='', description='O…"
       ]
      },
-     "execution_count": 4,
+     "execution_count": 6,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -128,12 +130,124 @@
     "!python sendfile_osm_oauth_protector/oauth_cookie_client.py -u \"{osm_user.value}\" -p \"{osm_password.value}\" -c \"https://osm-internal.download.geofabrik.de/get_cookie\" -f netscape -o geofabrik_cookie.txt;wget --max-redirect 0 --load-cookies geofabrik_cookie.txt {file_url.value}"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "78358b23-cf0b-4890-83bc-70ead669b77d",
+   "metadata": {},
+   "source": [
+    "Once this is finished, you can move down to **Starting to make comparison images**!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "66cc684e-90e9-4ccb-a851-9315aaac5362",
+   "metadata": {},
+   "source": [
+    "#### 2.1 Option b): Using an existing Geofabrik cookie from your browser. \n",
+    "\n",
+    "This method allows you to paste the cookie that Geofabrik uses to check if you're allowed to download a file. As a result, **this method does not need to enter your OSM username/password**. The downside is that it's slightly more complicated, as it requires finding the cookie value in your web browser. \n",
+    "\n",
+    "To use this option, do the following: \n",
+    "\n",
+    "1. Head to the [OSM internal Geofabrik download page](https://osm-internal.download.geofabrik.de/)\n",
+    "2. Log-in on that page with your existing OSM credentials, this will create the necessary cookie. \n",
+    "3. Find the cookie in your web-browser: If you're using Firefox, press `F12` on your keyboard to open the web developer tools\n",
+    "4. Go to the tab called `Storage` in the bar that opens at the bottom of your browser\n",
+    "5. On the bottom-left, you will see `Cookies`, click on it and select the cookie that starts with `https://osm-internal.download.geofabrik.de`.\n",
+    "6. You will see a Name/Value pair displayed, with the name reading `gf_download_oauth`. Select this cookie\n",
+    "7. On the right-hand side you will see a `Data` option, which gives the full cookie contents. You can right-click on this and select `Copy`.\n",
+    "\n",
+    "You can then paste this cookie data into the form below, alongside the link to the file you want to download:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "id": "e5f626cc-f267-4f19-a290-17144f1ffde9",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "1f012554f3fb458b84fa56c27973df37",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "VBox(children=(Text(value='', description='gf_download_oauth cookie value:', placeholder='login|2018-04-12|WVR…"
+      ]
+     },
+     "execution_count": 31,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "geofabrik_cookie = ('gf_download_oauth', widgets.Text(\n",
+    "    placeholder='login|2018-04-12|WVRced34c...',\n",
+    "    description='gf_download_oauth cookie value:',\n",
+    "    disabled=False))\n",
+    "file_url = widgets.Text(\n",
+    "    placeholder='https://osm-internal.download.geofabrik.de/south-america/argentina-internal.osh.pbf',\n",
+    "    description='Link to OSM history file:',\n",
+    "    disabled=False)\n",
+    "\n",
+    "widgets.VBox([\n",
+    "    geofabrik_cookie[1],\n",
+    "    file_url])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "55e67302-5653-4f4f-b2c5-3963379e4ecc",
+   "metadata": {},
+   "source": [
+    "Once you have added those, run the cell below to download your file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "id": "4eb7ed00-1902-4d5e-8ba3-f149ad85c61d",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "login|2018-04-12|eRRYaHdVQwFceNZlGOUawLbKH8oR-OjHNwBmVcnkpOaJJa0-H9X6zfFgJLWrfwbGXzdyh7bNUPH1WjrvkkpWAY4kI6rSSg_A_HP2CSpAS4XEr91hLamrnPsrgbYETIUWkvKRrFzVthHkHJDALGVmd7E8fgNKe_uL9xeJaKAWU_Ydi_uu4a86xKhgkfZv1NBI_g27phWwTtvt-TE57v9Vygc_jgozXpnmL3ENny3g\n",
+      "--2024-12-26 14:53:24--  https://osm-internal.download.geofabrik.de/europe/malta-internal.osh.pbf\n",
+      "Resolving osm-internal.download.geofabrik.de (osm-internal.download.geofabrik.de)... 65.109.22.172, 2a01:4f9:5a:1426::2\n",
+      "connected. to osm-internal.download.geofabrik.de (osm-internal.download.geofabrik.de)|65.109.22.172|:443... \n",
+      "HTTP request sent, awaiting response... 200 OK\n",
+      "Length: 18084230 (17M) [application/octet-stream]\n",
+      "Saving to: ‘malta-internal.osh.pbf’\n",
+      "\n",
+      "malta-internal.osh. 100%[===================>]  17.25M  4.12MB/s    in 4.2s    \n",
+      "\n",
+      "2024-12-26 14:53:29 (4.12 MB/s) - ‘malta-internal.osh.pbf’ saved [18084230/18084230]\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "cookie_name = geofabrik_cookie[0]\n",
+    "if geofabrik_cookie[1].value.startswith(\"gf_download_oauth\"):\n",
+    "    cookie_value = geofabrik_cookie[1].value.replace('gf_download_oauth:\"','')\n",
+    "    cookie_value = cookie_value[:-1]\n",
+    "    print(cookie_value)\n",
+    "else:\n",
+    "    cookie_value = geofabrik_cookie[1].value\n",
+    "    \n",
+    "!wget --max-redirect 0 --header=\"Cookie: {cookie_name}={cookie_value}\" {file_url.value}"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "bfe0b5b8-cebd-4975-8447-f2189459dbcb",
    "metadata": {},
    "source": [
-    "Once the file is completely downloaded (or if you uploaded the file using one of the other methods), you can continue with the next step below. \n",
+    "Once the file is completely downloaded (or if you uploaded the file using one of the other methods above), you can continue with the next step below. \n",
     "\n",
     "## Starting to make the comparison images. \n",
     "\n",
@@ -144,26 +258,22 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 13,
    "id": "cf6cc023-ea39-46eb-885e-86a184dc1850",
-   "metadata": {
-    "jupyter": {
-     "source_hidden": true
-    }
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "ef5e2a5f18f54653bf1e7d39e90559ac",
+       "model_id": "2bd9d49b689145f093b8fe9445d0ac7c",
        "version_major": 2,
        "version_minor": 0
       },
       "text/plain": [
-       "VBox(children=(Dropdown(description='File:', options=('external/argentina-internal.osh.pbf', 'external/argenti…"
+       "VBox(children=(Dropdown(description='File:', options=('uruguay-internal.osh.pbf', 'external/argentina-internal…"
       ]
      },
-     "execution_count": 1,
+     "execution_count": 13,
      "metadata": {},
      "output_type": "execute_result"
     }