[ST]: Replace HTML scraper with ArcGIS API scraper (#219)

* Replace Sachsen-Anhalt HTML scraper with ArcGIS API scraper

Switch from HTML scraping (bildung-lsa.de) to ArcGIS FeatureServer API.
This provides cleaner data access and adds geolocation support.

Changes:
- Replace HTML parsing with ArcGIS REST API JSON parsing
- Add coordinate transformation (EPSG:25832 -> WGS84) using pyproj
- Add geolocation coverage: 100% (857 schools)
- Update ID scheme: ST-1001186 -> ST-ARC00001 (OBJECTID-based)
- Update README: Mark ST as having geolocation via ArcGIS
- Note: OBJECTID stability is uncertain (may change on reimport)

Data source: services-eu1.arcgis.com ArcGIS FeatureServer
Coverage: 857 schools (excludes vocational schools)

* Remove personal dev files from .gitignore (now in global gitignore)

* Use None instead of empty strings for missing data

Replace empty string defaults with None when extracting school attributes.
This provides clearer semantics for missing data and follows database
best practices (NULL vs empty string).

Note: Current ArcGIS dataset has no missing values, but this change
future-proofs the code and follows Python/SQL conventions.

* Remove exception handling from coordinate transformation

Let coordinate transformation fail loudly if it encounters issues
rather than silently logging a warning and continuing with null coordinates.

This follows the 'fail fast' principle - if transformation fails, we want
to know immediately so we can fix the root cause rather than silently
producing incomplete data.

---------

Co-authored-by: tim <[email protected]>

Author: tifa365

README.md

@@ -35,7 +35,7 @@ In details, the IDs are sourced as follows:
 |RP| `Schulnummer` from the school's details page                                                                 | `RP-50720`                                                                 |✅ likely|
 |SL| `OBJECTID` from the WFS service                                                                              | `SL-255`                                                                   |❌ no (confirmed with data provider but no alternative available) |
 |SN| Field `id` from the API                                                                                      | `SN-4062`                                                                  |✅ likely|
-|ST| `ID` query param from the details page URL                                                                   | `ST-1001186`                                                               |❓ probably?|
+|ST| `OBJECTID` from the ArcGIS FeatureServer API (prefixed with `ARC`)                                           | `ST-ARC00001`                                                              |❓ unlikely (OBJECTID may change on data reimport)|
 |TH| `Schulnummer` from the WFS service                                                                           | `TH-10601`                                                                 |✅ likely|
 
 ## Geolocations
@@ -56,7 +56,7 @@ When available, we try to use the geolocations provided by the data publishers.
 | RP    | ❌ No                 | -                                            |
 | SL    | ✅ Yes                | WFS                                          |
 | SN    | ✅ Yes                | API                                          |
-| ST    | ❌ No                 | -                                            |
+| ST    | ✅ Yes                | ArcGIS (converted from EPSG:25832)           |
 | TH    | ✅ Yes                | WFS                                          |
 
 ## Additional Data Fields

jedeschule/spiders/sachsen_anhalt.py

@@ -1,64 +1,65 @@
 # -*- coding: utf-8 -*-
-import re
 import scrapy
 from scrapy import Item
+from pyproj import Transformer
 from jedeschule.items import School
 from jedeschule.spiders.school_spider import SchoolSpider
 
 
 class SachsenAnhaltSpider(SchoolSpider):
     name = "sachsen-anhalt"
+
+    # ArcGIS FeatureServer API - contains 857 schools with coordinates
+    # Note: This dataset excludes vocational schools (Berufsbildende Schulen)
+    # but includes all primary, secondary, grammar, and special needs schools
     start_urls = [
-        "https://www.bildung-lsa.de/ajax.php?m=getSSResult&q=&lk=-1&sf=-1&so=-1&timestamp=1480082277128/"
+        "https://services-eu1.arcgis.com/3jNCHSftk0N4t7dd/arcgis/rest/services/"
+        "Schulenstandorte_EPSG25832_2024_25_Sicht/FeatureServer/44/query?"
+        "where=1%3D1&outFields=*&f=json"
     ]
 
-    detail_url = "https://www.bildung-lsa.de/ajax.php?m=getSSDetails&id={}&timestamp=1480082332787"
-
     def parse(self, response):
-        js_callbacks = response.css("span ::attr(onclick)").extract()
-        pattern = "getSSResultItemDetail\((\d+)\)"
-        ids = [re.match(pattern, text).group(1) for text in js_callbacks]
-        names = response.css("b::text").extract()
-        for id, name in zip(ids, names):
-            request = scrapy.Request(
-                self.detail_url.format(id), callback=self.parse_detail
-            )
-            request.meta["id"] = id
-            request.meta["name"] = name.strip()
-            yield request
+        """Parse ArcGIS FeatureServer JSON response"""
+        data = response.json()
 
-    def parse_detail(self, response):
-        tables = response.css("table")
+        # EPSG:25832 (UTM zone 32N) to EPSG:4326 (WGS84) transformer
+        transformer = Transformer.from_crs("EPSG:25832", "EPSG:4326", always_xy=True)
 
-        content = {}
-        # Only the second and third table contain interesting data
-        for table in tables[1:3]:
-            trs = table.css("tr")
-            for tr in trs:
-                tds = tr.css("td")
-                key = tds[0].css("::text").extract_first()[:-2]
-                value = " ".join(tds[1].css("::text").extract())
+        for feature in data.get("features", []):
+            attrs = feature["attributes"]
+            geom = feature.get("geometry", {})
 
-                content[key] = value
-        content["Name"] = response.meta["name"]
-        content["ID"] = response.meta["id"]
-        # The name is included in the "Adresse" field so we remove that
-        # in order to get only the address
-        content["Adresse"] = (
-            content["Adresse"].replace(response.meta["name"], "").strip()
-        )
-        yield content
+            # Transform coordinates from EPSG:25832 to WGS84
+            latitude = None
+            longitude = None
+            if geom and "x" in geom and "y" in geom:
+                longitude, latitude = transformer.transform(geom["x"], geom["y"])
+
+            # Extract school information from ArcGIS attributes
+            yield {
+                "name": attrs.get("Name"),
+                "city": attrs.get("Ort"),
+                "school_type": attrs.get("Schulform"),
+                "category": attrs.get("Kategorie"),
+                "provider": attrs.get("Traeg_Anw"),
+                "latitude": latitude,
+                "longitude": longitude,
+                "object_id": attrs.get("OBJECTID"),
+            }
 
     @staticmethod
     def normalize(item: Item) -> School:
+        """Normalize ArcGIS data to School item"""
+        # Generate ID from OBJECTID
+        school_id = f"ST-ARC{item.get('object_id', 0):05d}"
+
         return School(
-            name=item.get("Name"),
-            id="ST-{}".format(item.get("ID")),
-            address=re.split(r"\d{5}", item.get("Adresse").strip())[0].strip(),
-            zip=re.findall(r"\d{5}", item.get("Adresse").strip())[0],
-            city=re.split(r"\d{5}", item.get("Adresse").strip())[1].strip(),
-            website=item.get("Homepage"),
-            email=item.get("E-Mail"),
-            fax=item.get("Telefax"),
-            phone=item.get("Telefon"),
+            name=item.get("name"),
+            id=school_id,
+            city=item.get("city"),
+            school_type=item.get("school_type"),
+            legal_status=item.get("category"),
+            provider=item.get("provider"),
+            latitude=item.get("latitude"),
+            longitude=item.get("longitude"),
         )