Skip to content

Commit 6c8e5b0

Browse files
rzmkrzmk
authored
Merge branch 'ckan:master' into rzmk/gztr
2 parents f36745c + 1e109aa commit 6c8e5b0

File tree

15 files changed

+127
-13
lines changed

15 files changed

+127
-13
lines changed

‎.github/workflows/test.yml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,16 +28,13 @@ jobs:
2828
ckan-image: "ckan/ckan-dev:2.10-py3.10"
2929
solr-image: "2.10-solr9-spatial"
3030
harvester-version: 'master'
31-
- ckan-version: 2.9
32-
ckan-image: "amercader/ckan-dev:2.9-py3.9"
33-
solr-image: 2.9-solr9-spatial
34-
harvester-version: 'master'
3531
fail-fast: false
3632

3733
name: CKAN ${{ matrix.ckan-version }}, Solr ${{ matrix.solr-image }}
3834
runs-on: ubuntu-latest
3935
container:
4036
image: ${{ matrix.ckan-image }}
37+
options: --user root
4138
services:
4239
solr:
4340
image: ckan/ckan-solr:${{ matrix.solr-image }}
@@ -49,7 +46,7 @@ jobs:
4946
POSTGRES_DB: postgres
5047
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
5148
redis:
52-
image: redis:3
49+
image: redis:7
5350
env:
5451
CKAN_SQLALCHEMY_URL: postgresql://ckan_default:pass@postgres/ckan_test
5552
CKAN_DATASTORE_WRITE_URL: postgresql://datastore_write:pass@postgres/datastore_test
@@ -63,7 +60,8 @@ jobs:
6360

6461
- name: Install dependencies (common)
6562
run: |
66-
DEBIAN_FRONTEND=noninteractive apt-get --assume-yes --quiet install \
63+
DEBIAN_FRONTEND=noninteractive apt-get update && \
64+
apt-get --assume-yes --quiet install \
6765
python3-dev \
6866
libxml2-dev \
6967
libxslt1-dev \
@@ -72,7 +70,7 @@ jobs:
7270
- name: Install dependencies from requirements.txt
7371
run: |
7472
pip install -r requirements.txt
75-
pip install pytest-ckan
73+
pip install -r dev-requirements.txt
7674
7775
- name: Install harvester
7876
run: |
@@ -95,6 +93,9 @@ jobs:
9593
run: |
9694
# Replace default path to CKAN core config file with the one on the container
9795
sed -i -e 's/use = config:.*/use = config:\/srv\/app\/src\/ckan\/test-core.ini/' test.ini
98-
96+
- name: Set up db
97+
run: |
98+
ckan -c test.ini db init
99+
ckan -c test.ini db pending-migrations --apply
99100
- name: Run tests
100101
run: pytest --ckan-ini=test.ini --cov=ckanext.spatial --cov-report term-missing --cov-append --disable-warnings ckanext/spatial/tests

‎CHANGELOG.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,11 @@
11
# Changelog
22

3-
## [Unreleased](https://github.com/ckan/ckanext-dcat/compare/v2.2.0...HEAD)
3+
## [Unreleased](https://github.com/ckan/ckanext-dcat/compare/v2.3.0...HEAD)
44

5+
## [v2.3.0](https://github.com/ckan/ckanext-spatial/compare/v2.2.0...v2.3.0) - 2025-03-06
6+
7+
* Bump Shapely requirement [#343](https://github.com/ckan/ckanext-spatial/pull/343)
8+
* Fix WAF harvester [#342](https://github.com/ckan/ckanext-spatial/pull/342)
59

610
## [v2.2.0](https://github.com/ckan/ckanext-spatial/compare/v2.1.1...v2.2.0) - 2024-11-08
711

@@ -22,7 +26,7 @@
2226

2327
* Dropped support for Python 2
2428
* Dropped support for the PostGIS search backend
25-
* Updated the common map JS module to support many different tile providers. The default Stamen Terrain tile will no longer work, and users will need to configure a map tiles provider. Please check the [documentation](https://docs.ckan.org/projects/ckanext-spatial/en/latest/map-widgets.html) for full details.
29+
* Updated the common map JS module to support many different tile providers. The default Stamen Terrain tile will no longer work, and users will need to configure a map tiles provider. Please check the [documentation](https://docs.ckan.org/projects/ckanext-spatial/en/latest/map-widgets/) for full details.
2630
* Upgrade tests to check all envs, including CKAN 2.10 with Python 3.10 [#308](https://github.com/ckan/ckanext-spatial/pull/308)
2731
* TypeError when spatial is missing [#306](https://github.com/ckan/ckanext-spatial/pull/306)
2832
* Fix requirements [#313](https://github.com/ckan/ckanext-spatial/pull/313)

‎ckanext/spatial/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,4 @@
66
import pkgutil
77
__path__ = pkgutil.extend_path(__path__, __name__)
88

9-
__version__ = "2.2.0"
9+
__version__ = "2.3.0"

‎ckanext/spatial/harvesters/waf.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -312,6 +312,8 @@ def _extract_waf(content, base_url, scraper, results = None, depth=0):
312312
if 'mailto:' in url:
313313
continue
314314
if '..' not in url and url[-1] == '/':
315+
if scraper == 'apache' and url[0] == '/':
316+
continue
315317
new_depth = depth + 1
316318
if depth > 10:
317319
log.info('Max WAF depth reached')

‎ckanext/spatial/public/js/common_map.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@
8585
onAdd: (map) => {
8686
let element = document.createElement("div");
8787
element.className = "leaflet-control-no-provider";
88-
element.innerHTML = 'No map provider set. Please check the <a href="https://docs.ckan.org/projects/ckanext-spatial/en/latest/map-widgets.html">documentation</a>';
88+
element.innerHTML = 'No map provider set. Please check the <a href="https://docs.ckan.org/projects/ckanext-spatial/en/latest/map-widgets/">documentation</a>';
8989
return element;
9090
},
9191
onRemove: (map) => {}

‎ckanext/spatial/tests/waf_extract/html_files/apache-folder.html‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
2+
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
3+
<html>
4+
<head>
5+
<title>Index of /apache-folder</title>
6+
</head>
7+
<body>
8+
<h1>Index of /apache-folder</h1>
9+
<pre> <a href="?C=N;O=D">Name</a> <a href="?C=M;O=A">Last modified</a> <a href="?C=S;O=A">Size</a> <a href="?C=D;O=A">Description</a><hr> <a href="/">Parent Directory</a> -
10+
<a href="record-1.xml">record-1.xml</a> 2024-11-07 15:00 356K
11+
<a href="subfolder/">subfolder/</a> 2024-11-12 15:00 -
12+
<hr></pre>
13+
</body></html>

‎ckanext/spatial/tests/waf_extract/html_files/apache-subfolder.html‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
2+
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
3+
<html>
4+
<head>
5+
<title>Index of /apache-folder/subfolder</title>
6+
</head>
7+
<body>
8+
<h1>Index of /apache-folder/subfolder</h1>
9+
<pre> <a href="?C=N;O=D">Name</a> <a href="?C=M;O=A">Last modified</a> <a href="?C=S;O=A">Size</a> <a href="?C=D;O=A">Description</a><hr> <a href="/folder/">Parent Directory</a> -
10+
<a href="record-2.xml">record-2.xml</a> 2024-11-07 16:59 182K
11+
<hr></pre>
12+
</body></html>

‎ckanext/spatial/tests/waf_extract/html_files/iis-folder.html‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
<html><head><title>iis.server - /iis-folder/</title></head><body><H1>iis.server - /iis-folder/</H1><hr>
2+
3+
<pre><A HREF="/">[To Parent Directory]</A><br><br> 11/7/2024 7:20 AM &lt;dir&gt; <A HREF="/iis-folder/subfolder/">subfolder</A><br> 11/7/2024 3:00 PM 168 <A HREF="/iis-folder/record-1.xml">record-1.xml</A><br></pre><hr></body></html>
4+
5+

‎ckanext/spatial/tests/waf_extract/html_files/iis-subfolder.html‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
<html><head><title>iis.server - /iis-folder/subfolder/</title></head><body><H1>iis.server - /iis-folder/subfolder/</H1><hr>
2+
3+
<pre><A HREF="/iis-folder/">[To Parent Directory]</A><br><br> 11/7/2024 4:59 PM 8958 <A HREF="/iis-folder/subfolder/record-2.xml">record-2.xml</A><br></pre><hr></body></html>

‎ckanext/spatial/tests/waf_extract/html_files/nginx-folder.html‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
2+
<html>
3+
<head><title>Index of /nginx/</title></head>
4+
<body bgcolor="white">
5+
<h1>Index of /nginx/</h1><hr><pre><a href="../">../</a>
6+
<a href="subfolder/">subfolder/</a> 07-Nov-2024 15:00 -
7+
<a href="record-1.xml">record-1.xml</a> 07-Nov-2024 15:00 364868
8+
</pre><hr></body>
9+
</html>

0 commit comments

Comments
 (0)