Working implementation of journal meta data using openalex. Passes te… by BharatVe · Pull Request #168 · GeoinformationSystems/optimap

BharatVe · 2025-06-04T16:00:00Z

Closes #5

Data Model Changes
Added a new Journal model with fields name, issn_l, openalex_id, openalex_url, publisher_name, works_count, works_api_url and converted Publication.source from a text field to a ForeignKey pointing at this Journal model .

Serializers and ViewSets
Created a JournalSerializer exposing all eight journal fields, then updated PublicationSerializer (a GeoFeatureModelSerializer) to include a nested field source_details = JournalSerializer(source="source"). The PublicationViewSet filters out any publications lacking a valid source or null geometry so that GeoJSON serialization (via DRF‐GIS) never fails .

OpenAlex Sync Command
Wrote a management command update_openalex_journals that, for each Journal with a non-null ISSN-L, fetches metadata from OpenAlex (using requests.get("https://api.openalex.org/sources/issn:")). It populates or updates each journal’s openalex_id, openalex_url, publisher_name, works_count, and works_api_url, saving only when changes occur.

Test Coverage for Journals + Publications

Confirm /api/v1/journals/ lists all journals with the correct eight fields.
Confirm /api/v1/journals/{pk}/ returns exactly the expected fields.
Confirm that /api/v1/publications/ responses include a nested source_details block whose keys match the Journal model.
Confirm that filtering by ?journal_id= returns only publications whose source_details["id"] == pk .

TODO:

Review the code.
(maybe) Add the implementation for the Issue Add admin action to manually trigger the data generation task #148.

…anagement for journal entries. Todo: fix geometries, add tests.

…sts, needs review.

BharatVe · 2025-06-04T17:04:03Z

@nuest I think the changes I made to add the journal functionality might be causing issues with harvesting in tasks.py. Would it be okay to change some of the previous code a little to fix the conflicts and get everything running smoothly?

nuest · 2025-06-04T18:33:18Z

Yes, of course. Please fix the tests. I can nevertheless make a first review even with the current changes. Tomorrow.

nuest

Thanks! This is a major update, so let's try to get things as right as possible:

Let's change the name from "journals" to "sources", in line with the entity names in OpenAlex, see https://docs.openalex.org/api-entities/entities-overview

We certainly also want to geolocate preprint articles, and then "journal = preprint server" makes so sense.

"publications" also is not a good name? Right, we'll get to that in #169

nuest · 2025-06-05T12:42:56Z

fixtures/test_data.json

+    "model": "publications.journal",
+    "pk": 1,
+    "fields": {
+    "name": "Nature",                    


I understand where you're coming from, but this is actually not what our values are.

If we want to use real journals here because their ISSN actually exists, then we go with diamond open access journals (https://en.wikipedia.org/wiki/Diamond_open_access).

https://github.com/loreabad6/doaj-geo is a good starting point, so let's use some that we might also want to collaborate with. Please add the following journals to the test data:

https://www.jvolcanica.org/ojs/index.php/volcanica

https://josis.org/index.php/josis/index

https://journals.open.tudelft.nl/ejtir

https://agile-giss.copernicus.org

Changed in the test data, @BharatVe double checks.

fixtures/test_data.json

publications/management/commands/update_openalex_journals.py

publications/serializers.py

publications/static/js/main.js

tests/test_journal_publication.py

…tional fields for sources. Testing ongoing.

BharatVe · 2025-07-01T08:00:46Z

These are the errors I am having the most difficulty resolving

TypeError: cannot unpack non-iterable NoneType object
======================================================================
FAIL: test_parse_missing (tests.test_htmlparser.SimpleTest.test_parse_missing)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/optimap/optimap/tests/test_htmlparser.py", line 58, in test_parse_missing
    self.assertEqual(period_start, [None])
AssertionError: Lists differ: [] != [None]
Second list contains 1 additional elements.
First extra element 0:
None
- []
+ [None]

FAIL: test_api_publication_2 (tests.test_harvesting.SimpleTest.test_api_publication_2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/optimap/optimap/tests/test_harvesting.py", line 109, in test_api_publication_2
    self.assertEqual(resp.status_code, 200)
AssertionError: 404 != 200
======================================================================
FAIL: test_no_duplicates_after_initial_harvest (tests.test_harvesting.SimpleTest.test_no_duplicates_after_initial_harvest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/optimap/optimap/tests/test_harvesting.py", line 162, in test_no_duplicates_after_initial_harvest
    self.assertEqual(Publication.objects.count(), 2)
AssertionError: 0 != 2


FAIL: test_api_redirect (tests.test_publications_api.PublicationsApiTest.test_api_redirect)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/optimap/optimap/tests/test_publications_api.py", line 48, in test_api_redirect
    self.assertEqual(response.status_code, 302)
AssertionError: 404 != 302
======================================================================
FAIL: test_publication_includes_source_details (tests.test_source_publication.PublicationAPITest.test_publication_includes_source_details)
GET /api/v1/publications/ should return ≥1 publication,
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/optimap/optimap/tests/test_source_publication.py", line 185, in test_publication_includes_source_details
    self.assertEqual(len(pubs_list), 1)
AssertionError: 2 != 1

nuest · 2025-07-01T10:45:01Z

publications/tasks.py

+
+                if src and getattr(src, "is_preprint", False) and geom.empty:
+                    try:
+                        loc = Nominatim(user_agent="optimap-tasks").geocode(


Where does the geocoding come from? Can you please clarify which issue this implements?

nuest · 2025-07-01T10:45:18Z

publications/tasks.py

+                        )
+
+                ps_list, pe_list = extract_timeperiod_from_html(soup)
+            except Exception:


Never silently catch exceptions.

nuest · 2025-07-01T12:27:45Z

This my my local test output:

Found 49 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
....................ss.....................F...ss
======================================================================
FAIL: test_publication_includes_source_details (tests.test_source_publication.PublicationAPITest)
GET /api/v1/publications/ should return ≥1 publication,
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/daniel/git/KOMET/optimap/tests/test_source_publication.py", line 203, in test_publication_includes_source_details
    self.assertIn(key, details)
AssertionError: 'openalex_id' not found in {'id': 42, 'name': 'API source', 'issn_l': '1111-2222', 'openalex_url': 'https://openalex.org/S011112222'}

----------------------------------------------------------------------
Ran 49 tests in 20.408s

FAILED (failures=1, skipped=4)
Destroying test database for alias 'default'...

nuest · 2025-07-08T08:13:32Z

Failed remaining test obsoleted by 2fa0215

BharatVe added 5 commits May 12, 2025 20:23

Updates for adding ISSN information and adding OpenAlex functionality.

10df5e1

Updated Sync files and added information

1715735

Update journal entries, corrected data retrival, added api and data m…

66272a8

…anagement for journal entries. Todo: fix geometries, add tests.

Changes for updated ISSN

4b64c3b

Working implementation of journal meta data using openalex. Passes te…

1e7a1ee

…sts, needs review.

nuest requested changes Jun 5, 2025

View reviewed changes

BharatVe and others added 6 commits June 17, 2025 09:24

Updated test_data and tests, changed journal to source and added addi…

783ab4e

…tional fields for sources. Testing ongoing.

Merge branch 'feature/add-journal-data-5' into add-journal-data-5

a88e631

Merge branch 'main' into feature/add-journal-data-5

1396b8f

Fix merge issues and change remaining journal to source changes

f16a39d

Updated tets and files, some errors still need to be worked on.

2f7ab63

Minor updates, testing ongoing

f4aa463

BharatVe marked this pull request as ready for review June 30, 2025 20:57

BharatVe added 2 commits July 1, 2025 09:06

Update requirements-dev.txt

885a61f

Update urls.py

abd419a

BharatVe closed this Jul 1, 2025

BharatVe reopened this Jul 1, 2025

nuest added 2 commits July 1, 2025 11:02

re-add geopy dependency, wrongly removed in other PR

d890cb2

re-add historic test data

94b93ca

nuest reviewed Jul 1, 2025

View reviewed changes

nuest added 3 commits July 1, 2025 13:58

fix tests around harvesting

7490e8f

Merge branch 'main' into feature/add-journal-data-5

cff2171

fix CI

f2849cc

Fix tests

240db24

nuest merged commit 0e1809c into main Jul 8, 2025
1 of 2 checks passed

nuest mentioned this pull request Jul 8, 2025

Enhance works/publications metadata from OpenAlex #171

Closed

Conversation

BharatVe commented Jun 4, 2025

Uh oh!

BharatVe commented Jun 4, 2025

Uh oh!

nuest commented Jun 4, 2025

Uh oh!

nuest left a comment

Choose a reason for hiding this comment

Uh oh!

nuest Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

nuest Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BharatVe commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nuest Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

nuest Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

nuest commented Jul 1, 2025

Uh oh!

nuest commented Jul 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

BharatVe commented Jul 1, 2025 •

edited

Loading