Strategy for filtering out component-level metadata for complex objects, in OAI-PMH feed (UCSC)

[aturner]

Did a random check of number of objects returned, for a handful of UCSC collections, and seeing anomalies. Noted below.

1. UCSC Campus History - Photographs and Maps
   http://cdm16019.contentdm.oclc.org/oai/oai.php?verb=ListRecords&metadataPrefix=oai_dc&set=p265101coll25
   https://registry.cdlib.org/admin/library_collection/collection/127/

7,948 results in CONTENTdm
http://digitalcollections.ucsc.edu/cdm/search/collection/p265101coll25

7,467 results harvested
https://54.84.142.143/couchdb/ucldc/_design/all_provider_docs/_view/by_provider_name_count?group=true&key=%22127%22

2. Santa Cruz County History Digital Collections
   http://cdm16019.contentdm.oclc.org/oai/oai.php?verb=ListRecords&metadataPrefix=oai_dc&set=p265101coll23
   https://registry.cdlib.org/admin/library_collection/collection/134/

247 results in CONTENTdm
http://digitalcollections.ucsc.edu/cdm/search/collection/p265101coll23

335 results harvested
https://54.84.142.143/couchdb/ucldc/_design/all_provider_docs/_view/by_provider_name_count?group=true&key=%22134%22

3. UCSC Campus History Publications
   http://cdm16019.contentdm.oclc.org/oai/oai.php?verb=ListRecords&metadataPrefix=oai_dc&set=p16019coll3
   https://registry.cdlib.org/admin/library_collection/collection/26101/

863 results in CONTENTdm
http://digitalcollections.ucsc.edu/cdm/search/collection/p16019coll3

8,675 records harvested
https://54.84.142.143/couchdb/ucldc/_design/all_provider_docs/_view/by_provider_name_count?group=true&key=%2226101%22

4. Organic Farming-Sustainable Agriculture Oral Histories
   http://cdm16019.contentdm.oclc.org/oai/oai.php?verb=ListRecords&metadataPrefix=oai_dc&set=p15130coll2
   https://registry.cdlib.org/admin/library_collection/collection/130/

57 results in CONTENTdm
http://digitalcollections.ucsc.edu/cdm/search/collection/p15130coll2

263 records harvested
https://54.84.142.143/couchdb/ucldc/_design/all_provider_docs/_view/by_provider_name_count?group=true&key=%22130%22

[aturner]

Oof, I think I see the reason for the issue with Collection ID 25101 (UCSC Campus History Publications) -- they are complex objects in CONTENTdm, and the OAI-PMH feed is providing records for every component. So the total number of records is way high:

http://cdm16019.contentdm.oclc.org/oai/oai.php?verb=ListRecords&metadataPrefix=oai_dc&set=p16019coll3

[mredar]

And this seems to be a problem for a number of the collections. We'll probably have to check with them to see how the records can be filtered to retrieve only the top level items.