Merge pull request #11489 from QualitativeDataRepository/counter_script_updates

MDC: script updates for counter-processor-1.06+

Author: ofahimIQSS

doc/release-notes/11489-CPUpdates.md

@@ -0,0 +1,5 @@
+Not sure what is already in the notes w.r.t. counter-processor updates (separate repo) so I'll try to be ~complete here
+
+### Make Data Count improvements
+
+Counter-processor, used to power Make Data Count metrics in Dataverse is now maintained in the https://github.com/gdcc/counter-processor repository. Multiple improvements to efficiency and scalability have been made. The example counter_daily.sh and counter_weekly.sh scripts that automate using counter-processor, available from the MDC section of the Dataverse Guides (https://guides.dataverse.org/en/latest/admin/make-data-count.html) have been updated to work with the latest counter-processor release and also have minor improvements.

doc/sphinx-guides/source/_static/util/counter_daily.sh

@@ -1,15 +1,17 @@
 #! /bin/bash
+#counter_daily.sh
 
 COUNTER_PROCESSOR_DIRECTORY="/usr/local/counter-processor-1.06"
 MDC_LOG_DIRECTORY="/usr/local/payara6/glassfish/domains/domain1/logs/mdc"
-
-# counter_daily.sh
+COUNTER_PROCESSOR_TMP_DIRECTORY="/tmp"
+# If you wish to keep the logs, use a directory that is not periodically cleaned, e.g.
+#COUNTER_PROCESSOR_TMP_DIRECTORY="/usr/local/counter-processor-1.06/tmp"
 
 cd $COUNTER_PROCESSOR_DIRECTORY
 
-echo >>/tmp/counter_daily.log
-date >>/tmp/counter_daily.log
-echo >>/tmp/counter_daily.log
+echo >>$COUNTER_PROCESSOR_TMP_DIRECTORY/counter_daily.log
+date >>$COUNTER_PROCESSOR_TMP_DIRECTORY/counter_daily.log
+echo >>$COUNTER_PROCESSOR_TMP_DIRECTORY/counter_daily.log
 
 # "You should run Counter Processor once a day to create reports in SUSHI (JSON) format that are saved to disk for Dataverse to process and that are sent to the DataCite hub."
 
@@ -22,15 +24,33 @@ d=$(date -I -d "$YEAR_MONTH-01")
 while [ "$(date -d "$d" +%Y%m%d)" -le "$(date -d "$LAST" +%Y%m%d)" ];
 do
   if [ -f "$MDC_LOG_DIRECTORY/counter_$d.log" ]; then
-#       echo "Found counter_$d.log"
+      echo "Found counter_$d.log"
   else
-        touch "$MDC_LOG_DIRECTORY/counter_$d.log"
+      touch "$MDC_LOG_DIRECTORY/counter_$d.log"
   fi
   d=$(date -I -d "$d + 1 day")
 done
 
 #run counter-processor as counter user
 
-sudo -u counter YEAR_MONTH=$YEAR_MONTH python3 main.py >>/tmp/counter_daily.log
-
-curl -X POST "http://localhost:8080/api/admin/makeDataCount/addUsageMetricsFromSushiReport?reportOnDisk=/tmp/make-data-count-report.json"
+sudo -u counter YEAR_MONTH=$YEAR_MONTH python3 main.py >>$COUNTER_PROCESSOR_TMP_DIRECTORY/counter_daily.log
+
+# Process all make-data-count-report.json.* files
+for report_file in $COUNTER_PROCESSOR_TMP_DIRECTORY/make-data-count-report.json.*; do
+    if [ -f "$report_file" ]; then
+        echo "Processing $report_file" >>$COUNTER_PROCESSOR_TMP_DIRECTORY/counter_daily.log
+        curl -X POST "http://localhost:8080/api/admin/makeDataCount/addUsageMetricsFromSushiReport?reportOnDisk=$report_file"
+        echo "Finished processing $report_file" >>$COUNTER_PROCESSOR_TMP_DIRECTORY/counter_daily.log
+        
+        # Extract the base filename and the extension
+        file_base=$(basename "$report_file" | sed 's/\.json\..*//')
+        file_ext=$(echo "$report_file" | sed -n 's/.*\.json\.\(.*\)/\1/p')
+        echo $file_base
+        echo $file_ext
+        # Remove the old file if it exists
+        rm -f $COUNTER_PROCESSOR_TMP_DIRECTORY/${file_base}.${YEAR_MONTH}.json.${file_ext}
+
+        # Move the processed file
+        mv $report_file $COUNTER_PROCESSOR_TMP_DIRECTORY/${file_base}.${YEAR_MONTH}.json.${file_ext}
+    fi
+done

doc/sphinx-guides/source/_static/util/counter_weekly.sh

@@ -6,6 +6,7 @@
 
 # A recursive method to process each Dataverse
 processDV () {
+echo "Running counter_weekly.sh on $(date)"
 echo "Processing Dataverse ID#: $1"
 
 #Call the Dataverse API to get the contents of the Dataverse (without credentials, this will only list published datasets and dataverses
@@ -44,5 +45,5 @@ done
 
 }
 
-# Call the function on the root dataverse to start processing 
+# Call the function on the root dataverse to start processing
 processDV 1

doc/sphinx-guides/source/admin/make-data-count.rst

@@ -130,7 +130,7 @@ Populate Views and Downloads Nightly
 
 Running ``main.py`` to create the SUSHI JSON file and the subsequent calling of the Dataverse Software API to process it should be added as a cron job.
 
-The Dataverse Software provides example scripts that run the steps to process new accesses and uploads and update your Dataverse installation's database :download:`counter_daily.sh <../_static/util/counter_daily.sh>` and to retrieve citations for all Datasets from DataCite :download:`counter_weekly.sh <../_static/util/counter_weekly.sh>`. These scripts should be configured for your environment and can be run manually or as cron jobs.
+The Dataverse Software provides an example script that run the steps to process new accesses and uploads and update your Dataverse installation's database :download:`counter_daily.sh <../_static/util/counter_daily.sh>` The script should be configured for your environment and can be run manually or as a cron job.
 
 Sending Usage Metrics to the DataCite Hub
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -160,6 +160,7 @@ To confirm that the environment variable was set properly, you can use echo like
 ``echo $DOI``
 
 On some periodic basis (perhaps weekly) you should call the following curl command for each published dataset to update the list of citations that have been made for that dataset.
+The example :download:`counter_weekly.sh <../_static/util/counter_weekly.sh>` will do this for you. The script should be configured for your environment and can be run manually or as a cron job.
 
 ``curl -X POST "http://localhost:8080/api/admin/makeDataCount/:persistentId/updateCitationsForDataset?persistentId=$DOI"``