Merge pull request #5 from EarthScope/dq-45-update-dependencies

DQ 45 Update Dependency Versions

Author: laura-iris

CHANGELOG.txt

@@ -1,3 +1,8 @@
+Version 1.2.0
+- Updated to use python 3.12
+- Updated dependency versions, requiring the retirement of deprecated methods used internally. 
+- Converted remaining references of "IRIS" to "EarthScope". Use of "IRIS" in the preference file will continue to work, but suggest updating to "EarthScope"
+
 Version 1.1.1
 - Update to use python 3.6.15
 

IRIS_metadata.txt EarthScope_metadata.txtIRIS_metadata.txt renamed to EarthScope_metadata.txt

@@ -1,10 +1,10 @@
 # QuARG
 
-#### Questions or comments can be directed to the EarthScope (formerly IRIS) Quality Assurance Group at <a href="mailto:data-help@earthscope.org">data-help@earthscope.org</a>.
+#### Questions or comments can be directed to the EarthScope (formerly IRIS) Quality Assurance Group at <a href="mailto:help@earthscope.org">help@earthscope.org</a>.
 For detailed documentation, check out [EarthScope.github.io/quarg/](https://EarthScope.github.io/quarg/DOCUMENTATION.html)  
 <br />
 
-**QuARG**, the Quality Assurance Report Generator, is a Python client that allows network operators to generate quality assurance (QA) reports from start to finish. These reports utilize EarthScope’s database of [MUSTANG](http://service.iris.edu/mustang/) data quality metrics to find and highlight potential issues in the data, reducing the amount of time that analysts need to spend scanning the data for problems. 
+**QuARG**, the Quality Assurance Report Generator, is a Python client that allows network operators to generate quality assurance (QA) reports from start to finish. These reports utilize EarthScope’s database of [MUSTANG](https://service.earthscope.org/mustang/) data quality metrics to find and highlight potential issues in the data, reducing the amount of time that analysts need to spend scanning the data for problems. 
 
 Over the years that IRIS produced Quality Assurance Reports, we refined the process of generating a report into four primary steps:
 
@@ -21,7 +21,7 @@ Users have the ability to customize QuARG to adapt to their particular network.
 * Add, edit, or remove Thresholds based on what best fits the network instrumentation.
 * Group instrumentation by Network, Station, Locations, or Channels, defining thresholds individually for each group. 
 * Use metric values sourced from either EarthScope or a local ISPAQ database. Similarly, locally-sourced or EarthScope-provided metadata.
-* Create preference files to minimize the number of fields that users need to input, easily track what thresholds were used to find issues, and potentially create a series of files to be utilized for different use-cases within a network.  An example file preference_file_IRIS.py is provided in the base quarg directory. 
+* Create preference files to minimize the number of fields that users need to input, easily track what thresholds were used to find issues, and potentially create a series of files to be utilized for different use-cases within a network.  An example file preference_file_EarthScope.py is provided in the base quarg directory. 
 * Use the built-in ticketing system or an external one, whichever works better for your workflow. 
 
 This utility guides users through the process of generating a list of potential issues, examining and tracking issues, and generating a report.  
@@ -97,16 +97,18 @@ Instructions for Linux or macOS (Intel chip)
 ```
 cd quarg
 conda update conda
-conda create --name quarg -c conda-forge --file quarg-conda-install.txt
+conda create --name quarg -c conda-forge  python=3.12
 conda activate quarg
+conda install -c conda-forge --file quarg-conda-install.txt
 ```
 
-Instructions for macOS (Apple M1 or M2 chip):
+Instructions for macOS (Apple M-series chips, e.g., M1 - M5):
 ```
 cd quarg
 conda update conda
-CONDA_SUBDIR=osx-64 conda create --name quarg -c conda-forge --file quarg-conda-install.txt
+CONDA_SUBDIR=osx-64 conda create --name quarg -c conda-forge  python=3.12
 conda activate quarg
+CONDA_SUBDIR=osx-64 conda install -c conda-forge --file quarg-conda-install.txt
 ```
 
 See what is installed in our (quarg) environment with:

QuARG.py

@@ -27,42 +27,78 @@
 import pandas as pd
 
 
-
 # TODO: If ts_ metrics are used, must propagate through into the thresholds file
 
 # ============================#
 # LOAD INPUT ARGUMENTS
-network = ''; station = ''; location = ''; channels =  ''; start = '';end = ''; outfile = ''
+network = ""
+station = ""
+location = ""
+channels = ""
+start = ""
+end = ""
+outfile = ""
 
 args = reportUtils.getArgs()
-start= args.start
+start = args.start
 end = args.end
 # month = args.month
 
 preferenceFile = args.preference_file
 
 if not preferenceFile:
     # If no preference file included, run everything
-    thresholdGroups = ['Completeness','Amplitudes','Timing','State of Health','Metadata']
-    groupsDict = {'Completeness':['avgGaps','gapsRatioGt12','noData'],
-                  'Amplitudes' : ['flat','lowRms','hiAmp','lowAmp','badResp',
-                                  'avgSpikes','pegged','dead','noise1','noise2',
-                                  'medianUnique','rmsRatio','xTalk',
-                                  'gainRatio','nonCoher','polarity',
-                                  'dcOffsets','nSpikes','rmsRatio'],
-                  'Timing' : ['poorTQual','suspectTime','noTime'],
-                  'State of Health' : ['ampSat','filtChg','clip',
-                                       'spikes','glitch','padding','tSync'],
-                  'Metadata' : ['zDip','horDip','zeroZ','lowScale','nonMSUnits']}
-    
+    thresholdGroups = [
+        "Completeness",
+        "Amplitudes",
+        "Timing",
+        "State of Health",
+        "Metadata",
+    ]
+    groupsDict = {
+        "Completeness": ["avgGaps", "gapsRatioGt12", "noData"],
+        "Amplitudes": [
+            "flat",
+            "lowRms",
+            "hiAmp",
+            "lowAmp",
+            "badResp",
+            "avgSpikes",
+            "pegged",
+            "dead",
+            "noise1",
+            "noise2",
+            "medianUnique",
+            "rmsRatio",
+            "xTalk",
+            "gainRatio",
+            "nonCoher",
+            "polarity",
+            "dcOffsets",
+            "nSpikes",
+            "rmsRatio",
+        ],
+        "Timing": ["poorTQual", "suspectTime", "noTime"],
+        "State of Health": [
+            "ampSat",
+            "filtChg",
+            "clip",
+            "spikes",
+            "glitch",
+            "padding",
+            "tSync",
+        ],
+        "Metadata": ["zDip", "horDip", "zeroZ", "lowScale", "nonMSUnits"],
+    }
+
 else:
     try:
         with open(preferenceFile) as f:
             exec(compile(f.read(), preferenceFile, "exec"))
     except OSError:
-        print('Cannot open', preferenceFile)
+        print("Cannot open", preferenceFile)
         quit()
-        
+
 # Commandline arguments override preference file values, if provided
 if args.network:
     network = args.network
@@ -71,7 +107,7 @@
 if args.locations:
     location = args.locations
 if args.channels:
-    channels=  args.channels
+    channels = args.channels
 if args.outfile:
     outfile = args.outfile
 if args.metricsource:
@@ -88,6 +124,11 @@
     print("WARNING: No threshold file provided. Exiting.")
     quit()
 
+# Allow for backward compatibility in preference file
+if metricSource == "IRIS":
+    metricSource = "EarthScope"
+if metadataSource == "IRIS":
+    metadataSource = "EarthScope"
 
 directory = os.path.dirname(outfile)
 
@@ -102,53 +143,64 @@
 
 if os.path.isfile(outfile):
     resp1 = input("This file already exists - overwrite?[y/n]:  ")
-    if (resp1.upper() == 'Y') or (resp1.upper() == 'YES'):
-        print('Removing existing file')
+    if (resp1.upper() == "Y") or (resp1.upper() == "YES"):
+        print("Removing existing file")
         os.remove(outfile)
-        
-    elif (resp1.upper() == 'N') or (resp1.upper()== 'NO'):
-        resp2= input('Should I append to the existing file?[y/n]: ')
-        if (not resp2.upper() == 'Y') and (not resp2.upper() == 'YES'):
+
+    elif (resp1.upper() == "N") or (resp1.upper() == "NO"):
+        resp2 = input("Should I append to the existing file?[y/n]: ")
+        if (not resp2.upper() == "Y") and (not resp2.upper() == "YES"):
             quit("Exiting")
     else:
-        print('Input not recognized, cancelling')
+        print("Input not recognized, cancelling")
         quit()
-    
+
 # Load up list of metrics and metadata, for reference later on
 if os.path.isfile(metrics_file):
-    with open(metrics_file,'r') as f:
+    with open(metrics_file, "r") as f:
         metricsList = f.read().splitlines()
 else:
     # This should not happen unless running outside of QuARG since QuARG.py has a check before running findIssues.py
-    print("WARNING: Could not find list of MUSTANG metrics in file %s - does it exist?" % metrics_file)
-    print("         You can create this list by entering the Thresholds Editor - it will automatically generate there")
+    print(
+        "WARNING: Could not find list of MUSTANG metrics in file %s - does it exist?"
+        % metrics_file
+    )
+    print(
+        "         You can create this list by entering the Thresholds Editor - it will automatically generate there"
+    )
     quit()
-  
+
 if os.path.isfile(metadata_file):
-    with open(metadata_file,'r') as f:
+    with open(metadata_file, "r") as f:
         metadataList = f.read().splitlines()
 else:
     # This should not happen unless running outside of QuARG since QuARG.py has a check before running findIssues.py
-    print("WARNING: Could not find list of IRIS metadata fields in file %s - does it exist?" % metadata_file)
-    print("         You can create this list by entering the Thresholds Editor - it will automatically generate there")
+    print(
+        "WARNING: Could not find list of EarthScope metadata fields in file %s - does it exist?"
+        % metadata_file
+    )
+    print(
+        "         You can create this list by entering the Thresholds Editor - it will automatically generate there"
+    )
     quit()
-             
+
 
 # ============================#
 # GO THROUGH THRESHOLDS
 
 # Add the header to the file
-with open(outfile, 'w') as f:
+with open(outfile, "w") as f:
     f.write("# Threshold|Target|Start|End|Ndays|Status|Value|Notes\n")
-f.close() 
+f.close()
 
 # Get metadata dataframe at the beginning to use wherever necessary, since it is always the same
-metadataDF = reportUtils.getMetadata(network, station, location, channels, start, end, metadataSource)
+metadataDF = reportUtils.getMetadata(
+    network, station, location, channels, start, end, metadataSource
+)
 
 
 failedMetricsAll = list()
 failedThresholdsAll = list()
-# thresholdFile = './groupsTEST.txt'
 for thresholdGroup in thresholdGroups:
     print()
     print("Running %s Thresholds" % thresholdGroup)
@@ -157,46 +209,65 @@
     except:
         print("   Could not find any thresholds for %s" % thresholdGroup)
         continue
-    
+
     thresholdsList.sort()
-    
-    allMetrics, failedThresholds = thresholds.get_threshold_metrics(thresholdsList, thresholdFile)
+
+    allMetrics, failedThresholds = thresholds.get_threshold_metrics(
+        thresholdsList, thresholdFile
+    )
     metadatas = [e for e in metadataList if e in allMetrics]
     metrics = [e for e in metricsList if e in allMetrics]
 
-#     hasMetadata = False; 
     hasMetrics = False
-#     if len(metadatas) > 0:
-#         print("This thresholds Group contains some metadata fields")
-#         hasMetadata = True
     if len(metrics) > 0:
         hasMetrics = True
 
-    
     if hasMetrics:
-        metricDF, failedMetrics = reportUtils.mergeMetricDF(network, station, location, channels, start, end, metrics, metricSource)
+        metricDF, failedMetrics = reportUtils.mergeMetricDF(
+            network, station, location, channels, start, end, metrics, metricSource
+        )
     else:
-        metricDF = pd.DataFrame(columns=['value','target','start','end','network','station','location','channel'])
+        metricDF = pd.DataFrame(
+            columns=[
+                "value",
+                "target",
+                "start",
+                "end",
+                "network",
+                "station",
+                "location",
+                "channel",
+            ]
+        )
         failedMetrics = list()
-    
+
     for failedThreshold in failedThresholds:
         if not failedThreshold in failedThresholdsAll:
             failedThresholdsAll.append(failedThreshold)
-    
+
     for failedMetric in failedMetrics:
         if not failedMetric in failedMetricsAll:
             failedMetricsAll.append(failedMetric)
-    
-#     if hasMetrics == True and  not metricDF.empty:
+
     for threshold in thresholdsList:
         if not threshold in failedThresholds:
-            thresholds.do_threshold(threshold, thresholdFile, metricDF, metadataDF, outfile, instruments, start, end, hasMetrics, chanTypes)
+            thresholds.do_threshold(
+                threshold,
+                thresholdFile,
+                metricDF,
+                metadataDF,
+                outfile,
+                instruments,
+                start,
+                end,
+                hasMetrics,
+                chanTypes,
+            )
 
-with open('failedMetrics.txt','w') as f:
+with open("failedMetrics.txt", "w") as f:
     for failedThreshold in failedThresholdsAll:
-        f.write('threshold: %s\n' % failedThreshold)
+        f.write("threshold: %s\n" % failedThreshold)
     for failedMetric in failedMetricsAll:
-        f.write('metric: %s\n' % failedMetric)
+        f.write("metric: %s\n" % failedMetric)
 
 print("INFO: Completed generating issue file")
-