Merge pull request #178 from ehinman/update-readme-samples

Remove qwdata documentation, tests, etc. and replace with waterdata.get_samples()

Author: ehinman

README.md

@@ -4,7 +4,8 @@
 ![Conda Version](https://img.shields.io/conda/v/conda-forge/dataretrieval)
 ![Downloads](https://static.pepy.tech/badge/dataretrieval)
 
-:warning: USGS data availability and format are changing on Water Quality Portal (WQP). Since March 2024, data obtained from WQP legacy profiles will not include new USGS data or recent updates to existing data. 
+:warning: USGS data availability and format have changed on Water Quality Portal (WQP). Since March 2024, data obtained from WQP legacy profiles will not include new USGS data or recent updates to existing data. All USGS data (up to and beyond March 2024) are available using the new WQP beta services. You can access the beta services by setting `legacy=False` in the functions in the `wqp` module.
+
 To view the status of changes in data availability and code functionality, visit: https://doi-usgs.github.io/dataRetrieval/articles/Status.html
 
 :mega: **09/03/2024:** The groundwater levels service has switched endpoints, and `dataretrieval` was updated accordingly in [`v1.0.10`](https://github.com/DOI-USGS/dataretrieval-python/releases/tag/v1.0.10). Older versions using the discontinued endpoint will return 503 errors for `nwis.get_gwlevels` or the `service='gwlevels'` argument. Visit [Water Data For the Nation](https://waterdata.usgs.gov/blog/wdfn-waterservices-2024/) for more information.
@@ -34,15 +35,11 @@ import dataretrieval.nwis as nwis
 # specify the USGS site code for which we want data.
 site = '03339000'
 
-
 # get instantaneous values (iv)
 df = nwis.get_record(sites=site, service='iv', start='2017-12-31', end='2018-01-01')
 
-# get water quality samples (qwdata)
-df2 = nwis.get_record(sites=site, service='qwdata', start='2017-12-31', end='2018-01-01')
-
 # get basic info about the site
-df3 = nwis.get_record(sites=site, service='site')
+df2 = nwis.get_record(sites=site, service='site')
 ```
 Services available from NWIS include:
 - instantaneous values (iv)
@@ -51,13 +48,16 @@ Services available from NWIS include:
 - site info (site)
 - discharge peaks (peaks)
 - discharge measurements (measurements)
-* water quality samples (qwdata)
 
-To access the full functionality available from NWIS web services, nwis.get record appends any additional kwargs into the REST request. For example
+Water quality data are available from:
+- [Samples](https://waterdata.usgs.gov/download-samples/#dataProfile=site) - Discrete USGS water quality data only
+- [Water Quality Portal](https://www.waterqualitydata.us/) - Discrete water quality data from USGS and EPA. Older data are available in the legacy WQX version 2 format; all data are available in the beta WQX3.0 format.
+
+To access the full functionality available from NWIS web services, nwis.get record appends any additional kwargs into the REST request. For example, this function call:
 ```python
 nwis.get_record(sites='03339000', service='dv', start='2017-12-31', parameterCd='00060')
 ```
-will download daily data with the parameter code 00060 (discharge).
+...will download daily data with the parameter code 00060 (discharge).
 
 ## Accessing the "Internal" NWIS
 If you're connected to the USGS network, dataretrieval call pull from the internal (non-public) NWIS interface.
@@ -97,9 +97,11 @@ Any help in testing, development, documentation and other tasks is welcome.
 For more details, see the file [CONTRIBUTING.md](CONTRIBUTING.md).
 
 
-## Package Support
-The Water Mission Area of the USGS supports the development and maintenance of `dataretrieval`
-and most likely further into the future.
+## Need help?
+
+The Water Mission Area of the USGS supports the development and maintenance of `dataretrieval`. Any questions can be directed to the Computational Tools team at 
+[email protected]. 
+
 Resources are available primarily for maintenance and responding to user questions.
 Priorities on the development of new features are determined by the `dataretrieval` development team.
 

dataretrieval/__init__.py

@@ -5,6 +5,7 @@
 from dataretrieval.samples import *
 from dataretrieval.streamstats import *
 from dataretrieval.utils import *
+from dataretrieval.waterdata import *
 from dataretrieval.waterwatch import *
 from dataretrieval.wqp import *
 

dataretrieval/nwis.py

@@ -36,7 +36,6 @@
 
 WATERSERVICES_SERVICES = ["dv", "iv", "site", "stat"]
 WATERDATA_SERVICES = [
-    "qwdata",
     "gwlevels",
     "measurements",
     "peaks",
@@ -135,125 +134,13 @@ def get_qwdata(
     **kwargs,
 ) -> Tuple[pd.DataFrame, BaseMetadata]:
     """
-    Get water sample data from qwdata service.
-
-    .. warning::
-
-        WARNING: Beginning in March 2024 the NWIS qw data endpoint will
-        not deliver new data or updates to existing data.
-        Eventually the endpoint will be retired. For updated information visit:
-        https://waterdata.usgs.gov.nwis/qwdata
-        For additional details, see the R package vignette:
-        https://doi-usgs.github.io/dataRetrieval/articles/Status.html
-        If you have additional questions about the qw data service,
-        email [email protected].
-
-    Parameters
-    ----------
-    sites: string or list of strings, optional, default is None
-        If the qwdata parameter site_no is supplied, it will overwrite the
-        sites parameter
-    start: string, optional, default is None
-        If the qwdata parameter begin_date is supplied, it will overwrite the
-        start parameter (YYYY-MM-DD)
-    end: string, optional, default is None
-        If the qwdata parameter end_date is supplied, it will overwrite the
-        end parameter (YYYY-MM-DD)
-    multi_index: bool, optional
-        If False, a dataframe with a single-level index (datetime) is returned,
-        default is True
-    wide_format : bool, optional
-        If True, return data in wide format with multiple samples per row and
-        one row per time, default is True
-    datetime_index : bool, optional
-        If True, create a datetime index, default is True
-    ssl_check: bool, optional
-        If True, check SSL certificates, if False, do not check SSL,
-        default is True
-    **kwargs: optional
-        If supplied, will be used as query parameters
-
-    Returns
-    -------
-    df: ``pandas.DataFrame``
-        Times series data from the NWIS JSON
-    md: :obj:`dataretrieval.utils.Metadata`
-        A custom metadata object
-
-    Examples
-    --------
-    .. doctest::
-
-        >>> # get water sample information for site 11447650
-        >>> df, md = dataretrieval.nwis.get_qwdata(
-        ...     sites="11447650", start="2010-01-01", end="2010-02-01"
-        ... )
+    Get water sample data from qwdata service - deprecated, use `get_samples()`
+    in the waterdata module.
 
     """
-    warnings.warn(
-        (
-            "WARNING: Starting in March 2024, the NWIS qw data endpoint is "
-            "retiring and no longer receives updates. For more information, "
-            "refer to https://waterdata.usgs.gov.nwis/qwdata and "
-            "https://doi-usgs.github.io/dataRetrieval/articles/Status.html "
-            "or email [email protected]."
+    raise NameError(
+        "`nwis.get_qwdata` has been replaced with `waterdata.get_samples()`."
         )
-    )
-
-    _check_sites_value_types(sites)
-
-    kwargs["site_no"] = kwargs.pop("site_no", sites)
-    kwargs["begin_date"] = kwargs.pop("begin_date", start)
-    kwargs["end_date"] = kwargs.pop("end_date", end)
-    kwargs["multi_index"] = multi_index
-    if wide_format:
-        kwargs["qw_sample_wide"] = "qw_sample_wide"
-
-    payload = {
-        "agency_cd": "USGS",
-        "format": "rdb",
-        "pm_cd_compare": "Greater than",
-        "inventory_output": "0",
-        "rdb_inventory_output": "file",
-        "TZoutput": "0",
-        "rdb_qw_attributes": "expanded",
-        "date_format": "YYYY-MM-DD",
-        "rdb_compression": "value",
-        "submitted_form": "brief_list",
-    }
-
-    # check for parameter codes, and reformat query args
-    qwdata_parameter_code_field = "parameterCd"
-    if kwargs.get(qwdata_parameter_code_field):
-        parameter_codes = kwargs.pop(qwdata_parameter_code_field)
-        parameter_codes = to_str(parameter_codes)
-        kwargs["multiple_parameter_cds"] = parameter_codes
-        kwargs["param_cd_operator"] = "OR"
-
-        search_criteria = kwargs.get("list_of_search_criteria")
-        if search_criteria:
-            kwargs["list_of_search_criteria"] = "{},{}".format(
-                search_criteria, "multiple_parameter_cds"
-            )
-        else:
-            kwargs["list_of_search_criteria"] = "multiple_parameter_cds"
-
-    kwargs.update(payload)
-
-    warnings.warn(
-        "NWIS qw web services are being retired. "
-        + "See this note from the R package for more: "
-        + "https://doi-usgs.github.io/dataRetrieval/articles/qwdata_changes.html",
-        category=DeprecationWarning,
-    )
-    response = query_waterdata("qwdata", ssl_check=ssl_check, **kwargs)
-
-    df = _read_rdb(response.text)
-
-    if datetime_index is True:
-        df = format_datetime(df, "sample_dt", "sample_tm", "sample_start_time_datum_cd")
-
-    return format_response(df, **kwargs), NWIS_Metadata(response, **kwargs)
 
 
 def get_discharge_measurements(
@@ -269,14 +156,10 @@ def get_discharge_measurements(
     Parameters
     ----------
     sites: string or list of strings, optional, default is None
-        If the qwdata parameter site_no is supplied, it will overwrite the
-        sites parameter
-    start: string, optional, default is None
-        If the qwdata parameter begin_date is supplied, it will overwrite the
-        start parameter (YYYY-MM-DD)
+    start: string, optional, default is None 
+        Supply date in the format: YYYY-MM-DD
     end: string, optional, default is None
-        If the qwdata parameter end_date is supplied, it will overwrite the
-        end parameter (YYYY-MM-DD)
+        Supply date in the format: YYYY-MM-DD
     ssl_check: bool, optional
         If True, check SSL certificates, if False, do not check SSL,
         default is True
@@ -1190,7 +1073,6 @@ def get_record(
     service: string, default is 'iv'
         - 'iv' : instantaneous data
         - 'dv' : daily mean data
-        - 'qwdata' : discrete samples
         - 'site' : site description
         - 'measurements' : discharge measurements
         - 'peaks': discharge peaks
@@ -1219,9 +1101,6 @@ def get_record(
         >>> # Get latest daily mean data from site 01585200
         >>> df = dataretrieval.nwis.get_record(sites="01585200", service="dv")
 
-        >>> # Get all discrete sample data from site 01585200
-        >>> df = dataretrieval.nwis.get_record(sites="01585200", service="qwdata")
-
         >>> # Get site description for site 01585200
         >>> df = dataretrieval.nwis.get_record(sites="01585200", service="site")
 
@@ -1287,18 +1166,6 @@ def get_record(
         )
         return df
 
-    elif service == "qwdata":
-        df, _ = get_qwdata(
-            site_no=sites,
-            begin_date=start,
-            end_date=end,
-            multi_index=multi_index,
-            wide_format=wide_format,
-            ssl_check=ssl_check,
-            **kwargs,
-        )
-        return df
-
     elif service == "site":
         df, _ = get_info(sites=sites, ssl_check=ssl_check, **kwargs)
         return df