Merge pull request #11850 from GlobalDataverseCommunityConsortium/TDL-BigDataDocs

TDL: Provide guidance for site admins w.r.t. big data

Author: pdurbin

doc/release-notes/11850-big-data-admin-guide.md

@@ -0,0 +1,3 @@
+### Big Data Admin Section
+
+- A new section - Scaling Dataverse with Data Size - has been added to the Admin Guide. It is intended to help administrators configure Dataverse appropriately to handle larger amounts of data.

doc/sphinx-guides/source/admin/big-data-administration.rst

@@ -35,3 +35,4 @@ This guide documents the functionality only available to superusers (such as "da
    maintenance
    backups
    troubleshooting
+   big-data-administration

doc/sphinx-guides/source/admin/index.rst

@@ -196,6 +196,6 @@ As described in that document, Globus transfers can be initiated by choosing the
 
 An overview of the control and data transfer interactions between components was presented at the 2022 Dataverse Community Meeting and can be viewed in the `Integrations and Tools Session Video <https://youtu.be/3ek7F_Dxcjk?t=5289>`_ around the 1 hr 28 min mark.
 
-See also :ref:`Globus settings <:GlobusSettings>`.
+See also :ref:`Globus settings <:GlobusSettings>` and :ref:`globus-stores`.
 
 An alternative, experimental implementation of Globus polling of ongoing upload transfers has been added in v6.4. This framework does not rely on the instance staying up continuously for the duration of the transfer and saves the state information about Globus upload requests in the database. Due to its experimental nature it is not enabled by default. See the ``globus-use-experimental-async-framework`` feature flag (see :ref:`feature-flags`) and the JVM option :ref:`dataverse.files.globus-monitoring-server`.

doc/sphinx-guides/source/developers/big-data-support.rst

@@ -1036,15 +1036,18 @@ File Storage
 
 By default, a Dataverse installation stores all data files (files uploaded by end users) on the filesystem at ``/usr/local/payara6/glassfish/domains/domain1/files``. This path can vary based on answers you gave to the installer (see the :ref:`dataverse-installer` section of the Installation Guide) or afterward by reconfiguring the ``dataverse.files.\<id\>.directory`` JVM option described below.
 
-A Dataverse installation can alternately store files in a Swift or S3-compatible object store, or on a Globus endpoint, and can now be configured to support multiple stores at once. With a multi-store configuration, the location for new files can be controlled on a per-Dataverse collection basis.
-
+A Dataverse installation can alternately store files in a Swift or S3-compatible object store, or on a Globus endpoint, and can now be configured to support multiple stores at once.
 A Dataverse installation may also be configured to reference some files (e.g. large and/or sensitive data) stored in a web or Globus accessible trusted remote store.
+With a multi-store configuration, the location for new files can be controlled on a per-Dataverse collection or per-dataset basis.
+:doc:`/admin/big-data-administration` provides more detail about the pros and cons of different types of storage.
 
 A Dataverse installation can be configured to allow out of band upload by setting the ``dataverse.files.\<id\>.upload-out-of-band`` JVM option to ``true``.
 By default, Dataverse supports uploading files via the :ref:`add-file-api`. With S3 stores, a direct upload process can be enabled to allow sending the file directly to the S3 store (without any intermediate copies on the Dataverse server).
 With the upload-out-of-band option enabled, it is also possible for file upload to be managed manually or via third-party tools, with the :ref:`Adding the Uploaded file to the Dataset <direct-add-to-dataset-api>` API call (described in the :doc:`/developers/s3-direct-upload-api` page) used to add metadata and inform Dataverse that a new file has been added to the relevant store.
 
-The following sections describe how to set up various types of stores and how to configure for multiple stores.
+The following sections describe how to set up various types of stores and how to configure for multiple stores. See also :ref:`choose-store`.
+
+.. _multiple-stores:
 
 Multi-store Basics
 ++++++++++++++++++
@@ -1105,6 +1108,8 @@ File stores have one option - the directory where files should be stored. This c
 
 Multiple file stores should specify different directories (which would nominally be the reason to use multiple file stores), but one may share the same directory as "\-Ddataverse.files.directory" option - this would result in temp files being stored in the /temp subdirectory within the file store's root directory.
 
+See also :ref:`file-stores`.
+
 Swift Storage
 +++++++++++++
 
@@ -1200,6 +1205,8 @@ The Dataverse Software S3 driver supports multi-part upload for large files (ove
 
 **Note:** The Dataverse Project Team is most familiar with AWS S3, and can provide support on its usage with the Dataverse Software. Thanks to community contributions, the application's architecture also allows non-AWS S3 providers. The Dataverse Project Team can provide very limited support on these other providers. We recommend reaching out to the wider Dataverse Project Community if you have questions.
 
+See also :ref:`s3-stores`.
+
 First: Set Up Accounts and Access Credentials
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -1430,6 +1437,8 @@ You may provide the values for these via any `supported MicroProfile Config API
 2. A non-empty ``dataverse.files.<id>.profile`` will be ignored when no credentials can be found for this profile name.
    Current codebase does not make use of "named profiles" as seen for AWS CLI besides credentials.
 
+.. _s3-compatible:
+
 Reported Working S3-Compatible Storage
 ######################################
 
@@ -1516,29 +1525,33 @@ In addition to having the type "remote" and requiring a label, Trusted Remote St
 These and other available options are described in the table below.
 
 Trusted remote stores can range from being a static trusted website to a sophisticated service managing access requests and logging activity
-and/or managing access to a secure enclave.  See :doc:`/developers/big-data-support` for additional information on how to use a trusted remote store. For specific remote stores, consult their documentation when configuring the remote store in your Dataverse installation.
+and/or managing access to a secure enclave.  See :doc:`/admin/big-data-administration` (specifically :ref:`remote-stores`) and :doc:`/developers/big-data-support` for additional information on how to use a trusted remote store. For specific remote stores, consult their documentation when configuring the remote store in your Dataverse installation.
 
-Note that in the current implementation, activites where Dataverse needs access to data bytes, e.g. to create thumbnails or validate hash values at publication will fail if a remote store does not allow Dataverse access. Implementers of such trusted remote stores should consider using Dataverse's settings to disable ingest, validation of files at publication, etc. as needed.
+Note that in the current implementation, activities where Dataverse needs access to data bytes, e.g. to create thumbnails or validate hash values at publication will fail if a remote store does not allow Dataverse access. Implementers of such trusted remote stores should consider using Dataverse's settings to disable ingest, validation of files at publication, etc. as needed.
 
 Once you have configured a trusted remote store, you can point your users to the :ref:`add-remote-file-api` section of the API Guide.
 
 .. table::
     :align: left
 
-    ===========================================  ==================  ==========================================================================  ===================
-    JVM Option                                   Value               Description                                                                 Default value
-    ===========================================  ==================  ==========================================================================  ===================
-    dataverse.files.<id>.type                    ``remote``          **Required** to mark this storage as remote.                                (none)
-    dataverse.files.<id>.label                   <?>                 **Required** label to be shown in the UI for this storage.                  (none)
-    dataverse.files.<id>.base-url                <?>                 **Required** All files must have URLs of the form <baseUrl>/* .             (none)
-    dataverse.files.<id>.base-store              <?>                 **Required** The id of a base store (of type file, s3, or swift).           (the default store)
-    dataverse.files.<id>.download-redirect       ``true``/``false``  Enable direct download (should usually be true).                            ``false``
-    dataverse.files.<id>.secret-key               <?>                 A key used to sign download requests sent to the remote store. Optional.   (none)
-    dataverse.files.<id>.url-expiration-minutes  <?>                 If direct downloads and using signing: time until links expire. Optional.   60
-    dataverse.files.<id>.remote-store-name       <?>                 A short name used in the UI to indicate where a file is located. Optional.  (none)
-    dataverse.files.<id>.remote-store-url        <?>                 A url to an info page about the remote store used in the UI. Optional.      (none)
+    =======================================================  ==================  ==========================================================================  ===================
+    JVM Option                                               Value               Description                                                                 Default value
+    =======================================================  ==================  ==========================================================================  ===================
+    dataverse.files.<id>.type                                ``remote``          **Required** to mark this storage as remote.                                (none)
+    dataverse.files.<id>.label                               <?>                 **Required** label to be shown in the UI for this storage.                  (none)
+    dataverse.files.<id>.base-url                            <?>                 **Required** All files must have URLs of the form <baseUrl>/* .             (none)
+    dataverse.files.<id>.base-store                          <?>                 **Required** The id of a base store (of type file, s3, or swift).           (the default store)
+    dataverse.files.<id>.upload-out-of-band                  ``true``            **Required to be true** Dataverse does not manage file placement            ``false``
+    dataverse.files.<id>.download-redirect                   ``true``/``false``  Enable direct download (should usually be true).                            ``false``
+    dataverse.files.<id>.secret-key                          <?>                 A key used to sign download requests sent to the remote store. Optional.    (none)
+    dataverse.files.<id>.public                              ``true``/``false``  True if the remote store does not enforce Dataverse access controls         ``false``
+    dataverse.files.<id>.ingestsizelimit                     <size in bytes>     Maximum size of files that should be ingested                               (none)
+    dataverse.files.<id>.url-expiration-minutes              <?>                 If direct downloads and using signing: time until links expire. Optional.   60
+    dataverse.files.<id>.remote-store-name                   <?>                 A short name used in the UI to indicate where a file is located. Optional.  (none)
+    dataverse.files.<id>.remote-store-url                    <?>                 A URL to an info page about the remote store used in the UI. Optional.      (none)
+    dataverse.files.<id>.files-not-accessible-by-dataverse   ``true``/``false``  True if the file is at the URL provided, false if that is a landing page    ``false``
 
-    ===========================================  ==================  ==========================================================================  ===================
+    =======================================================  ==================  ==========================================================================  ===================
 
 .. _globus-storage:
 
@@ -1578,6 +1591,7 @@ Once you have configured a globus store, or configured an S3 store for Globus ac
                                                                                  for a managed store) - using a microprofile alias is recommended            (none)
     dataverse.files.<id>.reference-endpoints-with-basepaths  <?>                 A comma separated list of *remote* trusted Globus endpoint id/<basePath>s   (none)
     dataverse.files.<id>.files-not-accessible-by-dataverse   ``true``/``false``  Should be false for S3 Connector-based *managed* stores, true for others    ``false``
+    dataverse.files.<id>.public                              ``true``/``false``  True can be used to disable users ability restrict/embargo files            ``false``
 
     =======================================================  ==================  ==========================================================================  ===================
 
@@ -2804,6 +2818,8 @@ when using it to configure your core name!
 
 Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_SOLR_PATH``.
 
+.. _dataverse.solr.min-files-to-use-proxy:
+
 dataverse.solr.min-files-to-use-proxy
 +++++++++++++++++++++++++++++++++++++
 
@@ -2815,6 +2831,8 @@ A recommended value would be ~1000 but the optimal value may vary depending on d
 
 Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_SOLR_MIN_FILES_TO_USE_PROXY``.
 
+.. _dataverse.solr.concurrency.max-async-indexes:
+
 dataverse.solr.concurrency.max-async-indexes
 ++++++++++++++++++++++++++++++++++++++++++++
 
@@ -4447,6 +4465,8 @@ Notes:
 
 - For larger file upload sizes, you may need to configure your reverse proxy timeout. If using apache2 (httpd) with Shibboleth, add a timeout to the ProxyPass defined in etc/httpd/conf.d/ssl.conf (which is described in the :doc:`/installation/shibboleth` setup).
 
+.. _:MultipleUploadFilesLimit:
+
 :MultipleUploadFilesLimit
 +++++++++++++++++++++++++
 
@@ -4525,6 +4545,8 @@ Examples:
 
    ``curl -X PUT -d '{"default":"0", "CSV":"268435456"}' http://localhost:8080/api/admin/settings/:TabularIngestSizeLimit``
 
+.. _:ZipUploadFilesLimit:
+
 :ZipUploadFilesLimit
 ++++++++++++++++++++
 
@@ -4543,13 +4565,17 @@ By default your Dataverse installation will attempt to connect to Solr on port 8
 
 **Note:** instead of using a database setting, you could alternatively use JVM settings like :ref:`dataverse.solr.host`.
 
+.. _:SolrFullTextIndexing:
+
 :SolrFullTextIndexing
 +++++++++++++++++++++
 
 Whether or not to index the content of files such as PDFs. The default is false.
 
 ``curl -X PUT -d true http://localhost:8080/api/admin/settings/:SolrFullTextIndexing``
 
+.. _:SolrMaxFileSizeForFullTextIndexing:
+
 :SolrMaxFileSizeForFullTextIndexing
 +++++++++++++++++++++++++++++++++++
 
@@ -4571,12 +4597,15 @@ To enable the setting::
 
   curl -X PUT -d true "http://localhost:8080/api/admin/settings/:DisableSolrFacets"
 
+.. _:DisableSolrFacetsForGuestUsers:
 
 :DisableSolrFacetsForGuestUsers
 +++++++++++++++++++++++++++++++
 
 Similar to the above, but will disable the facets for Guest (unauthenticated) users only. 
 
+.. _:DisableSolrFacetsWithoutJsession:
+
 :DisableSolrFacetsWithoutJsession
 +++++++++++++++++++++++++++++++++
 
@@ -5079,6 +5108,8 @@ If you don’t want date facets to be sorted chronologically, set:
 
 ``curl -X PUT -d 'false' http://localhost:8080/api/admin/settings/:ChronologicalDateFacets``
 
+.. _:CustomZipDownloadServiceUrl:
+
 :CustomZipDownloadServiceUrl
 ++++++++++++++++++++++++++++
 
@@ -5210,6 +5241,8 @@ A suggested minimum includes author, datasetContact, and contributor, but additi
 
 ``curl -X PUT -d 'author, datasetContact, contributor, depositor, grantNumber, publication' http://localhost:8080/api/admin/settings/:AnonymizedFieldTypeNames``
 
+.. _:DatasetChecksumValidationSizeLimit:
+
 :DatasetChecksumValidationSizeLimit
 +++++++++++++++++++++++++++++++++++
 
@@ -5225,6 +5258,8 @@ Refer to "Physical Files Validation in a Dataset" API :ref:`dataset-files-valida
 
 Also refer to the "Datafile Integrity" API  :ref:`datafile-integrity`
 
+.. _:DataFileChecksumValidationSizeLimit:
+
 :DataFileChecksumValidationSizeLimit
 ++++++++++++++++++++++++++++++++++++
 
@@ -5486,6 +5521,8 @@ To use the current GDCC version directly:
 
 ``curl -X PUT -d 'https://gdcc.github.io/dvwebloader/src/dvwebloader.html' http://localhost:8080/api/admin/settings/:WebloaderUrl``
 
+.. _:CategoryOrder:
+
 :CategoryOrder
 ++++++++++++++
 
@@ -5495,6 +5532,8 @@ The default is category ordering disabled.
 
 ``curl -X PUT -d 'Documentation,Data,Code' http://localhost:8080/api/admin/settings/:CategoryOrder``
 
+.. _:OrderByFolder:
+
 :OrderByFolder
 ++++++++++++++
 

doc/sphinx-guides/source/installation/config.rst

@@ -117,7 +117,7 @@ Decisions to Make
 
 Here are some questions to keep in the back of your mind as you test and move into production:
 
-- How much storage do I need?
+- How much storage do I need? What is the scale of data I will need to handle (see :doc:`/admin/big-data-administration`)?
 - Which features do I want based on :ref:`architecture`?
 - How do I want my users to log in to the Dataverse installation? With local accounts? With Shibboleth/SAML? With OAuth providers such as ORCID, GitHub, or Google?
 - Do I want to to run my app server on the standard web ports (80 and 443) or do I want to "front" my app server with a proxy such as Apache or nginx? See "Network Ports" in the :doc:`config` section.

doc/sphinx-guides/source/installation/prep.rst

@@ -391,14 +391,17 @@ If the bounding box was successfully populated, :ref:`geospatial-search` should
 Compressed Files
 ----------------
 
-Compressed files in .zip format are unpacked automatically. If a .zip file fails to unpack for whatever reason, it will upload as is. If the number of files inside are more than a set limit (1,000 by default, configurable by the Administrator), you will get an error message and the .zip file will upload as is.
+Depending on the configuration, compressed files in .zip format are unpacked automatically. If a .zip file is not unpacked, it will upload as is.
+If the number of files inside are more than a set limit (1,000 by default, configurable by the Administrator), you will get an error message and the .zip file will upload as is.
 
 If the uploaded .zip file contains a folder structure, the Dataverse installation will keep track of this structure. A file's location within this folder structure is displayed in the file metadata as the File Path. When you download the contents of the dataset, this folder structure will be preserved and files will appear in their original locations. 
 
 These folder names are subject to strict validation rules. Only the following characters are allowed: the alphanumerics, '_', '-', '.' and ' ' (white space). When a zip archive is uploaded, the folder names are automatically sanitized, with any invalid characters replaced by the '.' character. Any sequences of dots are further replaced with a single dot. For example, the folder name ``data&info/code=@137`` will be converted to ``data.info/code.137``. When uploading through the Web UI, the user can change the values further on the edit form presented, before clicking the 'Save' button. 
 
 .. note:: If you upload multiple .zip files to one dataset, any subdirectories that are identical across multiple .zips will be merged together when the user downloads the full dataset.
 
+If a .zip file is not unpacked and Zip Previewer is installed (see :ref:`file-previews`), it will be possible for users to view the contents of the zip file and to download individual files from within the .zip.
+
 Other File Types
 ----------------