updated docs and user/dev examples with better notes

Author: Vedant1

docs/backends.rst

@@ -38,12 +38,12 @@ DuckDB
    :members:
    :special-members: __init__
 
-SQLAlchemy
------------
+.. SQLAlchemy
+.. -----------
 
-.. automodule:: dsi.backends.sqlalchemy
-   :members:
-   :special-members: __init__
+.. .. automodule:: dsi.backends.sqlalchemy
+..    :members:
+..    :special-members: __init__
 
 GUFI
 ------

docs/core.rst

@@ -60,9 +60,9 @@ by creating a DSI database in the process, or retrieving an existing DSI databas
 Examples
 --------
 Examples below display various ways users can incorporate DSI into their data science workflows.
-They are located in ``examples/developer/`` and must be run from that directory.
+They must be executed from their directory in ``examples/developer/``
 
-Most of them either load or refer to data from ``examples/clover3d/``.
+To run them successfully, please unzip ``clover3d.zip`` located in ``examples/clover3d/``, and execute ``requirements.extras.txt``.
 
 Example 1: Intro use case
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

docs/introduction.rst

@@ -35,16 +35,21 @@ The DSI API is broken into three main categories:
 Expected Data Standards
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
-Before using DSI, users are expected to preprocess their data into a standardized format.
-DSI's actions are strict and will not commit any database actions if the data is unstable.
+Before using DSI, users should first standardize their data in a format that can be represented as a table. 
+DSI supports many widely used formats, including, but not limited to, CSV, JSON, YAML, TOML, and in-memory dictionaries.
+If the data is structured in a unique format, users can create an external DSI reader by following the steps in :ref:`custom_reader`.
 
-This can be achieved by organizing the data so it can be represented as a table in DSI. 
-It is also expected that each data point is a discrete value, rather than a complex data structure.
-If metadata is crucial to data representation, users should ensure it is stored with the data to be captured by DSI actions.
+When using a DSI-supported Reader, each data point is expected to be a discrete value — not a nested structure.
+Users must flatten any nested data to ensure compatibility with DSI.
 
-Users expecting to load a complex schema into DSI should also consider which columns in tables will be related to each other.
-This requires prior knowledge of primary and foreign keys, and writing a JSON file to represent this schema. 
-For more information on creating a schema compatible with DSI, please view :ref:`user_schema_example_label`.
+Metadata is important for many data workflows and should be stored with the data when relevant.
+For example, if simulation parameters are required for future analysis, that metadata should be included in the same table as the data.
+
+Advanced users familiar with database relationships can also load a complex relational schema into DSI alongside their data.
+This requires prior knowledge of primary and foreign keys, as well as how columns across tables should be related.
+
+Users must load this relational schema as a JSON into DSI using a Schema Reader.
+For more information on formatting the schema file correctly, refer to :ref:`user_schema_example_label`.
 
 DSI Readers/Writers
 ~~~~~~~~~~~~~~~~~~~~
@@ -62,6 +67,7 @@ Currently, DSI has the following Readers:
   - Schema (A complex schema reader)
   - YAML1
   - TOML1
+  - Collection (To load a Python dictionary or OrderedDict)
   - Bueno
   - Ensemble (Reader to ingest ensemble data. Ex: the `Wildfire ensemble dataset <https://github.com/lanl/dsi/tree/main/examples/wildfire>`_ . 
     Assumes each data row is a separate sim.)

docs/python_api.rst

@@ -69,9 +69,9 @@ Examples of each data card standard for the Wildfire dataset can be found in ``e
 User Examples
 --------------
 Examples below display various ways users can incorporate DSI into their data science workflows.
-They are located in ``examples/user/`` and must be run from that directory.
+They must be executed from their directory in ``examples/user/``
 
-All of them either load or refer to data in ``examples/clover3d/``. 
+To run them successfully, please unzip ``clover3d.zip`` located in ``examples/clover3d/``, and execute ``requirements.extras.txt``.
 
 Example 1: Intro use case
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -93,15 +93,20 @@ Printing various data and metadata from a DSI backend - number of tables, list o
 
 Example 4: Find data
 ~~~~~~~~~~~~~~~~~~~~
-Finding data from an active DSI backend that matches an input query - a string or a number.
-Prints all matches by default. If ``True`` is passed as an additional argument, returns rows of the first table that satisfies the query.
+Finding data from an active DSI backend that matches an input object. 
+
+If using ``search()``, the input can be a string or number. 
+If using ``find()``, the input must be a string in the form of a condition - [column] [operator] [value].
+
+By default, all matches are printed. If ``True`` is passed as an additional argument, the matching rows are returned as a DataFrame instead.
 
 .. literalinclude:: ../examples/user/4.find.py
 
 Example 5: Update data
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Updating data from the edited output of ``find()``. Input can be output of either ``find()``, ``query()``, or ``get_table()``.
-Users must NOT change metadata columns starting with **`dsi_`** even if adding new rows.
+Updating data from the edited output of ``find()``. Users must NOT modify metadata columns starting with **`dsi_`** even when adding new rows.
+
+The input can be the output of either ``find()``, ``search()``, ``query()``, or ``get_table()``.
 
 .. literalinclude:: ../examples/user/5.update.py
 
@@ -110,6 +115,8 @@ Example 6: Query data
 Querying data from an active DSI backend. 
 Users can either use ``query()`` to view specific data with a SQL statement, or ``get_table()`` to view all data from a specified table.
 
+By default, all matches are printed. If ``True`` is passed as an additional argument, the matching rows are returned as a DataFrame instead.
+
 .. literalinclude:: ../examples/user/6.query.py
 
 Example 7: Complex schema with data

examples/developer/10.notebook.py

@@ -4,9 +4,8 @@
 terminal_notebook = Terminal()
 
 #read data
-# UNZIP clover3d.zip INSIDE EXAMPLES/CLOVER3D/
 terminal_notebook.load_module('plugin', 'Schema', 'reader', filename="../clover3d/schema.json")
-terminal_notebook.load_module('plugin', 'Cloverleaf', 'reader', folder_path="../clover3d/clover3d/")
+terminal_notebook.load_module('plugin', 'Cloverleaf', 'reader', folder_path="../clover3d/")
 
 #ingest data to Sqlite backend
 terminal_notebook.load_module('backend','Sqlite','back-write', filename='jupyter_data.db')

examples/developer/2.ingest.py

@@ -3,8 +3,7 @@
 
 terminal_ingest = Terminal()
 
-# UNZIP clover3d.zip INSIDE EXAMPLES/CLOVER3D/
-terminal_ingest.load_module('plugin', 'Cloverleaf', 'reader', folder_path="../clover3d/clover3d/")
+terminal_ingest.load_module('plugin', 'Cloverleaf', 'reader', folder_path="../clover3d/")
 
 terminal_ingest.load_module('backend','Sqlite','back-write', filename='data.db')
 

examples/developer/3.schema.py

@@ -3,15 +3,13 @@
 
 terminal = Terminal()
 
-# UNZIP clover3d.zip INSIDE EXAMPLES/CLOVER3D/
 terminal.load_module('plugin', 'Schema', 'reader', filename="../clover3d/schema.json")
 
-terminal.load_module('plugin', 'Cloverleaf', 'reader', folder_path="../clover3d/clover3d/")
+terminal.load_module('plugin', 'Cloverleaf', 'reader', folder_path="../clover3d/")
 
 terminal.load_module('backend','Sqlite','back-write', filename='schema_data.db')
 
 terminal.artifact_handler(interaction_type='ingest')
 
-# EXECUTE requirements.extras.txt  to be able to run this
 terminal.load_module('plugin', 'ER_Diagram', 'writer', filename = 'er_diagram.png')
 terminal.transload()

examples/developer/4.visualize.py

@@ -25,7 +25,4 @@
 # prints numerical stats for only 'input'
 terminal_visualize.summary("input")
 
-# prints numerical stats for only 'input' and prints first 5 rows of the actual table
-terminal_visualize.summary("input", 5)
-
 terminal_visualize.close()

examples/developer/5.process.py

@@ -7,7 +7,6 @@
 terminal_process.load_module('backend','Sqlite','back-read', filename='schema_data.db')   
 terminal_process.artifact_handler(interaction_type="process")
 
-# EXECUTE requirements.extras.txt  to be able to run this
 terminal_process.load_module('plugin', 'ER_Diagram', 'writer', filename = 'er_diagram.png')
 
 terminal_process.load_module('plugin', "Table_Plot", "writer", table_name = "output", filename = "output_plot.png")

examples/user/2.read.py

@@ -4,8 +4,7 @@
 read_dsi = DSI("data.db") # Target a backend, defaults to SQLite if not defined
 
 #dsi.read(path, reader)
-# UNZIP clover3d.zip INSIDE EXAMPLES/CLOVER3D/
-read_dsi.read("../clover3d/clover3d/", 'Cloverleaf') # Read data into memory
+read_dsi.read("../clover3d/", 'Cloverleaf') # Read data into memory
 
 #dsi.display(table_name)
 read_dsi.display("input") # Print the specific table's data from the Cloverleaf data