Updated driver.rst and covariance.rst files

Author: slilonfe5

doc/OnlineDocs/explanation/analysis/parmest/covariance.rst

@@ -1,67 +1,76 @@
+.. _covariancesection:
+
 Covariance Matrix Estimation
 ============================
 
-The uncertainty in the estimated parameters is quantified using the covariance matrix.
-The diagonal of the covariance matrix contains the variance of the estimated parameters.
-Assuming Gaussian independent and identically distributed measurement errors, the
-covariance matrix of the estimated parameters can be computed using the following
-methods which have been implemented in parmest.
+The goal is parameter estimation (see :ref:`driversection` Section) is to estimate unknown model parameters
+from experimental data. When the model parameters are estimated from the data, their accuracy is measured by
+computing the covariance matrix. The diagonal of this covariance matrix contains the variance of the
+estimated parameters which is used to calculate their uncertainty. Assuming Gaussian independent and identically
+distributed measurement errors, the covariance matrix of the estimated parameters can be computed using the
+following methods which have been implemented in parmest.
 
 1. Reduced Hessian Method
 
     When the objective function is the sum of squared errors (SSE) defined as
-    :math:`\text{SSE} = \sum_{i = 1}^n
-    \left(\boldsymbol{y}_{i} - \hat{\boldsymbol{y}}_{i}\right)^2`,
+    :math:`\text{SSE} = \sum_{i = 1}^{n}
+    \left(\boldsymbol{y}_{i} - \boldsymbol{f}(\boldsymbol{x}_{i};\boldsymbol{\theta})\right)^2`,
     the covariance matrix is:
 
     .. math::
        \boldsymbol{V}_{\boldsymbol{\theta}} = 2 \sigma^2 \left(\frac{\partial^2 \text{SSE}}
-        {\partial \boldsymbol{\theta} \partial \boldsymbol{\theta}}\right)^{-1}_{\boldsymbol{\theta}
+        {\partial \boldsymbol{\theta}^2}\right)^{-1}_{\boldsymbol{\theta}
         = \hat{\boldsymbol{\theta}}}
 
     Similarly, when the objective function is the weighted SSE (WSSE) defined as
-    :math:`\text{WSSE} = \frac{1}{2} \sum_{i=1}^{n}
-    \left(\boldsymbol{y}_{i} - \hat{\boldsymbol{y}}_{i}\right)^\text{T}
-    \boldsymbol{\Sigma}_{\boldsymbol{y}}^{-1}
-    \left(\boldsymbol{y}_{i} - \hat{\boldsymbol{y}}_{i}\right)`,
+    :math:`\text{WSSE} = \frac{1}{2} \sum_{i = 1}^{n} \left(\boldsymbol{y}_{i} -
+    \boldsymbol{f}(\boldsymbol{x}_{i};\boldsymbol{\theta})\right)^\text{T} \boldsymbol{\Sigma}_{\boldsymbol{y}}^{-1}
+    \left(\boldsymbol{y}_{i} - \boldsymbol{f}(\boldsymbol{x}_{i};\boldsymbol{\theta})\right)`,
     the covariance matrix is:
 
     .. math::
        \boldsymbol{V}_{\boldsymbol{\theta}} = \left(\frac{\partial^2 \text{WSSE}}
-        {\partial \boldsymbol{\theta} \partial \boldsymbol{\theta}}\right)^{-1}_{\boldsymbol{\theta}
+        {\partial \boldsymbol{\theta}^2}\right)^{-1}_{\boldsymbol{\theta}
         = \hat{\boldsymbol{\theta}}}
 
     Where :math:`\boldsymbol{V}_{\boldsymbol{\theta}}` is the covariance matrix of the estimated
     parameters :math:`\hat{\boldsymbol{\theta}}`, :math:`\boldsymbol{y}` are observations of the measured variables,
-    :math:`\hat{\boldsymbol{y}}` are model predictions of the measured variables,
-    :math:`n` is the number of experiments, and :math:`\boldsymbol{\Sigma}_{\boldsymbol{y}}` is the measurement error
-    covariance matrix, whose leading diagonal contains the inverse of the variance of the measurement errors,
-    :math:`\sigma^2`. When the standard deviation of the measurement error is not supplied by the user, parmest
-    approximates :math:`\sigma^2` as: :math:`\hat{\sigma}^2 = \frac{1}{n-l} \sum_{i=1}^{n} e_i^2`, where :math:`l` is
-    the number of fitted parameters, and :math:`e_i` is the residual between the data and model prediction for
-    experiment :math:`i`.
+    :math:`n` is the number of experiments, :math:`\boldsymbol{\Sigma}_{\boldsymbol{y}}` is the measurement error
+    covariance matrix, and :math:`\sigma^2` is the variance of the measurement error. When the standard deviation of
+    the measurement error is not supplied by the user, parmest approximates :math:`\sigma^2` as:
+    :math:`\hat{\sigma}^2 = \frac{1}{n-l} \sum_{i=1}^{n} e_i^2`, where :math:`l` is the number of fitted parameters,
+    and :math:`e_i` is the residual between the data and model for experiment :math:`i`.
 
     In parmest, this method computes the inverse of the Hessian by scaling the
     objective function (SSE or WSSE) with a constant probability factor, :math:`\frac{1}{n}`.
 
 2. Finite Difference Method
 
     In this method, the covariance matrix, :math:`\boldsymbol{V}_{\boldsymbol{\theta}}`, is
-    calculated by applying the Gauss-Newton approximation to the Hessian,
+    computed by differentiating the Hessian,
     :math:`\frac{\partial^2 \text{SSE}}{\partial \boldsymbol{\theta} \partial \boldsymbol{\theta}}`
     or
-    :math:`\frac{\partial^2 \text{WSSE}}{\partial \boldsymbol{\theta} \partial \boldsymbol{\theta}}`,
-    leading to:
+    :math:`\frac{\partial^2 \text{WSSE}}{\partial \boldsymbol{\theta} \partial \boldsymbol{\theta}}`, and
+    applying Gauss-Newton approximation which results in:
 
     .. math::
        \boldsymbol{V}_{\boldsymbol{\theta}} = \left(\sum_{i = 1}^n \boldsymbol{G}_{i}^{\text{T}}
         \boldsymbol{\Sigma}_{\boldsymbol{y}}^{-1} \boldsymbol{G}_{i} \right)^{-1}
 
-    This method uses central finite difference to compute the Jacobian matrix,
-    :math:`\boldsymbol{G}_{i}`, for experiment :math:`i`, which is the sensitivity of
-    the measured variables :math:`\boldsymbol{y}` with respect to the parameters, :math:`\boldsymbol{\theta}`.
-    :math:`\boldsymbol{\Sigma}_{\boldsymbol{y}}` is the measurement error covariance matrix, whose leading diagonal
-    contains the inverse of the variance of the measurement errors, :math:`\sigma^2`.
+    where
+
+    .. math::
+       \boldsymbol{G}_{i} = \frac{\partial \boldsymbol{f}(\boldsymbol{x}_i;\boldsymbol{\theta})}
+        {\partial \boldsymbol{\theta}}
+
+    This method uses central finite difference to compute the Jacobian matrix, :math:`\boldsymbol{G}_{i}`,
+    for experiment :math:`i`.
+
+    .. math::
+       \boldsymbol{G}_{i}[:,\,k] \approx \frac{\boldsymbol{f}(\boldsymbol{x}_i;\theta_k + \Delta \theta_k)
+        \vert_{\hat{\boldsymbol{\theta}}} - \boldsymbol{f}(\boldsymbol{x}_i;\theta_k - \Delta \theta_k)
+        \vert_{\hat{\boldsymbol{\theta}}}}{2 \Delta \theta_k} \quad \forall \quad \theta_k \, \in \,
+        [\theta_1,\cdots, \theta_p]
 
 3. Automatic Differentiation Method
 
@@ -71,23 +80,24 @@ methods which have been implemented in parmest.
        \boldsymbol{V}_{\boldsymbol{\theta}} = \left( \sum_{i = 1}^n \boldsymbol{G}_{\text{kaug},\, i}^{\text{T}}
         \boldsymbol{\Sigma}_{\boldsymbol{y}}^{-1} \boldsymbol{G}_{\text{kaug},\, i} \right)^{-1}
 
-    However, this method uses the model optimality (KKT) condition to compute the Jacobian matrix,
-    :math:`\boldsymbol{G}_{\text{kaug},\, i}`, for experiment :math:`i`.
+    However, this method uses implicit differentiation and the model-optimality or Karush–Kuhn–Tucker (KKT) conditions
+    to compute the Jacobian matrix, :math:`\boldsymbol{G}_{\text{kaug},\, i}`, for experiment :math:`i`.
+
+    .. math::
+       \boldsymbol{G}_{\text{kaug},\,i} = \frac{\partial \boldsymbol{f}(\boldsymbol{x}_i,\boldsymbol{\theta})}
+        {\partial \boldsymbol{\theta}} + \frac{\partial \boldsymbol{f}(\boldsymbol{x}_i,\boldsymbol{\theta})}
+        {\partial \boldsymbol{x}_i}\frac{\partial \boldsymbol{x}_i}{\partial \boldsymbol{\theta}}
 
-The covariance matrix calculation is only supported with the built-in objective
-functions "SSE" or "SSE_weighted".
+The covariance matrix calculation is only supported with the built-in objective functions "SSE" or "SSE_weighted".
 
 In parmest, the covariance matrix can be computed after creating the
 :class:`~pyomo.contrib.parmest.experiment.Experiment` class,
-defining the :class:`~pyomo.contrib.parmest.parmest.Estimator` object, and estimating the unknown parameters using
-:class:`~pyomo.contrib.parmest.parmest.Estimator.theta_est` (these steps were addressed in the
-:ref:`driversection` Section).
+defining the :class:`~pyomo.contrib.parmest.parmest.Estimator` object,
+and estimating the model parameters using :class:`~pyomo.contrib.parmest.parmest.Estimator.theta_est`
+(all these steps were addressed in the :ref:`driversection` Section).
 
 To estimate the covariance matrix, with the default method being "finite_difference", call
-the :class:`~pyomo.contrib.parmest.parmest.Estimator.cov_est` function as follows (note that this call should be made
-after creating the :class:`~pyomo.contrib.parmest.experiment.Experiment` class and the list of
-:class:`~pyomo.contrib.parmest.experiment.Experiment` objects as shown in the :ref:`driversection` Section):
-:
+the :class:`~pyomo.contrib.parmest.parmest.Estimator.cov_est` function as follows:
 
 .. testsetup:: *
     :skipif: not __import__('pyomo.contrib.parmest.parmest').contrib.parmest.parmest.parmest_available