reorganised docs

Author: robertmartin8

README.md

@@ -330,8 +330,10 @@ ef.max_sharpe()
 
 ### Other optimisers
 
-The features above mostly pertain to solving efficient frontier optimisation problems via quadratic programming (though this is taken care of by `cvxpy`). However, we offer different optimisers as well:
+The features above mostly pertain to solving mean-variance optimisation problems via quadratic programming (though this is taken care of by `cvxpy`). However, we offer different optimisers as well:
 
+- Mean-semivariance optimisation
+- Mean-CVaR optimisation
 - Hierarchical Risk Parity, using clustering algorithms to choose uncorrelated assets
 - Markowitz's critical line algorithm (CLA)
 

docs/EfficientFrontier.rst

@@ -14,12 +14,12 @@ superior models and feed them into the optimiser.
 
 .. caution::
 
-    In my experience, supplying expected returns often does more harm than good. If
+    Supplying expected returns can do more harm than good. If
     predicting stock returns were as easy as calculating the mean historical return,
     we'd all be rich! For most use-cases, I would suggest that you focus your efforts
     on choosing an appropriate risk model (see :ref:`risk-models`). 
 
-    As of v0.5.0, you can use :ref:`black-litterman` to greatly improve the quality of
+    As of v0.5.0, you can use :ref:`black-litterman` to significantly improve the quality of
     your estimate of the expected returns.
 
 .. automodule:: pypfopt.expected_returns
@@ -33,7 +33,7 @@ superior models and feed them into the optimiser.
 
         This is probably the default textbook approach. It is intuitive and easily interpretable,
         however the estimates are subject to large uncertainty. This is a problem especially in the
-        context of a quadratic optimiser, which will maximise the erroneous inputs.
+        context of a mean-variance optimiser, which will maximise the erroneous inputs.
 
 
     .. autofunction:: ema_historical_return

docs/ExpectedReturns.rst

@@ -0,0 +1,212 @@
+.. _efficient-frontier:
+
+##########################
+General Efficient Frontier
+##########################
+
+The mean-variance optimisation methods described previously can be used whenever you have a vector
+of expected returns and a covariance matrix. The objective and constraints will be some combination
+of the portfolio return and portfolio volatility. 
+
+However, you may want to construct the efficient frontier for an entirely different type of risk model
+(one that doesn't depend on covariance matrices), or optimise an objective unrelated to portfolio
+return (e.g tracking error). PyPortfolioOpt comes with several popular alternatives and provides support
+for custom optimisation problems.
+
+Efficient Semivariance
+======================
+
+Instead of penalising volatility, mean-semivariance optimisation seeks to only penalise
+downside volatility, since upside volatility may be desirable. 
+
+There are two approaches to the mean-semivariance optimisation problem. The first is to use a
+heuristic (i.e "quick and dirty") solution: pretending that the semicovariance matrix
+(implemented in :py:mod:`risk_models`) is a typical covariance matrix and doing standard
+mean-variance optimisation. It can be shown that this *does not* yield a portfolio that
+is efficient in mean-semivariance space (though it might be a good-enough approximation).
+
+Fortunately, it is possible to write mean-semivariance optimisation as a convex problem
+(albeit one with many variables), that can be solved to give an "exact" solution.
+For example, to maximise return for a target semivariance
+:math:`s^*` (long-only), we would solve the following problem:
+
+.. math::
+
+    \begin{equation*}
+    \begin{aligned}
+    & \underset{w}{\text{maximise}} & & w^T \mu \\
+    & \text{subject to} & & n^T n \leq s^*  \\
+    &&& B w - p + n = 0 \\
+    &&& w^T \mathbf{1} = 1 \\
+    &&& n \geq 0 \\
+    &&& p \geq 0. \\
+    \end{aligned}
+    \end{equation*}
+
+Here, **B** is the :math:`T \times N` (scaled) matrix of excess returns:
+``B = (returns - benchmark) / sqrt(T)``. Additional linear equality constraints and
+convex inequality constraints can be added. 
+
+PyPortfolioOpt allows users to optimise along the efficient semivariance frontier
+via the :py:class:`EfficientSemivariance` class. :py:class:`EfficientSemivariance` inherits from
+:py:class:`EfficientFrontier`, so it has the same utility methods 
+(e.g :py:func:`add_constraint`, :py:func:`portfolio_performance`), but finds portfolios on the mean-semivariance
+frontier. Note  that some of the parent methods, like :py:func:`max_sharpe` and :py:func:`min_volatility`
+are not applicable to mean-semivariance portfolios, so calling them returns ``NotImplementedError``.
+
+:py:class:`EfficientSemivariance` has a slightly different API to :py:class:`EfficientFrontier`. Instead of passing
+in a covariance matrix, you should past in a dataframe of historical/simulated returns (this can be constructed
+from your price dataframe using the helper method :py:func:`expected_returns.returns_from_prices`). Here
+is a full example, in which we seek the portfolio that minimises the semivariance for a target
+annual return of 20%::
+
+    from pypfopt import expected_returns, EfficientSemivariance
+
+    df = ... # your dataframe of prices
+    mu = expected_returns.mean_historical_returns(df)
+    historical_returns = expected_returns.returns_from_prices(df)
+
+    es = EfficientSemivariance(mu, historical_returns)
+    es.efficient_return(0.20)
+
+    # We can use the same helper methods as before
+    weights = es.clean_weights() 
+    print(weights)
+    es.portfolio_performance(verbose=True)
+
+The ``portfolio_performance`` method outputs the expected portfolio return, semivariance,
+and the Sortino ratio (like the Sharpe ratio, but for downside deviation).
+
+Interested readers should refer to Estrada (2007) [1]_ for more details. I'd like to thank 
+`Philipp Schiele <https://github.com/phschiele>`_ for authoring the bulk
+of the efficient semivariance functionality and documentation (all errors are my own). The
+implementation is based on Markowitz et al (2019) [2]_.
+
+.. caution::
+
+    Finding portfolios on the mean-semivariance frontier is computationally harder
+    than standard mean-variance optimisation: our implementation uses ``2T + N`` optimisation variables, 
+    meaning that for 50 assets and 3 years of data, there are about 1500 variables.  
+    While :py:class:`EfficientSemivariance` allows for additional constraints/objectives in principle,
+    you are much more likely to run into solver errors. I suggest that you keep :py:class:`EfficientSemivariance`
+    problems small and minimally constrained.
+
+.. autoclass:: pypfopt.efficient_frontier.EfficientSemivariance
+    :members:
+    :exclude-members: max_sharpe, min_volatility
+
+Efficient CVaR
+==============
+
+The **conditional value-at-risk** (a.k.a **expected shortfall**) is a popular measure of tail risk. The CVaR can be
+thought of as the average of losses that occur on "very bad days", where "very bad" is quantified by the parameter
+:math:`\beta`.
+
+For example, if we calculate the CVaR to be 10% for :math:`\beta = 0.95`, we can be 95% confident that the worst-case
+average daily loss will be 3%. Put differently, the CVaR is the average of all losses so severe that they only occur
+:math:`(1-\beta)\%` of the time. 
+
+While CVaR is quite an intuitive concept, a lot of new notation is required to formulate it mathematically (see
+the `wiki page <https://en.wikipedia.org/wiki/Expected_shortfall>`_ for more details). We will adopt the following
+notation: 
+
+- *w* for the vector of portfolio weights
+- *r* for a vector of asset returns (daily), with probability distribution :math:`p(r)`.
+- :math:`L(w, r) = - w^T r` for the loss of the portfolio
+- :math:`\alpha` for the portfolio value-at-risk (VaR) with confidence :math:`\beta`.
+
+The CVaR can then be written as:
+
+.. math::
+    CVaR(w, \beta) = \frac{1}{1-\beta} \int_{L(w, r) \geq \alpha (w)} L(w, r) p(r)dr.
+
+This is a nasty expression to optimise because we are essentially integrating over VaR values. The key insight
+of Rockafellar and Uryasev (2001) [3]_ is that we can can equivalently optimise the following convex function:
+
+.. math:: 
+    F_\beta (w, \alpha) = \alpha + \frac{1}{1-\beta} \int [-w^T r - \alpha]^+ p(r) dr,
+
+where :math:`[x]^+ = \max(x, 0)`. The authors prove that minimising :math:`F_\beta(w, \alpha)` over all
+:math:`w, \alpha` minimises the CVaR. Suppose we have a sample of *T* daily returns (these
+can either be historical or simulated). The integral in the expression becomes a sum, so the CVaR optimisation
+problem reduces to a linear program:
+
+.. math::
+
+    \begin{equation*}
+    \begin{aligned}
+    & \underset{w, \alpha}{\text{minimise}} & & \alpha + \frac{1}{1-\beta} \frac 1 T \sum_{i=1}^T u_i \\
+    & \text{subject to} & & u_i \geq 0  \\
+    &&&  u_i \geq -w^T r_i - \alpha. \\
+    \end{aligned}
+    \end{equation*}
+
+This formulation introduces a new variable for each datapoint (similar to Efficient Semivariance), so
+you may run into performance issues for long returns dataframes. At the same time, you should aim to
+provide a sample of data that is large enough to include tail events. 
+
+I am grateful to `Nicolas Knudde <https://github.com/nknudde>`_ for the initial draft (all errors are my own).
+The implementation is based on Rockafellar and Uryasev (2001) [3]_.
+
+
+.. autoclass:: pypfopt.efficient_frontier.EfficientCVaR
+    :members:
+    :exclude-members: max_sharpe, min_volatility, max_quadratic_utility
+
+
+.. _custom-optimisation:
+
+Custom optimisation problems
+============================
+
+We have seen previously that it is easy to add constraints to ``EfficientFrontier`` objects (and
+by extension, other general efficient frontier objects like ``EfficientSemivariance``). However, what if you aren't interested
+in anything related to ``max_sharpe()``, ``min_volatility()``, ``efficient_risk()`` etc and want to
+set up a completely new problem to optimise for some custom objective?
+
+For example, perhaps our objective is to construct a basket of assets that best replicates a
+particular index, in otherwords, to minimise the **tracking error**. This does not fit within
+a mean-variance optimisation paradigm, but we can still implement it in PyPortfolioOpt::
+
+    from pypfopt.base_optimizer import BaseConvexOptimizer
+    from pypfopt.objective_functions import ex_post_tracking error
+
+    historic_returns = ... # dataframe of historic asset returns
+    S = risk_models.sample_cov(historic_returns, returns_data=True)
+
+    opt = BaseConvexOptimizer(
+        n_assets=len(historic_returns.columns), 
+        tickers=historic_returns.index,
+        weight_bounds=(0, 1)
+    )
+    opt.convex_objective(
+        objective_functions.ex_post_tracking_error,
+        historic_returns=historical_rets,
+        benchmark_returns=benchmark_rets,
+    ) 
+    weights = opt.clean_weights()
+
+The ``EfficientFrontier`` class inherits from ``BaseConvexOptimizer``. It may be more convenient
+to call ``convex_objective`` from an ``EfficientFrontier`` instance than from ``BaseConvexOptimizer``,
+particularly if your objective depends on the mean returns or covariance matrix. 
+
+You can either optimise some generic ``convex_objective``
+(which *must* be built using ``cvxpy`` atomic functions -- see `here <https://www.cvxpy.org/tutorial/functions/index.html>`_)
+or a ``nonconvex_objective``, which uses ``scipy.optimize`` as the backend and thus has a completely
+different API. For more examples, check out this `cookbook recipe
+<https://github.com/robertmartin8/PyPortfolioOpt/blob/master/cookbook/3-Advanced-Mean-Variance-Optimisation.ipynb>`_.
+
+    .. class:: pypfopt.base_optimizer.BaseConvexOptimizer
+
+        .. automethod:: convex_objective
+
+        .. automethod:: nonconvex_objective
+
+
+
+References
+==========
+
+.. [1] Estrada, J (2007). `Mean-Semivariance Optimization: A Heuristic Approach <https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1028206>`_.
+.. [2] Markowitz, H.; Starer, D.; Fram, H.; Gerber, S. (2019). `Avoiding the Downside <https://www.hudsonbaycapital.com/documents/FG/hudsonbay/research/599440_paper.pdf>`_.
+.. [3] Rockafellar, R.; Uryasev, D. (2001). `Optimization of conditional value-at-risk <https://www.ise.ufl.edu/uryasev/files/2011/11/CVaR1_JOR.pdf>`_