cleanups and example additions along with doctests

Author: sagarlf

docs/python/classes.md

@@ -8,14 +8,17 @@ sidebar_label: Classes
 
 * Avoid inbuilt names.
 * Classes names should always be `PascalCase`. i.e. `MyClass` 
-* Use [abstract containers](https://docs.python.org/3/library/collections.abc.html#module-collections.abc) if need to override datatypes.
-    + If you must build datatypes based on inbuilt class use PascalCase. i.e. `MyDict(dict):`.
+* **Abstract Classes and Mixins**
+    + Use [abstract containers](https://docs.python.org/3/library/collections.abc.html#module-collections.abc) if need to override datatypes.
+        - If you must build datatypes based on inbuilt class use PascalCase. i.e. `MyDict(dict):`.
+    + Use [`abc`](https://docs.python.org/3/library/abc.html) if you need pure OOP style abstract classes. Use `NotImplementedError` exceptions with overrides.
+    + mixin should be named with `Mixin` suffix such as `class LoginRequiredMixin` which can be used in multiple inheritance.
 * Describe the class responsibility in name.
 * Custom Exceptions should always be named ending with `Error` i.e. `MyCustomError`
 
 ##### Example
 
-```
+```python
 class HelloWorld:
     pass
 

docs/python/docstrings.md

@@ -32,7 +32,7 @@ may simply be construed as a comment
 
 To illustrate this try the following in the python console
 
-```python3
+```python
 class Test: 
    """This is a class docstring 
    """ 
@@ -48,28 +48,28 @@ class Test:
 ```
 
 
-```python3
+```python
 > print(Test.__doc___)
 This is a class docstring
 >
 > print(Test.example_method.__doc__)
 This is a method docstring
 >
-> print(Test.example_method_2.__doc__
+> print(Test.example_method_2.__doc__)
 None
 ```
 
 As you can see from the examples above, docstrings get 
 attached to the `__doc__` property of the code itself whereas, 
-the comments do not
+the comments do not.
 
 
 ### Usage of docstrings
 
 From the console, you can use docstrings to an overview of 
-code as follows
+code as follows:
 
-```python3
+```python
 > help(Test)
 Help on class Test in module __main__:
 
@@ -87,7 +87,9 @@ class Test(builtins.object)
 ```
 
 If a docstring is provided, you can get more readable
-information about python code
+information about python code. 
+
+** They are also used by your IDE to give you information while developing.**
 
 Furthermore, there are tools that can take this to the
 next level by creating a static website of documentation
@@ -122,12 +124,11 @@ template for consistency and also for libraries to be able to
 parse your docstring easily.
 
 The official documentation standard for Python is ReStructed Text docstrings ([PEP 287](https://www.python.org/dev/peps/pep-0287/)). 
-However, Google docstrings have been widely accepted by the community as such we recommend it. 
-If you're using using Numpy related libraries, it is better
-to use Numpy Docstrings
+However, Google docstrings have been widely accepted by the community as such we recommend it as we find it more readable and pythonic.
+If you're using using Numpy related libraries, you should be using Numpy Docstrings
 
-- [reStructured Text Docstrings](http://docutils.sourceforge.net/rst.html)
 - [Google docstrings](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings) (**Recommended**)
+- [reStructured Text Docstrings](http://docutils.sourceforge.net/rst.html) (Seen in many inbuilt python libraries.)
 - [Numpy Docstrings](https://numpydoc.readthedocs.io/en/latest/format.html) (**Recommended for AI projects**)
 
 
@@ -193,7 +194,7 @@ to check that your docstrings render correctly
 
 ### Google Docstrings (recommended)
 
-```python3
+```python
 
 """Example Google style docstrings.
 
@@ -496,9 +497,12 @@ class ExampleClass:
 
 ### ReStructured Text Doc strings
 
-Here are few examples of docstrings to get you started
+You can see this kind of docstring especially in python libraries. 
+Please use **Google** style as they are more readable.
+
+##### Examples:
 
-```python3
+```python
 """The method below prints a given string twice
 
 The print method has been called twice for 
@@ -514,12 +518,8 @@ def print_twice(param1):
     print(param1)
 
     return len(param1)
-```
 
-The following shows type definition on the same line. 
-
-```python3
-"""The method below prints a given string twice
+"""The method below prints a given string twice. This is for type annotation.
 
 The print method has been called twice for 
 implementing this method
@@ -528,13 +528,84 @@ implementing this method
 :return: Length of the input string
 :rtype: int
 """
-def print_twice(param1):
+def print_twice(param1: str) -> int:
     print(param1)
     print(param1)
 
     return len(param1)
 ```
 
+## Doctests
+:::caution
+While this should not be used for all checks and [testing](testing.md) should be followed. They can be used for simple parameter checking. i.e. some default case like module level doctest in example below. 
+
+They can be used with [pytest](https://docs.pytest.org/en/latest/doctest.html) as well.
+:::
+
+Python provides tests through docstrings which can be leveraged by [doctests](https://docs.python.org/3/library/doctest.html).
+
+Example
+```python
+"""
+This is the "example" module.
+
+The example module supplies one function, factorial().  For example,
+
+>>> factorial(5)
+120
+"""
+
+def factorial(n):
+    """Return the factorial of n, an exact integer >= 0.
+
+    >>> [factorial(n) for n in range(6)]
+    [1, 1, 2, 6, 24, 120]
+    >>> factorial(30)
+    265252859812191058636308480000000
+    >>> factorial(-1)
+    Traceback (most recent call last):
+        ...
+    ValueError: n must be >= 0
+
+    Factorials of floats are OK, but the float must be an exact integer:
+    >>> factorial(30.1)
+    Traceback (most recent call last):
+        ...
+    ValueError: n must be exact integer
+    >>> factorial(30.0)
+    265252859812191058636308480000000
+
+    It must also not be ridiculously large:
+    >>> factorial(1e100)
+    Traceback (most recent call last):
+        ...
+    OverflowError: n too large
+    """
+
+    import math
+    if not n >= 0:
+        raise ValueError("n must be >= 0")
+    if math.floor(n) != n:
+        raise ValueError("n must be exact integer")
+    if n+1 == n:  # catch a value like 1e300
+        raise OverflowError("n too large")
+    result = 1
+    factor = 2
+    while factor <= n:
+        result *= factor
+        factor += 1
+    return result
+
+
+if __name__ == "__main__":
+    import doctest
+    doctest.testmod()
+```
+This can be tested with:
+```
+$python example.py -v
+```
+
 ## References
 
 Thanks to the following

docs/python/environment_and_dependency.md

@@ -6,7 +6,7 @@ sidebar_label: Environment Isolation and Dependency Management
 
 #### Information on development environment and dependency:
 
-##### Environment Isolation:
+### Environment Isolation:
 
 * System installed `python` should never be used for development. Isolate your development.
 * Any of the following can be used for python isolation:
@@ -17,7 +17,8 @@ sidebar_label: Environment Isolation and Dependency Management
 
 
 
-##### Dependency Management:
+### Dependency Management:
 
 * [poetry](https://python-poetry.org/) is recommended as it handles dependency as well as build system.
 * You can use `setuptools` and `setup.py` as well for requirements handling through `requires`. They **must** be used for install-able modules.
+* `requirements.txt`  style should not be avoided.

docs/python/exceptions.md

@@ -9,10 +9,33 @@ sidebar_label: Exception Handling
 * `Exception` handling is a must and should be mitigated.
 * Do not use bare `except` or `except Exception` which catches all the exception. 
     - Always be specific on exception. E.g. catch only `FileNotFoundError` if you are say moving a file.
-* User Defined Exceptions:
+* **User Defined Exceptions**:
     - Write your custom error only when your error is not described or fulfilled by [internal exceptions](https://docs.python.org/3/library/exceptions.html).
     - Create custom `Exception` class primarily suffixing it with `Error` such as `MyCustomError(Exception)` and use it.
     - Always use `Exception` as your parent class for user defined exceptions. Donot use `BaseException`.
 * Add traceback to your mitigation. i.e. either `logging` or mails. Donot `pass`.
 * The `try` block should be specific to desired exception. Donot use huge code chunk in `try`. Use `else` if needed.
+```python
+    try:
+        value = int(some_str)
+    except ValueError: # when not using exception object
+        WHEN some_str IS not valid as value.
+    except TypeError:
+        WHEN some_str IS OTHER TYPE such as [], ()
+    else: #can be avoided if exceptions are handled and returned. This is in context to try block.
+        DO SOMETHING WITH VALUE
+    
+    try:
+        value = int(some_str)
+    except (ValueError, TypeError) as error:
+        HANDLE BOTH exception and use exception object
+    else: # If needed
+        do something when try succeeds.
+    finally:
+        codebase to run anyway
+```
+* `finally` can be used if you need to run the block whatever the case. `context` can be used in many cases to avoid `finally`.
+* `sys.exc_info` and [`traceback`](https://docs.python.org/3/library/traceback.html) can be used for traceback.
+* Please read [this](https://cosmicpercolator.com/2016/01/13/exception-leaks-in-python-2-and-3/) on exceptions handling internals and leaks when referencing exceptions.
+
 

docs/python/functions.md

@@ -14,7 +14,7 @@ sidebar_label: Functions
 * `decorators` should be named in function convention.
 
 
-```
+```python
 def get_db_connection(username, db_name):
     return connection
 

docs/python/general.md

@@ -23,34 +23,51 @@ sidebar_label: General Coding Guidelines
     - Try to use parent namespace for actions. i.e. `import sys: sys.path` rather that `from sys import path`
     - Never use `import *` i.e. `from MODULE import *`
     - use namespace whenever possible. Use `as SOMEOTHERNAMESPACE` for collision of namespace
-* Use `else: #nobreak` if you are using `else` with loops that has `break`.
+* If you are using `else` with loops that has `break`. Just comment `#nobreak` for reference as it is for that usage. See [this](http://python-notes.curiousefficiency.org/en/latest/python_concepts/break_else.html) for some clarity.
+    ```python
+        for each in each_collection:
+            if condition:
+                break
+        else: #nobreak
+            ELSE LOGIC
+        
+        while True:
+            if condition:
+                break
+        else: #nobreak
+            ELSE LOGIC
+    ```
 * Use `pathlib` for path related use case rather than `os.path`
 * Use `mypy` and type annotation when possible for type safe code.
-* `Docker` can be used for deployment. Use `python` images for [docker](https://hub.docker.com/_/python).
+* `Docker` can be used for deployment. Use `python` images for [`docker`](https://hub.docker.com/_/python).
 * Use `generators` and `yield` instead of data structures for high streams of data.
-* Use `itertools`, `functools` for utilities and `collection` for data structures.
+* Use `itertools`, `functools` for utilities and `collections` for data structures.
 * Use `dataclasses` if available. Go for `attrs` library if `dataclass` is not present.
-* Use `is not` and `is` for `None`, `True` and `False` specific check only. If most cases truthy and falsy can be checked with if `VARNAME:`. 
+* Use `is not` and `is` for `None`, `True` and `False` specific check only. If most cases truthy and falsy can be checked with  `if VARNAME:`. 
 * Strings: 
-    - Adhere to one quote practice. Double quote is recommended. Python doesnot differentiate between **'** or **"**.
+    - Adhere to one quote practice. Double quote is recommended. Python doesnot differentiate between **'** or **"**. This may be controlled by `formatter` used as well.
     - Should be interpolated with either [fstring](https://www.python.org/dev/peps/pep-0498/) or `.format` methods. Try to avoid `%`.
         + `format_map` should be used for key mapped formatting.
     - `+` can be used for direct string concatenation. Use `join` method for concatenation instead of `+=` when iterating.
 * Use `context` whenever supported especially for io related closing actions.
     - i.e. `with` statement when supported.
     - Always remember to close on exit. i.e. if you open the file `close` on `finally` or better use `with` or `contextlib.closing`.
-* While `python` is an OOP, you can always choose `functions` and `modules` over `class` if there is only one `object` to be created.
-* Use `property` setter when writing OOP and you need readonly attributes.
+* **OOP**
+    - While `python` is an OOP, you can always choose `functions` and `modules` over `class` if there is only one `object` to be created like `Singletons`.
+    - Use [`abc`](https://docs.python.org/3/library/abc.html) if you need abstraction. Mixins are more famous in python due to multiple inheritance.
+    - Use `property` setter getter only when you need readonly attributes. `__` variables can be used for some privacy.
+    - Use `super` for overrides and parent calls.
 * Use `pdb` as debugger whenever required.
 * Multi-threading can be especially used when we have io bound and network bound multiple operation. Multiprocessing can be used to use multiple cores.
     - Recommended module is `concurrent.futures` in most cases. If lower level API is needed there is always `threading` and `multiprocessing` module.
-    - Be very carefult on threads and locks, so always discuss what you are doing.
+    - Be very carefult on threads and locks, so always discuss what you are doing as it may not always be optimized.
     - Use `asyncio` for IO bound async flow. This is something new and constantly changing in `python`.
 * Recommended third party modules:
     - For Relational Database:
-        + Use `sqlalchemy` [core](https://docs.sqlalchemy.org/en/13/core/) for DB abstraction. This is particularly helpful when doing testing in `sqlite` and some other database for production. Also, for query consistency.
-        + Use `sqlalchemy` [ORM](https://docs.sqlalchemy.org/en/13/orm/) or framework supported ORM when using specific framework.
-        + Use DBAPI drivers such as `pyodbc`, `sqlite`, `mysqlclient` etc only when you donot want `sqlalchemy` dependency or when you are very performance conscious. While the API will be mostly compatible for this as python has DBAPI specification. Parameters binding and some methods may be incompatible or unavailable. 
+        + Use `sqlalchemy` [core](https://docs.sqlalchemy.org/en/13/core/) for DB abstraction. This is particularly helpful when doing testing in `sqlite` and some other database for production. Also, for query and parameters consistency.
+        + Use `sqlalchemy` [ORM](https://docs.sqlalchemy.org/en/13/orm/) or framework supported **ORM** when using specific framework.
+        + Use DBAPI drivers such as `pyodbc`, `sqlite`, `mysqlclient` etc only when you donot want `sqlalchemy` dependency or when you are very performance conscious. While the API will be mostly compatible for this as python has DBAPI specification. Parameters binding and some methods may be incompatible or unavailable. **sqlalchemy core is recommended.**
     - `requests` for http request stuff.
-    - `attrs` for data oriented objects and class.
+        + `aiohttp` or `httpx` are also good.
+    - `attrs` for data oriented objects and classes design.
     - `pytest` for tests.

docs/python/logging.md

@@ -13,8 +13,8 @@ sidebar_label: Logging Convention
     - **WARN**: log user security and other warnings that may require attention or may need to be avoided.
     - **ERROR**: errors in programs.
     - **CRITICAL**: blocking issues or immediate attention issues.
-* **ERROR and CRITICAL** levels should be mitigated and informed.
 * `logger` is used for naming single logger object. Use `NAME_logger` name for more than one logger when required.
 * It is singleton and single threaded by default for given name of the logger. Can be [non-blocking](https://docs.python.org/3/howto/logging-cookbook.html#dealing-with-handlers-that-block) if required.
-* [Logging Cookbook](https://docs.python.org/3/howto/logging-cookbook.html) for reference.
-* Always use `exception` method rather than `error` method of `logger` object to log traceback when catching exceptions.
+* See [Logging Cookbook](https://docs.python.org/3/howto/logging-cookbook.html) for reference.
+* **ERROR and CRITICAL** levels should be mitigated and informed.
+    - Always use `exception` method rather than `error` method of `logger` object to log traceback when catching exceptions.

docs/python/project_structure.md

@@ -4,7 +4,7 @@ title: Project Structure and Templates
 sidebar_label: Project Structure
 ---
 
-### The following folder structure should be used for projects:
+#### The following folder structure should be used for projects:
 
 
 * :file_folder: Project Root:
@@ -24,13 +24,13 @@ sidebar_label: Project Structure
     - :memo: README (can be `md` or `rst`)
     - Other configurable from third parties (**OPTIONAL**) such as tox.ini
 
-
+:::note
 + **There can be cases where MVC folder structure as well as framework related folder structure can be used.** 
-    - The framework recommended structure shoudl be followed in such case.
-+ The OOP style cases of class as filename structue is not always necessary but can be used if needed.
-
+    - The framework recommended structure should be followed in such case.
++ The OOP style cases of class as filename structue is not always necessary or recommended but can be used if needed.
+:::
 
 
-##### Template Generation: 
+### Project Template Generation Tool 
 * Look into [cookiecutter](https://cookiecutter.readthedocs.io/en/1.7.2/) tool for template generation.
     - [List of templates for cookiecutter](http://cookiecutter-templates.sebastianruml.name/)

docs/python/testing.md

@@ -13,7 +13,7 @@ sidebar_label: Testing
 * `tox` and `nox` are vey good tools especially for CI and multiple version tests.
 * `hypothesis` and `mock` can be used for faking data and property testing.
 * Testing should be broken to `unit` as well as `functional`.
-* Use `coverage` to alert yourself of test coverage.
+* Use `coverage` to alert yourself of test coverage. Keep a target of **80 % - 90 %** coverage if **100%** is not achieved.  
 * Only test the changes you made or functionality you added when testing a codebase of well known frameworks.
 * `selenium` as well as `webtest` can be used for web based API testing.
 * `jsonschema` and `genson` like tool can be used for JSON validity.