Merge pull request #74 from timvink/73_filter_tables

Additional pandas reader macros

Author: timvink

README.md

@@ -40,8 +40,8 @@ In your markdown files you can now use:
 
 Where the path is relative to the location of your project's `mkdocs.yml` file, _or_ your project's `docs/` directory, _or_ the location of your markdown source file (all 3 possible locations will be searched, in that order).
 
-- There are [readers](https://timvink.github.io/mkdocs-table-reader-plugin/readers/) for `.csv`, `.fwf`, `.json`, `.xls`, `.xlsx`, `.yaml`, `.feather` and `.tsv` files. There is also the `read_raw()` reader that will allow you to insert tables (or other content) already in markdown format.
-- `table-reader` is compatible with [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/), which means you can [dynamically insert tables using jinja2 syntax](https://timvink.github.io/mkdocs-table-reader-plugin/howto/use_jinja2/).
+- There are [readers](https://timvink.github.io/mkdocs-table-reader-plugin/readers/) available for many common table formats, like `.csv`, `.fwf`, `.json`, `.xls`, `.xlsx`, `.yaml`, `.feather` and `.tsv`. There is also the `read_raw()` reader that will allow you to insert tables (or other content) already in markdown format.
+- `table-reader` is compatible with [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/). This enables further automation like filtering tables or inserting directories of tables. See the documentation on [compatibility with macros plugin](howto/use_jinja2.md) for more examples.
 
 ## Documentation and how-to guides
 

docs/howto/alternatives.md

@@ -39,7 +39,7 @@ Downsides:
 
 ## Execute python during build
 
-You could also choose to insert the markdown for tables dynamically, using packages like [markdown-exec]() or [mkdocs-macros-plugin](https://mkdocs-macros-plugin.readthedocs.io/).
+You could also choose to insert the markdown for tables dynamically, using packages like [markdown-exec](https://pypi.org/project/markdown-exec/).
 
 For example:
 

docs/howto/customize_tables.md

@@ -12,8 +12,10 @@ df = pd.read_csv('path_to_table.csv')
 df.to_markdown(index=False, tablefmt='pipe')
 ```
 
-Any keyword arguments you give to <code>\{\{ read_csv('path_to_your_table.csv') \}\}</code> will be matched and passed the corresponding [pandas.read_csv()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) and/or 
+{% raw %}
+Any keyword arguments you give to `{{ read_csv('path_to_your_table.csv') }}` will be matched and passed the corresponding [pandas.read_csv()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) and/or 
 [.to_markdown()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html) functions. 
+{% endraw %}
 
 Pandas's `.to_markdown()` uses the [tabulate](https://pypi.org/project/tabulate/) package and any keyword arguments that are passed to it. Tabulate in turn offers many customization options, see [library usage](https://github.com/astanin/python-tabulate#library-usage). 
 
@@ -23,21 +25,33 @@ Text columns will be aligned to the left [by default](https://github.com/astanin
 
 === ":arrow_left: left"
 
-    <code>\{\{ read_csv('tables/basic_table.csv', colalign=("left",)) \}\}</code>
-
+    {% raw %}
+    ```
     {{ read_csv('tables/basic_table.csv', colalign=("left",)) }}
+    ```
+    {% endraw %}
 
-=== ":left_right_arrow: center"
+    {{ read_csv('tables/basic_table.csv', colalign=("left",)) | add_indentation(spaces=4) }}
 
-    <code>\{\{ read_csv('tables/basic_table.csv', colalign=("center",)) \}\}</code>
+=== ":left_right_arrow: center"
 
+    {% raw %}
+    ```
     {{ read_csv('tables/basic_table.csv', colalign=("center",)) }}
+    ```
+    {% endraw %}
 
-=== ":arrow_right: right"
+    {{ read_csv('tables/basic_table.csv', colalign=("center",)) | add_indentation(spaces=4) }}
 
-    <code>\{\{ read_csv('tables/basic_table.csv', colalign=("right",)) \}\}</code>
+=== ":arrow_right: right"
 
+    {% raw %}
+    ```
     {{ read_csv('tables/basic_table.csv', colalign=("right",)) }}
+    ```
+    {% endraw %}
+
+    {{ read_csv('tables/basic_table.csv', colalign=("right",)) | add_indentation(spaces=4) }}
 
 ## Sortable tables
 
@@ -47,21 +61,33 @@ If you use [mkdocs-material](https://squidfunk.github.io/mkdocs-material), you c
 
 You can use [tabulate](https://pypi.org/project/tabulate/)'s [number formatting](https://github.com/astanin/python-tabulate#number-formatting). Example:
 
-=== ":zero:"
-
-    <code>\{\{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".0f") \}\}</code>
+=== "zero points"
 
+    {% raw %}
+    ```
     {{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".0f") }}
+    ```
+    {% endraw %}
 
-=== ":one:"
+    {{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".0f") | add_indentation(spaces=4) }}
 
-    <code>\{\{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".1f") \}\}</code>
+=== "one points"
 
+    {% raw %}
+    ```
     {{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".1f") }}
+    ```
+    {% endraw %}
 
-=== ":two:"
+    {{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".1f") | add_indentation(spaces=4) }}
 
-    <code>\{\{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".2f") \}\}</code>
+=== "two points"
 
+    {% raw %}
+    ```
     {{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".2f") }}
+    ```
+    {% endraw %}
+
+    {{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".2f") | add_indentation(spaces=4) }}
 

docs/howto/preprocess_tables.md

@@ -1,5 +1,7 @@
 # Preprocess input tables
 
+{% raw %}
+
 `mkdocs>=1.4` supports [hooks](https://www.mkdocs.org/user-guide/configuration/#hooks), which enable you to run python scripts on `mkdocs serve` or `mkdocs build`.
 
 Here are some example of workflows that use hooks and the `table-reader` plugin:
@@ -26,7 +28,7 @@ Here are some example of workflows that use hooks and the `table-reader` plugin:
 
     <code>
     My table:
-    \{\{ read_csv("docs/assets/output_table.csv") \}\}
+    {{ read_csv("docs/assets/output_table.csv") }}
     </code>
 
 === "mkdocs.yml"
@@ -74,7 +76,7 @@ Here are some example of workflows that use hooks and the `table-reader` plugin:
 
     <code>
     My table:
-    \{\{ read_csv("docs/assets/nyc_data.csv") \}\}
+    {{ read_csv("docs/assets/nyc_data.csv") }}
     </code>
 
 === "mkdocs.yml"
@@ -101,3 +103,5 @@ Here are some example of workflows that use hooks and the `table-reader` plugin:
     ```
 
 Note that during development when you use `mkdocs serve` and autoreload, you might not want to run this hook every time you make a change. You could use an environment variable inside your hook, for example something like `if os.environ['disable_hook'] == 1: return None`.
+
+{% endraw %}

docs/howto/project_structure.md

@@ -1,4 +1,5 @@
 # Choose a project structure
+{% raw %}
 
 You have different possible strategies to store and load your tables. This guide gives some examples.
 
@@ -23,14 +24,14 @@ If you only want to include an occasional table in a specific markdown file, jus
     ```md
     Here is the table:
 
-    \{\{ read_csv("another_table.csv") \}\}
+    {{ read_csv("another_table.csv") }}
     ```
 
 In `page.md`, to read `another_table.csv`, you can choose to use:
 
-- <code>\{\{ read_csv("docs/folder/another_table.csv") \}\}</code> (Path relative to mkdocs.yml)
-- <code>\{\{ read_csv("folder/another_table.csv") \}\}</code> (Path relative to docs/ directory)
-- <code>\{\{ read_csv("another_table.csv") \}\}</code> (Path relative to page source file)
+- `{{ read_csv("docs/folder/another_table.csv") }}` (Path relative to mkdocs.yml)
+- `{{ read_csv("folder/another_table.csv") }}` (Path relative to docs/ directory)
+- `{{ read_csv("another_table.csv") }}` (Path relative to page source file)
 
 ## Re-using tables across markdown files
 
@@ -54,7 +55,8 @@ Given the following project structure:
 
 In `page.md`, to read `another_table.csv`, you can choose to use:
 
-- <code>\{\{ read_csv("docs/assets/tables/another_table.csv") \}\}</code> (Path relative to mkdocs.yml)
-- <code>\{\{ read_csv("assets/tables/another_table.csv") \}\}</code> (Path relative to docs/ directory)
-- <code>\{\{ read_csv("../assets/tables/another_table.csv") \}\}</code> (Path relative to page source file _(note that `..` stands for "one directory up")_)
+- `{{ read_csv("docs/assets/tables/another_table.csv") }}` (Path relative to mkdocs.yml)
+- `{{ read_csv("assets/tables/another_table.csv") }}` (Path relative to docs/ directory)
+- `{{ read_csv("../assets/tables/another_table.csv") }}` (Path relative to page source file _(note that `..` stands for "one directory up")_)
 
+{% endraw %}

docs/howto/use_jinja2.md

@@ -1,4 +1,6 @@
-# Use jinja2 for automation
+# Compatibility with mkdocs-macros-plugin to enable further automation
+
+{% raw %}
 
 `table-reader` supports [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/), which enables you to use jinja2 syntax inside markdown files (among other things).
 
@@ -10,26 +12,19 @@ plugins:
     - table-reader
 ```
 
-Now you can do cool things like:
-
-## Dynamically load a list of tables
+Everything will work as before, _except_ indentation will not be retrained. This means components that rely on indentation (like [Admonitions](https://squidfunk.github.io/mkdocs-material/reference/admonitions/) and [Content tabs](https://squidfunk.github.io/mkdocs-material/reference/content-tabs/#usage)) will break.
 
-```markdown
-# index.md
+The solution is to use the custom _filter_ `add_indendation` (a filter added to `macros` by `table-reader` plugin, see the [readers](../readers.md)). For example:
 
-{% set table_names = ["basic_table.csv","basic_table2.csv"] %}
-{% for table_name in table_names %}
+```jinja
+!!! note "This is a note"
 
-{ { read_csv(table_name) }}
-
-{% endfor %}
+    {{ read_csv("basic_table.csv") | add_indentation(spaces=4) }}
 ```
 
-## Insert tables into content tabs
-
-If you inserted content has multiple lines, then indentation will be not be retained beyond the first line. This means things like [content tabs](https://squidfunk.github.io/mkdocs-material/reference/content-tabs/#usage) will not work as expected.
+The upside is you now have much more control. A couple of example use cases:
 
-To fix that, you can use the custom _filter_ `add_indendation` (a filter add to `macros` by `table-reader` plugin). For example:
+## Dynamically load a specified list of tables into tabs
 
 === "index.md"
 
@@ -39,7 +34,7 @@ To fix that, you can use the custom _filter_ `add_indendation` (a filter add to
 
     === "{{ table_name }}"
 
-        { { read_csv(table_name) | add_indentation(spaces=4) }}
+        {{ read_csv(table_name) | add_indentation(spaces=4) }}
 
     {% endfor %}
     ```
@@ -64,11 +59,6 @@ To fix that, you can use the custom _filter_ `add_indendation` (a filter add to
             alternate_style: true
     ``` 
 
-!!! note "Note the space in { {"
-
-    To avoid the tables being inserted into the code example, we replaced `{{` with `{ {`.
-    If you copy this example, make sure to fix.
-
 
 ## Recursively insert an entire directory of tables
 
@@ -100,12 +90,19 @@ Now you could do something like:
 
 {% for table_name in listdir('docs/assets/my_tables") %}
 
-{ { read_csv(table_name) }}
+{{ read_csv(table_name) }}
 
 {% endfor %}
 ```
 
-!!! note "Note the space in { {"
+## Filter a table before inserting it
+
+When you enable the `macros` plugin in your `mkdocs.yml`, `table-reader` will add additional _macros_ and _filters_ (see the [readers overview](../readers.md)).
+
+You can use the `pd_<reader>` variants to read a file to a pandas dataframe. Then you can use the pandas methods like [`.query()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html). For example:
+
+```
+{{ pd_read_csv("numeric_table.csv").query("column_name >= 3") | convert_to_md_table }}
+```
 
-    To avoid the tables being inserted into the code example, we replaced `{{` with `{ {`.
-    If you copy this example, make sure to fix.
+{% endraw %}

docs/options.md

@@ -4,6 +4,7 @@ hide:
 ---
 
 # Options
+{% raw %}
 
 You can customize the plugin by setting options in `mkdocs.yml`. For example:
 
@@ -24,7 +25,7 @@ Default is `.`. Set a default path to the searched directories in order to short
 
 Given a file path, `table-reader` will search for that file relative to your your project's `mkdocs.yml` and relative to your `docs/` folder. If you use a folder for all your table files you can shorten the path specification by setting the `data_path`.
 
-For example, if your table is located in `docs/assets/tables/basic_table.csv`, you can set `data_path` to `docs/assets/tables/`. Then you will be able to use <code>\{\{ read_csv("basic_table.csv") \}\}</code> instead of <code>\{\{ read_csv("docs/assets/tables/basic_table.csv") \}\}</code> inside any markdown page.
+For example, if your table is located in `docs/assets/tables/basic_table.csv`, you can set `data_path` to `docs/assets/tables/`. Then you will be able to use `{{ read_csv("basic_table.csv") }}` instead of `{{ read_csv("docs/assets/tables/basic_table.csv") }}` inside any markdown page.
 
 !!! info
 
@@ -38,7 +39,7 @@ Default: `False`. When enabled, if a filepath is not found, the plugin will rais
 
 ## `select_readers`
 
-Default: Selects all available readers. Specify a list of readers to improve documentation build times for very large sites.
+Default: Selects all available readers. Specify a list of readers to improve documentation build times for very large sites. This option is ignored when you use this plugin with `mkdocs-macros-plugin` ([read more](howto/use_jinja2.md))
 
 ## `enabled`
 
@@ -57,4 +58,6 @@ Which enables you to disable the plugin locally using:
 ```bash
 export ENABLED_TABLE_READER=false
 mkdocs serve
-```
+```
+
+{% endraw %}