The python "standard" way to (HTML) render a structured text file

[aborruso]

Description

The python "standard" way to (HTML) render a structured text file

I know, there is no "standard" way. But perhaps the most common way to start is to create a pandas dataframe and then render it.

If I have in example this CSV input file

Cat,Field 1,Field 2,Field 3
Cat,Field 1,Field 2,Field 3
A,Lorem,1236,2535.26
B,Ipsum,50478,235.89
C,Solat,2521,5789.56569

I can write this

```{python}
import pandas as pd
df = pd.read_csv('./resources/gt_input.csv')

df
```

to get in quarto this standard output

	Cat	Field 1	Field 2	Field 3
0	A	Lorem	1236	2535.26000
1	B	Ipsum	50478	235.89000
2	C	Solat	2521	5789.56569

What don't I like? Quarto is a really impressive scientific and technical publishing system. I know that even these are not general rules, but in my opinion there should be a way to have a default output that:

1. do not print index column, because it's not in the data, I will mainly use it for processing and not rendering;
2. always insert the thousand separator;
3. align the numeric columns to the right;
4. set the number of decimals to two.

And if I am doing a scientific/journalistic web publication in Italian, things get a little worse, because there is no direct way to say "we are in Italy, use Italian conventions for numbers, and use the , as decimal separator and . as thousand separator".

A way to do it, without using some other python libraries, is to write some code like this

import pandas as pd

# Function for formatting numbers in Italian format
def italian_format(x):
    if isinstance(x, float):
        return "{:,.2f}".format(x).replace(",", "X").replace(".", ",").replace("X", ".")
    elif isinstance(x, int):
        return "{:,}".format(x).replace(",", ".")
    else:
        return x

# To check if column is numeric
def is_numeric_column(col):
    return pd.api.types.is_numeric_dtype(col)

# To have Italian format for numbers and right alignment
def apply_italian_style(df):
    styled_df = df.style.format(italian_format).hide(axis="index")
    styled_df = styled_df.apply(lambda x: ["text-align: right" if is_numeric_column(df[x.name]) else "" for i in x], axis=0)
    return styled_df

# sample dataframe
df = pd.read_csv('./resources/gt_input.csv')

styled_df = apply_italian_style(df)
styled_df

to get

Cat	Field 1	Field 2	Field 3
A	Lorem	1.236	2.535,26
B	Ipsum	50.478	235,89
C	Solat	2.521	5.789,57

I love the observable "default" output. If I write

```{ojs}
//| code-fold: true
data = FileAttachment("resources/gt_input.csv").csv({ typed: true })
Inputs.table
```

I get

Cat	Field 1	Field 2	Field 3
A	Lorem	1,236	2,535.26
B	Ipsum	50,478	235.89
C	Solat	2,521	5,789.566

Some notes about this default output:

1. I have no index column in the output;
2. I have the thousand separator;
3. the numeric columns are right-aligned;
4. I have the number of decimals set to 3. Which is not 2, but is better than reading 5789.56569.

Moreover if I want Italian localization I can modify the code in this way

```{ojs}
//| code-fold: true
data = FileAttachment("resources/gt_input.csv").csv({ typed: true })
Inputs.table(data,{ locale: "it-IT" })
```

to get , as decimal separator and . as thousand separator.

Cat	Field 1	Field 2	Field 3
A	Lorem	1.236	2.535,26
B	Ipsum	50.478	235,89
C	Solat	2.521	5.789,566

I know that a great way to have it will be to use great-tables and this kind of code

```{python}
#| code-fold: true
from great_tables import GT, md, html
import pandas as pd
df = pd.read_csv('./resources/gt_input.csv')

GT(df,locale="it-IT")
```

but right now I have some problems with it, using locale parameter and with the default table rendering in Quarto 1.4.

I have been long, because it seems to me to be a topic of general interest, and because this long introduction was necessary to arrive at my proposal/reflection.

I know that the output here depends on pandas, but I also know that here quarto is the engine that will generate the scientific report.
I think there should be a possibility to tell to quarto: "use the standard way of rendering a table, applying the number formatting used in this country".

Could be additional yaml parameters:

---
format:
  html:
    toc: false
    page-layout: custom
    table:
      default-style: true
      locale: "it-IT"
---

Or without having a "standard" to define these individually

---
format:
  html:
    toc: false
    page-layout: custom
    table:
      thousand-separator: "."
      decimal-separator: ","
      number-of-decimals: 2
---

Clearly there must also be the possibility of not only defining it globally for the entire document, but also differently for each table in the document. But that is not the most important thing for now.

What do you think? Have I just written nonsense and is there already a ready-made way to do this?

I have a small working example in this page of this repo.

Thank you

[cscheid]

What don't I like? Quarto is a really impressive scientific and technical publishing system. I know that even these are not general rules, but...

The fixes you're asking for come from Pandas's styling of data frames. That would be the right place to fix them.

[aborruso]

The fixes you're asking for come from Pandas's styling of data frames. That would be the right place to fix them.

@cscheid thank you for your quick response and the outstanding things you do for this product.

I know it depends on "pandas" and I wrote it in my message. But I tried to develop another line of reasoning: don't you think it would be useful to have in the output of an html table generated by quarto some sort of standard way of representing it and being able to use localization?

Because now it is a bit inconvenient to do it in python. In R and observable everything is more ready for these goals.
And I think there is not in pandas a way to set locale: "it-IT".

But I understand your point.

[mcanouil]

Quarto cannot account for all possible Python modules, Julia libraries, R packages, etc.
Those libraries need to provide the "right" output or at least options.
Even with the knitr/rmarkdown ecosystem in R, I am 100% certain I can find an R package that does not do the right thing.

What you are asking is nearly impossible to achieve.
What Quarto tries to do is to provide an unified approach to handle markdown/html tables.
So the user is responsible for the code that creates the outputs using whatever language better suits his/her goals.

As far as I know Quarto does not (and should not) modify what (content) has been emitted by a code cell.
Seems pretty bad to remove columns from a table.
Quarto style output and that's it.
The content is up to the user as I said.

but right now I have some problems with it, using locale parameter and with the default table rendering in Quarto 1.4.

You can disable Quarto table processing as documented in https://quarto.org/ exactly for this reason and allow users to do whatever they want.