Impute Missing Values in DataFrame Using Tabular Model

[efstathios-chatzikyriakidis]

Hi @avsolatorio,

I hope you're doing well. I have a question regarding the library and its capabilities for generating new data rows. Specifically, I'm interested in a method that allows us to fill in missing values in a DataFrame using a trained tabular model.

Use Case

Suppose we have a DataFrame like this:

A	B	C	D
?	?	?	D1
NA	?	?	?
A1	?	?	?
A6	?	?	D1

I would like the output to retain the same columns and rows, filling in only the ? values with generated data while preserving the existing non-? values. For example, the filled DataFrame might look like this:

A	B	C	D
A3	B6	C3	D1
NA	B2	C1	D6
A1	B66	C1	D9
A6	B3	C31	D1

Full Generation Scenario

Additionally, if we provide a DataFrame where all values are missing, like this:

A	B	C	D
?	?	?	?
?	?	?	?
?	?	?	?
?	?	?	?

The model should behave similarly to rtf_model.sample(), generating entirely new rows of data. Essentially, the model should be capable of both imputing missing values in an existing DataFrame and generating new rows from scratch if all values are missing. Importantly, NA should remain a valid option for generated data.

Handling Different Data Types

Lastly, I am curious about the best approach to handle ? values in numerical and datetime columns. How can we differentiate between ? and other types of missing data like NaN or NaT?

Thanks for your help!