visitors with multiple columns

[useongkim]

Hello Hossein,

Thank you for the nice work.

I am playing with the dataframe and the visitors. I have a simple question.

How to apply the visitor for multiple columns? Here is a concrete example. (It seems the markdown has some issue with the header inclusions and main function syntax. So I excluded them.)

df is a three-column data frame. I can use the SumVisitor to calculate the sum of an individual column. But how to calculate the sum of multiple columns at once without calling the SumVisitor multiple times? For example, what can I do If I want to calculate the sum of all the elements in Treatment1 and Treatment2?

auto df = hmdf::StdDataFrame<int>{};

auto idx = std::vector<int>(5);
std::iota(idx.begin(), idx.end(), 0);

auto col1 = std::vector<double>{ 8, 9, 6, 8, 5 };
auto col2 = std::vector<double>{ 5, 4, 7, 6, 6 };
auto col3 = std::vector<double>{ 9,3,2,4, std::numeric_limits<double>::quiet_NaN() };

df.load_index(std::move(idx));
df.load_column("Treatment1", std::move(col1));
df.load_column("Treatment2", std::move(col2));
df.load_column("Treatment3", std::move(col3));

auto sum_v = hmdf::SumVisitor<double>{};
df.visit<double>("Treatment1", sum_v);

return 0;

Thank you.

Useong Kim

[hosseinmoein]

Hi,

I don't have that kind of interface, because I never personally had use case for it. But there are a couple of things you can do:

1. Write your own visitor that does that. It is very easy to write your own visitor. I specifically designed the visitor interface so it would be reasonably easy for end users to write their own algorithm. See the visitor section in docs (https://hosseinmoein.github.io/DataFrame/docs/HTML/DataFrame.html)
2. Use the async version of the visitor interface. Call the 3 sums with single_act_visit_async() which will run in parallel (https://hosseinmoein.github.io/DataFrame/docs/HTML/single_act_visit.html).

Best,
HM

[useongkim]

Hi HM,

Thank you for the reply. They are helpful.

Let me ask an additional question.

Does a simple constant offset for single or multiple columns require a custom visitor implementation? Suppose df is a standard data frame and all the column types support + operation with a numeric type. For example, I hope the following code works
auto df2 = df + 1; // All the elements of df2 get the value increased by 1.
It seems this can be done by a custom visitor as well. But It would be definitely more convenient if this kind of operations is supported out of box.

Thank you.

Useong Kim

[hosseinmoein]

That's a good question. This cannot be done with the syntax that you wrote above. That's because, as I mention in the docs, to access any column in a DataFrame, either explicitly or implicitly, you have to know its type at compile time (because C++ is staticly typed).
But it can be done this way https://hosseinmoein.github.io/DataFrame/docs/HTML/global_operators.html. One of the items in my todo list is to add a scaler operation to that interface.

[useongkim]

Thank you for the answer. I am trying to compare this data frame with pandas for my own toy cases so that I could use it as a part of a production tool. My impression is that it is fast. But it took me a little time to learn the usage. There are things trivial in pandas but not in the dataframe. I hope the usage of data frame would be as easy as that of pandas. I am looking forward to see the scalar operations.

[hosseinmoein]

But it took me a little time to learn the usage. There are things trivial in pandas but not in the dataframe.

That is not surprising at all. That's because the difference between the nature of the two languages. Python is a dynamically typed language. That means anything could be of any type in any container. That makes the syntax very free-wheeling and makes it possible to be elegant. But it makes the execution slow, in some circumstances extremely slow. C++ on the other hand is a statically typed language. That means all types must be known at compile time. That puts a lot of restrictions on the syntax free-wheeling. But it makes the execution fast.

In other words, there is no free lunch :-)