ENH: Query function to have a star * for all columns

[dstone42]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

I have a data frame that has a large number of columns that have positive numeric values except the last 4 columns. There are some rows with missing data, but in this dataset, they denote that by using a very large negative number. My plan was to use the query function to filter the rows to only those that have values greater than or equal to zero in every numeric column.

Feature Description

I wish the query function would let me do something like this df.query('* >= 0') the star * representing all columns. Better yet, would be if I could use star but subtract the columns to be excluded from that like df.query('* -state -county >= 0').

Alternative Solutions

What I am going to do is loop through the columns of the data frame and query on each one using a variable and exclude those non-numeric from my list. Like this:

column_list.remove('state')
column_list.remove('county')

for column in column_list:
    df = df.query('@column >= 0')

Additional Context

No response

[Delengowski]

Do you have an idea how this would be implemented with the ast parser used to parse the query string? What if there's a column named *? Can it even parse it?

[dstone42]

Sorry for the delay. @Delengowski

I have no idea what's going on in the implementation of the parser for the query string, so I don't have any specific ideas for how this would be implemented, but I assumed by naming the function "query" it was a callback to database languages, so I thought this query function might have been modeled somewhat on database languages. Using * in database languages is common, so maybe you could look at the implementation in database languages to see how they parse this.

As far as having a column named *, the two options would be to either disallow having a column named , or to use an escape character such as *. Although, I don't know if slash would work as it's already a built in escape character in python, unless it was only a literal slash that counts to escape it, so a query to that column could look like '\ >= 0'. Although, I don't know if it would be worth your time to add escaping in this parsing if there aren't other special characters that would ever need to be escaped, so it's only purpose would be this one functionality.

If it's too difficult and nobody else is interested in this kind of functionality, you can just close this and decide not to implement it. I forgot what code I even wanted this implementation for as I used some other work around and moved on. I'm not super attached to it, but if I ever come around again to using the query function, it would be a nice option.