Ibis row-wise operations

[stb-ssk]

Hello!

I am writing to ask if there are any ways to check values amongst each row in an ibis table.

For example, given an ibis table, is there a way to check all the column values for an each row in the table.

Any row-wise operations I can use?

Thanks

[gforsyth]

Hey @stb-ssk ! Could you give a more concrete example of the kind of thing you're looking for?

[stb-ssk]

Sure @gforsyth ! So for each row in an ibis table, I want to see if a value "1", exists across all the columns for that row. Or just return all the rows in the table, where "1" is not in all the columns for that row.

Essentially, I am looking for something similar .any() or .all() but for a row wise search.

[tkbadamdorj]

I was looking into this yesterday!

Something like the following should work

import ibis.selectors as s
from ibis import _

t.filter(s.if_any(s.all(), lambda x: x == "1")) 

# or using the deferred API  
t.filter(s.if_any(s.all(), _ == "1"))

Try it out and let me know if that works for you :D

[stb-ssk]

hi @tkbadamdorj! It does seem to work for smaller data tables, but I'm dealing with a table with over 200K rows. Any tips for doing a row wise search with such a large dataset.

[cpcloud]

The number of rows shouldn't matter here (or least it won't be less efficient because of Ibis) because this will all be executed as SQL directly against the engine.

Did you try running it on your larger dataset?

[stb-ssk]

I did. But when I print the head of the new table or even try to materialize the results with .execute(), it exhausts the time limit of my db.