Calling Gridify on a filtered set of data

[De-Novo-Research]

If I create a cursor, to filter only records of a TDataItem, can I think call gridify on that cursor? To only gridify the items of interest?

[davidberneda]

Yes it should work fine, there is a low level TDataCursor class, ToData function that returns the filtered and/or sorted data, as a normal TDataItem.

https://github.com/Steema/TeeBI/blob/master/sources/BI.DataSource.pas

Upper level there is TBIQuery component, that uses TDataCursor internally for the "where" and "order by" clauses. BIQuery1.Calculate returns a normal TDataItem, usable everywhere.

If I'm not wrong, both ways return a copy of the data, not the underlying source arrays.

[davidberneda]

BIQuery1 component has a design/runtime dialog to customize the Filter property without coding.

[De-Novo-Research]

Thank you. BTW, the component is amazing. Almost as useful as a pandas DataFrame. The main problem is that the dataframe has 10 billion lines of documentation on the web, and not as much for TeeBI. But amazing work!

[davidberneda]

Thank you a lot. Docs are far from being complete, the best attempt to document all features and how to put all together is at the wiki: https://github.com/Steema/TeeBI/wiki

And the demos try to showcase everything, but you're right the API reference lacks explanations and snippets, work-in-progress.

[davidberneda]

I've added a new test unit for TDataCursor class with small code snippets:

https://github.com/Steema/TeeBI/blob/master/tests/BI.Tests.DataCursor.pas

TDataCursor enables filtering, sorting any TDataItem. Easiest way to use it is Cursor.ToData function that returns a new TDataItem that is already filtered and/or sorted, etc.

If necessary, the cursor creates an index (an array of integers) that are the rows that pass the filter, in the optional order we want.

The TBIQuery component uses internally a TDataCursor.

The cursor Filter property is just a TExpression, that can be created in many ways, for example:

Cursor.Filter:=TDataFilter.FromString(Cursor.Data,'Country<>"Mexico"');

[De-Novo-Research]

Most of the time it does not matter but it is a shame to have to copy the data from the TDataCursor to use the sorted data outside a TBiQuery. In an idea world, many things that accept a .Data could accept a TDataCursor. Many things, like Charts, Grids etc do not need to know what is inside the Data.

[davidberneda]

This demo has examples for the basic concepts:
https://github.com/Steema/TeeBI/blob/master/docs/Starting%20Guide/Samples/BI_Unit.pas

[davidberneda]

Re: In an idea world, many things that accept a .Data could accept a TDataCursor.

TDataCursor derives from a base class called "TDataProvider", there are several classes that derive from it, and act as "providers" of data. Things like queries, importers from streams, cursors, etc.

TDataItem has a Provider property (optional), that is then used to "fill" the TDataItem structure and data arrays with content from the provider.

Charts grids etc could also accept a cursor (or base provider) as data source, but they would also create a TDataItem and fill it according the provider way of obtaining the data. I'm not sure if I understand the problem, for example

BIGrid1.Data := BIQuery1.Calculate;

This links the grid (or the chart or whatever), with the query, changing the query (filter "where", or order by sorting, etc) will recreate the query output, and refresh the grid (it should be done automatically).

That's the idea at the normal "high level" usage, internal notification lists to detect refreshing and destroying data to make things automatic.

[De-Novo-Research]

Again, this is more of a theoretical debate. If I am dealing with a dataset that has a million items, and then run a query on it, that returns 999,999 items, it seems based on the discussion above that eventually a dataItem will be created that has 999,999 rows, representing the queried data.
Same if I have a cursor, that eventually returns 999,999 items, I have to call Cursor.ToData to get the DataItem that is used for charting the Data.

In my way of thinking, there should be two levels (which I would do with interfaces, but never mind that for now)

TDataItemView - contains the read only methods of TDataItem
TDataItem =class(TDataItemView) that contains all the writing methods

Things that display the data should only need to accept a TDataItemView.

I could then make many Views into the same data, without duplicating the data. So I could have a TDataItemView descendent that contains an index list and another DataItemView. When someone request View.GetValue[x] on the 'outer view' it looks at its index list and returns InnerView.GetValue[indexList[x]]. Or you can create a view that only has certain columns from another view. The Cursor is then a view into the data with a sorted/filtered list but also can be displayed inside anything that accepts a TDataItemView.

Of course, views can be nested inside views at multiple levels and mabe you have a function to flatten them all into a real TDataItem if you want. But I think in the vast majority of use cases, this should allow you to do a lot of what you need with far less copying of the data around.

[davidberneda]

Same if I have a cursor, that eventually returns 999,999 items, I have to call Cursor.ToData to get the DataItem that is used for >>charting the Data.

TDataCursor is very low level, what I think its needed here is to use a TBIDataSet. This is like a normal TDataSet that uses a dataitem as the underlying data, has a filter and sorting index (it has an internal TDataCursor), and never copies the data.

I'll see how easy or not it is to add support for TDataCursor (or TDataItemView, which is more or less what a TDataCursor is), to all controls and other classes where a TDataItem is accepted.

The biggest issue might be internal notification about changes and destroys, ie, the views should respond to any change in the DataItem that might force to recreate the view (if possible)