perf: Replace expensive len() call with PandasBatches.total_rows in anywidget TableWidget

[shuoweil]

perf: Replace expensive len() call with PandasBatches.total_rows in anywidget TableWidget

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[tswast]

Please also update the benchmarks to use the total_rows parameter.

[shuoweil]

Please also update the benchmarks to use the total_rows parameter.

Let's use a separate PR for this request. #1949

[shuoweil]

Per checking, the failed testcase multimodal_test.py::test_multimodal_dataframe and /tmpfs/src/github/python-bigquery-dataframes/notebooks/location/regionalized.ipynb are not introduced by my change. @tswast

[tswast]

I think this would cause the query to execute twice. Please revert. Instead, it should match as closely as possible to what happens in TableWidget.

Suggested change

	execute_result = df._block.session._executor.execute(df._block.expr, ordered=True)
	execute_result.total_rows or 0
	next(iter(df.to_pandas_batches(page_size=PAGE_SIZE)))
	# Get number of rows (to calculate number of pages) and the first page.
	batches = df.to_pandas_batches(page_size=PAGE_SIZE)
	first_page = next(iter(batches))
	assert first_page is not None
	total_rows = batches.total_rows
	assert total_rows is not None

[tswast]

I think we should bubble up these exceptions and show an error message, not hide them.

Please revert.

Suggested change

	google.api_core.exceptions.GoogleAPICallError,
	TypeError,
	ValueError,

Also, can you share what situations happened where these occurred? It may indicate a bug.

[tswast]

Please add an assertion that references batches.total_rows. We want to mimic TableWidget as closely as we can.

For example:

Suggested change

	next(iter(batches))
	assert batches.total_rows >= 0
	next(iter(batches))

Same for the other benchmarks.

[shuoweil]

This change will trigger a mypy error, to_pandas_batches() actually returns a PandasBatches object that has the total_rows attribute, but the type annotations show it as Iterable[pandas.DataFrame]. I can assert here after casting