feat: using result sets outside the Do-block callbacks to process large data sets #1862
Description
Feature Request
Describe the Feature Request
Currently it's only possible to process the result of a query just within the Do-block:
err := ydbDriver.Query().Do(parentCtx, func(ctx context.Context, session query.Session) error {
var queryOpts []query.ExecuteOption
result, _ := session.Query(ctx, queryText, queryOpts...)
resultSet, _ := result.NextResultSet(ctx)
// ...
return nil
}You cannot pass the resultSet from this callback via channel to another goroutine for the further processing, because the internal context within SDK will be canceled as soon as the runtime leaves the Do-block (this change was introduced in v3.113.1, so fq-connector-go wasn't really affected before this release).
But this is not very convenient, because sometimes you just can't put all the data processing within a callback passed to the SDK call.
Describe Preferred Solution
All the popular relational database SDKs nowadays (PostgreSQL, ClickHouse, MySQL, MS SQL Server) provide you an iterable objects containing query results which one can use in any way he wants. The lifetime of these result iterators is much larger then the lifetime of a callbacks passed into Do-block in Go YDB SDK.
Related Code
Here is a minimal working example that shows, how the data can be processed outside the SDK callback.
The results of work for this application depend on the SDK version.
For YDB Go SDK <= 3.113.0:
./dump_tablet_id_data -token=$(yc --profile prod-sandbox-fed-user iam create-token) -database "/ydb_experimental_vla/b1gtl2kg13him37quoo6/etn8i0ga1e2grsvoqvv9" -resource-pool="default" -endpoint="lb.etn8i0ga1e2grsvoqvv9.ydb.mdb.yandexcloud.net:2135" -tablet-id="72075186224064770"
2025-09-17T13:02:07.527+0300 INFO dump_tablet_id_data/main.go:157 will use secure TLS connections
2025-09-17T13:02:07.527+0300 INFO dump_tablet_id_data/main.go:173 connecting to YDB {"dsn": "grpcs://lb.etn8i0ga1e2grsvoqvv9.ydb.mdb.yandexcloud.net:2135/ydb_experimental_vla/b1gtl2kg13him37quoo6/etn8i0ga1e2grsvoqvv9", "auth": "IAM token", "tls": true}
2025-09-17T13:02:07.534+0300 INFO dump_tablet_id_data/main.go:312 starting query execution {"tablet_id": "72075186224064770", "table": "tpch/s100/lineitem", "tablet_id": "72075186224064770"}
2025-09-17T13:02:07.534+0300 INFO dump_tablet_id_data/main.go:191 executing query {"tablet_id": "72075186224064770", "query": "SELECT 0 FROM `tpch/s100/lineitem` WITH TabletId='72075186224064770'", "table": "tpch/s100/lineitem"}
2025-09-17T13:02:12.975+0300 INFO dump_tablet_id_data/main.go:337 query completed {"tablet_id": "72075186224064770", "total_rows": 9377240, "tablet_id": "72075186224064770", "table": "tpch/s100/lineitem", "total_duration": 5.441455805, "rows_per_second": 1723296.1795598005}
Total rows: 9377240
For YDB Go SDK >= v3.113.1:
./dump_tablet_id_data -token=$(yc --profile prod-sandbox-fed-user iam create-token) -database "/ydb_experimental_vla/b1gtl2kg13him37quoo6/etn8i0ga1e2grsvoqvv9" -resource-pool="default" -endpoint="lb.etn8i0ga1e2grsvoqvv9.ydb.mdb.yandexcloud.net:2135" -tablet-id="72075186224064770"
2025-09-17T13:04:18.400+0300 INFO dump_tablet_id_data/main.go:157 will use secure TLS connections
2025-09-17T13:04:18.400+0300 INFO dump_tablet_id_data/main.go:173 connecting to YDB {"dsn": "grpcs://lb.etn8i0ga1e2grsvoqvv9.ydb.mdb.yandexcloud.net:2135/ydb_experimental_vla/b1gtl2kg13him37quoo6/etn8i0ga1e2grsvoqvv9", "auth": "IAM token", "tls": true}
2025-09-17T13:04:18.413+0300 INFO dump_tablet_id_data/main.go:312 starting query execution {"tablet_id": "72075186224064770", "table": "tpch/s100/lineitem", "tablet_id": "72075186224064770"}
2025-09-17T13:04:18.414+0300 INFO dump_tablet_id_data/main.go:191 executing query {"tablet_id": "72075186224064770", "query": "SELECT 0 FROM `tpch/s100/lineitem` WITH TabletId='72075186224064770'", "table": "tpch/s100/lineitem"}
2025-09-17T13:04:18.502+0300 INFO dump_tablet_id_data/main.go:259 processing result set {"tablet_id": "72075186224064770", "elapsed_time": 0.00000059}
2025-09-17T13:04:18.509+0300 FATAL dump_tablet_id_data/main.go:333 failed to process rows {"tablet_id": "72075186224064770", "error": "rows error: next row: transport/Canceled (code = 1, source error = \"rpc error: code = Canceled desc = context canceled\", address: \"lb.etn8i0ga1e2grsvoqvv9.ydb.mdb.yandexcloud.net:2135\") at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:171)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:232)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*streamResult).nextPart(result.go:214)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*streamResult).nextPartFunc.func1(result.go:331)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*resultSet).nextRow(result_set.go:176)`"}