Skip to content

Latest commit

 

History

History
181 lines (169 loc) · 36 KB

CHANGELOG.md

File metadata and controls

181 lines (169 loc) · 36 KB

Unreleased

Functionality

  • None:Added support for encrypted exports to S3, allowing secure storage of exported data. This feature is enabled by setting the enable_encrypted_export feature flag in the cluster configuration. (Vasily Gerasimov)
  • 15186:Increased the query text limit size in system views from 4 KB to 10 KB. #15186 (spuchin)
  • 15693:Added a health check configuration that administrators can customize: the number of node restarts, tablets, the time difference between database dynodes, and timeout (by default, the maximum response time from healthcheck). Documentation is under construction. #15693 (Andrei Rykov)
  • 17394:Added counters to the spilling IO queue to track the number of waiting operations. Documentation is under construction #17599. #17394 (Filitov Mikhail)
  • 17362:Added support for Google Breakpad inside YDB. Now you can set a minidumps path using an environment variable. #17362 (Олег)
  • 17148:Extended federated query capabilities to support a new external data source Prometheus. Documentation is under construction YQ-4261 #17148 (Stepan)
  • 17007:Extended federated query capabilities to support a new external data source Apache Iceberg. Documentation is under construction YQ-4266 #17007 (Slusarenko Igor)
  • 16652:Added support for creating external data sources for iceberg tables. #16652 (Slusarenko Igor)
  • 16957:Added support for a new external data source (Redis) in federated queries. #16957 (Gleb Solomennikov)
  • 17095:Lowered the severity level for FAULTY pdisks. #17095 (Andrei Rykov)
  • 16989:Added information about MODIFY(ALTER) USER operations to audit_log: change a password (or hash), block a user, unblock a user. #16989 (flown4qqqq)
  • 16857:Added information about the processing lag of committed messages to DescribeConsumer. #16857 (Nikolay Shestakov)
  • 16834:Enabled pushdown of string types in Generic provider. #16834 (Vitaly Isaev)
  • 16076:Added automatic cleanup of temporary tables and directories created during S3 export operations. Previously, users had to manually remove temporary directories and tables that were created as part of the export pipeline. #16076 (stanislav_shchetinin)
  • 18731:Added monitoring counters with the constant value of 1 for nodes, VDisks, and PDisks. #18731 (Sergey Belyakov)
  • 18376:Added a heuristic that should prevent a tablet that can overload a node on its own from endlessly moving between nodes. #18376 (vporyadke)
  • 18297:Added basic functionality for cluster bridging. #18297 (Alexander Rutkovsky)
  • 18258:Increased the timeout on BS_QUEUE reestablishing session when many actors try to reconnect simultaneously. #18258 (Sergey Belyakov)
  • 18137:Added support for vector indexes in index-only searching (#17770). #18137 (Vitaliy Filippov)
  • 18214:Added support for importing topic configurations from S3. #18214 (stanislav_shchetinin)
  • 18138:Added support for exporting topic configurations to S3. #18138 (stanislav_shchetinin)
  • 18095:Added audit logging to track create tenant database actions within the console tablet. #18095 (flown4qqqq)
  • 18063:Added support for the new version format Year.Major.Minor.Patch.Hotfix (the old format was Year.Major.Minor.Hotfix). #18063 (Sergey Belyakov)
  • 17957:Added BS Controller settings in the cluster configuration. (Semyon Danilov)
  • 17952:Implemented storage of access resource ID for cluster information as the container_id user attribute in the root database. Users can check permissions on this resource to access cluster-related information. #17952 (Andrei Rykov)
  • 17804:Added CPU limit per process in CS for integration with Workload Manager. #17804 (Pisarenko Grigoriy)
  • 17751:Added Create Time, End Time, and Created By (user SID) tracking for BuildIndex operations. #17751 (Vitaliy Filippov)
  • 18683:Added support for creating compacted topics for Kafka API. #18683 (FloatingCrowbar)
  • 17663:Added optimizations for reading from Solomon. #17663 (Ivan Sukhov)
  • 17525:Implemented automatic reconfiguration of StateStorage (change the number of rings or the number of replicas in a ring) in case of a node failure or load changes without downtime of the entire group. #17525 (Evgenik2)
  • 17421:Added support for runtime results during script execution. #17421 (Pisarenko Grigoriy)
  • 19315:Added distconf cache for group info propagation. #19315 (Alexander Rutkovsky)
  • 19227:Added support for REGEXP pushdown operations on Utf8 columns during federated query execution. #19227 (Vitaly Isaev)
  • 19216:Added basic checks for group names when creating or renaming them: the names may consists of lower and upper case Latin characters, digits, underscores and hyphens. #19216 (yurikiselev)
  • 19111:Added support for defining the size of sliced portions on the zero layer. #19111 (Oleg Doronin)
  • 18988:Added support for pile identifiers in dynamic nodes within the 2-dc cluster topology. #18988 (Alexander Rutkovsky)
  • 18859:Added support for Board reconfiguration in StateStorage via DistConf. #18859 (Evgenik2)
  • 19134:Added the running status to script execution statuses (improved output for the get script execution operation). #19134 (Pisarenko Grigoriy)
  • 18444:Added support for a new external data source OpenSearch in federated queries. #18444 (Arslan Giniyatullin)
  • 19893:Implemented a new robust algorithm for HDRF scheduler that does not allow executing more tasks than the demand allows. #19893 (Ivan)
  • 19735:Added the Accept header to the oidc white list. #19735 (Andrei Rykov)
  • 19642:Added ColumnShard statistics for the update/delete operations, in rows and bytes. #19642 (Vladilen)
  • 19618:Re-implemented the KQP scheduler for CPU using the HDRF model. #19618 (Ivan)
  • 19600:Implemented cluster bootstrapping state detection in Healthcheck API for configuration v2 environments. #19600 (vporyadke)
  • 19580:Added support fo changing schema object limits (MAX_SHARDS, MAX_PATHS, ...) with YQL. (Daniil Demin)
  • 19567:Added support for a new Increment operation, only Integer types are supported. #19567 (r314-git)
  • 18333:Added audit logs for YMQ events in common ydb format. #18333 (flown4qqqq)
  • 20176:Added automatic creation and deletion of a service consumer for topic compaction. #20176 (FloatingCrowbar)
  • 19994:Added memory limits for column-oriented tables to the Memory Controller configuration. #19994 (Vladilen)
  • 19678:Added support for idempotent producers to the YDB implementation of the Apache Kafka protocol. #19678 (qyryq)
  • 19444:Introduced a new UnorderedData protocol for full synchronization, which allows taking a full index snapshot only once per full sync session. The new protocol will be used to send DoNotKeep flags from Phantom Flag Storage, which will be sent without relying on the linear order. For now the new protocol is disabled by default. #19444 (Sergey Belyakov)
  • 19337:Introduced two new Distributed Storage parameters — GroupSizeInUnits property of storage groups and SlotSizeInUnits of PDisks. Docs: #19364 #19337 (Yaroslav Dynnikov)
  • 18955:Added a new external data source to YDB topics. #18955 (Dmitry Kardymon)
  • 18544:Added shared metadata cache. Now all column shards on the same node share the same metadata cache by default. To disable the shared metadata cache, use the EnableSharedMetadataAccessorCache flag. The default cache size is 1 GB, this value can be changed in the SharedMetadataAccessorCacheSize parameter. #18544 (Iurii Kravchenko)
  • 19680:Implemented the select load type for the vector workload test. #19680 (azevaykin)
  • 20040:Enabled verbose memory limit by default in recipe #20040. (Ivan)
  • 20272:Enabled the following feature flags by default: EnableTopicAutopartitioningForCDC, EnableTopicAutopartitioningForReplication, EnableTopicTransfer. #20272 (Nikolay Shestakov)
  • 20253:Separated bulk statistics operations from non-bulk. #20253 (Vladilen)
  • 20738:Added statistics for the grouped limiter and caches. #20738 (Vladilen)
  • 21183:Added audit log events for YMQ events. #21183 (flown4qqqq)
  • 21010:Added more attributes to DSProxy and VDisk spans to make it easier to find them in the metrics explorer. #21010 (Sergey Belyakov)
  • 19684:Added a command to dstool to move pdisk from one node to another, if both nodes have access to the underlying drive. This way we can skip data synchronization, since new disk will just use data of the "old" disk #19684 (Semyon Danilov)
  • 21448:Added support for a user-defined Certificate Authority (CA) in asynchronous replication. #21448 (Ilnaz Nizametdinov)
  • 21420:Topics are now created with the number of partitions equal to datashards/16 instead of 1 partition. This improves table replication speed by avoiding delays from autopartitioning. #21420 (Nikolay Shestakov)
  • 21391:Implemented an optimized HashV2 hash function that improves data distribution and enhances query execution performance. This mode is enabled by setting the table_service_config: default_hash_shuffle_func_type: HASH_V2 parameter in the cluster configuration. #21391 (Ivan)
  • 20303:Add --iam-token-file argument to ydb-dstool. #20303 (kruall)
  • 22511:Added the ICB control to change ReadRequestsInFlightLimit via changing dynconfig. #22511 (kruall)
  • 21997:Enabled the new compute scheduler based on the HDRF model. #21997 (Ivan)
  • 25759:ChunkBaseLimit parameter is configured via PDiskConfig, which can be inconvenient to manage. This change adds ICB control to manipulate this parameter. When set, ICB option has priority over config. #25759 (Sergey Belyakov)
  • 25579:CMS manual request approval now empties request's actions so that consecutive CheckRequest requests will return ALLOW code #25579 (Semyon Danilov)
  • 25538:added basic monitoring tests and separate events file #25538 (Andrei Rykov)
  • 25458:Сейчас при автопартициронировании топиков учитывается скорость записи различными producer-ами: партиция делится не пополам, а стараемся разделить партицию таким образом, что бы producer-ы распределились по новым партициям равномерно с учетом скорости записи. #25458 (Nikolay Shestakov)
  • 25387:Change the audit logging logic from AllowedList checking to DenyList checking #25387 (Andrei Rykov)

Bug fixes

  • 16021:Fixed a rare error that led to a VERIFY error when replicating data. #10650 #16021 (Alexander Rutkovsky)
  • 16061:Fixed a bug in handling scan queries with predicates in column-oriented tables. #16061 (Semyon)
  • 16879:Fix crashes on scan queries with a predicate and a limit in column-oriented tables. #16878 #16879 (Semyon)
  • 17650:Introduced Add and IsEmpty operations to Roaring UDF. #17650 (jsjant)
  • 17537:The language version may affect the computation layer, so it should be passed via dq task. #17537 (Vitaly Stoyan)
  • 17360:Added a setting that specifies the initial stream offset by timestamp or duration. #17360 (Alexey Pozdniakov)
  • 17314:Added a progress stats period to the execute script request API (periodic statistics updates). Added dynamic updates for the query service configuration. #17314 (Pisarenko Grigoriy)
  • 16932:Added the MaxFaultyPDisksPerNode parameter to the CMS configuration, so that Sentinel could not exceed that parameter value. This may be useful when a node with a large number of disks goes down, but is expected to be started soon. In this case YDB does not move many disks at once (which might take long in case of HDDs). If MaxFaultyPDisksPerNode is 0, then there is no limit. #16932 (Semyon Danilov)
  • 17911:Added the list of dependent scheme objects (external tables, column tables) to the description of an external data source. #17911 (Daniil Demin)
  • 17711:Added the --breakpad-minidumps-path and --breakpad-minidumps-script arguments to the ydbd server command for initializing minidumps instead environment. #17711 (Олег)
  • 17695:YDB FQ: Added support for the pushdown optimization in LIKE expressions. #17695 (Vitaly Isaev)
  • 17420:Added data erasure request for PQ tablets #17420 (Andrey Molotkov)
  • 17920:Added new PDisk status to BSC, which allows to reassign VDisks from PDisk via SelfHeal without considering VDisk erroneous. Intended to be used instead of FAULTY status when node is taken to long term maintenance. #17920 (Sergey Belyakov)
  • 17864:Added file names and line numbers to logs. #17864 (Vladislav Stepanyuk)
  • 17687:Changed the method of determining the PDisk status to using the PDisk State from BSC info instead of the PDisk status in CMS to avoid ambiguity. It is already used in the healthcheck fallback logic and reflects the actual disk condition. It is planned to become the primary source for PDisk state in healthcheck. #17687 (Andrei Rykov)
  • 16016:Fixed rare node failures during read session balancing. ydb-platform#16017 #16016 (Nikolay Shestakov)
  • 16423:Changed behavior — SHOW CREATE TABLE now fails on views instead of producing wrong output. #16423 (Daniil Demin)
  • 16764:Fixed redirects from cluster endpoints (storage nodes) to database nodes, resolving inconsistent behavior where some system tables were not visible. #16763 #16764 (Alexey Efimov)
  • 18698:Fixed an issue where checks for enabled encryption were missing during the zero copy routine. This issue resulted in attempts to send unencrypted data via an XDC socket. ydb-platform#18546 #18698 (Daniil Cherednik)
  • 17230:Added more information to the error message that occurs during Generic::TPartition parsing. #17230 (Vitaly Isaev)
  • 17009:Fixed an issue with compiling a JOIN operation, in which one side was an empty constant. #17009 (Nikita Vasilev)
  • 16901:Changed the error response in the RateLimiter service from INTERNAL_ERROR to SCHEME_ERROR when using non-existent coordination nodes or resources. ydb-platform#16914 #16901 (Vasily Gerasimov)
  • 17606:Fixed an issue with the check for unknown StatusV2 in the VDisks healthcheck. (ydb-platform#17619) Fixed a forward compatibility issue. #17606 (Andrei Rykov)
  • 17556:Fixed an EntityTooSmall error in S3 export multipart upload. ydb-platform#16873 #17556 (Vasily Gerasimov)
  • 17468:Fixed an issue with outputting optional structs to resolve #17226. #17468 (Alexey Efimov)
  • 17429:ydb-platform#16176 #17429 (Nikolay Shestakov)
  • 17418:Previously replication didn't handle StatusFlags from PDisk properly and didn't stop when available space was running low. Because of this, space was consumed until VDisk and corresponding storage group became read-only. With this change, replication will postpone on YELLOW_STOP color flag and resume after a certain delay if enough space is available. #17418 (Sergey Belyakov)
  • 16208:YQ-4205 Fixed the issue with a query service configuration passing into SS. #16208 (Pisarenko Grigoriy)
  • 17872:Added missing #ifdef. MSG_ZEROCOPY related code works only on Linux. #17872 (Daniil Cherednik)
  • 17826:Changed the logic to start TGuardActor exclusively when ZC transfers are not completed. #17826 (Daniil Cherednik)
  • 17780:Added support for TEvYardControl::PDiskStop event processing in PDisk’s Error and Init states. #17780 (Semyon Danilov)
  • 17743:Implemented comprehensive pushing of filters into column shards, primarily using scalar kernels. Once the scalar kernels are working, block kernels will be added for special cases. #17743 (Pavel Velikhov)
  • 18121:Fixed an issue where the ALTER TABLE table statements would fail for a table with a vector index. #18121 (azevaykin)
  • 18088:Added safeguards to prevent the occurrence of inconsistent results in specific read-write transactions. Fixes #18064. #18088 (Aleksei Borzenkov)
  • 18059:Fixed crashes by some SELECT queries with DESC in OLAP tables. #18059 (Semyon)
  • 17982:Fixed an issue where an ALTER TABLE table RENAME INDEX statement fails for a vector index. #17982 (azevaykin)
  • 17814:Fixed the incorrect order in which BuildIndex, Export, and Import operations were listed. (#17817) #17814 (Vitaliy Filippov)
  • 17729:Fixed schema version collisions in serverless DBs. #17184 #17729 (Semyon)
  • 18401:Fixed an issue where destination tables of ASYNC REPLICATION and its changefeeds were saved to local backups. This fix prevents duplication of changefeeds and reduces the amount of space the backup takes on disk. #18401 (Daniil Demin)
  • 18234:Fixed an issue where conflicting read-write transactions violated serializability after shard restarts. #18234 (Aleksei Borzenkov)
  • 19048:Fixed the issue causing a crash when the PassAway function was invoked twice. #19048 (Alexey Efimov)
  • 18924:Fixed a race condition between YardInit and Slay when a group (and therefore a VDisk) is created and then immediately deleted, removing "phantom vdisks" from pdisks. #18924 (Semyon Danilov)
  • 18764:Fixed an issue with a timestamp push down in OLAP. #18764 (Oleg Doronin)
  • 19466:Fixed float sum aggregation in arrow::Kernel. #19466 (Oleg Doronin)
  • 19781:Fixed an issue where CMS would try to lock pdisk without vdisks twice. #19781 (Semyon Danilov)
  • 19762:Fixed a bug with absolute paths in workloads. #19762 (Олег)
  • 20238:Fixed an issue where column shards could return different amounts of data for the same queries. Fixed a race in the CPU limiter. #20238 (Pisarenko Grigoriy)
  • 20217:Fixed an issue. Now the force availability mode ignores the limits of offline nodes. (Ilia Shakhov)
  • 20157:Fixed an issue that caused YDB node to segfault when users ran queries in Workload Manager pools with the TOTAL_CPU_LIMIT_PERCENT_PER_NODE restriction enabled. #20157 (Pisarenko Grigoriy)
  • 20084:Fixed an issue where committing message offsets when reading a topic might result in the following error: "Unable to navigate:path: 'Root/logbroker-federation/--cut--/stable/guidance' status: PathErrorUnknown". #20084 (Nikolay Shestakov)
  • 17198:Fixed an issue with UUID data type handling in YDB CLI backup/restore operations. #17198 (Semyon Danilov)
  • 20560:Resolved an issue with memory travel when a consumer commits an offset to the topic with autopartitioning enabled. #20560 (Nikolay Shestakov)
  • 20386:Resolved an issue with reporting of gRPC metrics for serverless databases. #20386 (Ilnaz Nizametdinov)
  • 20785:Fixed a data race in Jaeger settings configurator. #20785 (Sergey Belyakov)
  • 20670:Resolved the issue with DDL errors for external sources and added more information to the ALTER TABLE ... RENAME TO error. #20670 (Pisarenko Grigoriy)
  • 20519:Fixed an issue that caused VDisk to freeze in infinite local recovery mode when a ChunkRead request failed. This change will allow loader actor to terminate properly on PDisk errors, and LocalRecovery to get notified about this error and to finish with proper status. #20519 (Sergey Belyakov)
  • 22298:Fixed an issue where attach streams remained active after session shutdown, causing unexpected BadSession errors. #22298 (Kirill Kurdyukov)
  • 25868:Fix #25869 Removed wrong ensure at preparing batches for writing to ColumnShard. #25868 (Nikita Vasilev)
  • 25587:Fixes ydb-platform#25586

Removes misleading comments in code #25587 (Sergey Belyakov)

Also increased default test logs level #25408 (Pisarenko Grigoriy)

YDB UI

Performance

  • 16867:Enhanced column-oriented table query performance by applying binsearch for predicate bound detection in portions. #16867 (Semyon)
  • 17611:Introduced the Intersect boolean operation to Roaring UDF. #17611 (jsjant)
  • 17533:Extended constant folding to include deterministic UDFs, while non-deterministic UDFs are identified through a blacklist. #17533 (Pavel Velikhov)
  • 17794:Enabled the multi broadcast functionality in the table service by default. #17794 (Олег)
  • 17884:Increased the number of filter types pushed into column shards, resulting in improved performance.#17884 (Pavel Velikhov)
  • 18461:Corrected the task placement logic for read operations from external sources. #18461 (Pisarenko Grigoriy)
  • 18765:Optimized the process of copying TStorageConfig when sending a changed configuration to subscribers. #18765 (Alexander Rutkovsky)
  • 20195:Improved the algorithm of the new scheduler. #20195 (Ivan)
  • 20128:Added a cache for SchemeNavigate responses. Users will receive quicker confirmations when the server successfully writes messages. #20128 (Alek5andr-Kotov)
  • 20048:Improved performance of upsert operations for tables with defaults. Upsert operations now ignore the last DefaulFilledColumnsCnt columns if a row exist. This logic allows QP to avoid reading rows to check if it exists (the whole operation with default columns is processed at a shard). #20048 (r314-git)
  • 19724:Added the buffer_page_alloc_size configuration parameter with the default value of 4Kb. #19724 (Ivan)
  • 19687:Extracted the password verification logic into a dedicated actor, separating it from TSchemeShard local transactions for improved performance. #19687 (Yury Kiselev)
  • 20428:Improved parallel execution of queries to column-oriented tables. #20428 (Oleg Doronin)
  • 21705:Introduced a new priority system for PDisks, addressing performance slowdowns caused by shared queue usage for realtime and compaction writes. #21705 (Vlad Kuznetsov)
  • 25668:Used AS threads in topic sdk IO operations #25668 (Pisarenko Grigoriy)