Releases · lance-format/lance-spark

05 Mar 00:57

lance-community

v0.2.0-beta.2

06bc061

v0.2.0-beta.2 Pre-release

Pre-release

What's Changed

New Features 🎉

feat: add show indexes command to list all indexes in a lance dataset by @fangbo in #282

Bug Fixes 🐛

fix: fixedsizebinary datatype support by @jiaoew1991 in #273
fix: support Date(MILLISECOND) and FixedSizeBinary arrow types by @a-sane in #283
fix: setting namespace correctly when opening datasets by @hamersaw in #289

New Contributors

@a-sane made their first contribution in #283

Full Changelog: v0.2.0-beta.1...v0.2.0-beta.2

Contributors

a-sane, fangbo, and 2 other contributors

Assets 2

27 Feb 15:02

lance-community

v0.2.0-beta.1

ea9c717

v0.2.0-beta.1 Pre-release

Pre-release

What's Changed

Breaking Changes 🛠

refactor!: use single_level_ns instead of extra_level and change batch_size default by @jackye1995 in #258
refactor!: remove legacy LanceCatalog by @jackye1995 in #263

New Features 🎉

feat: wire CDF and stable row IDs support from Java API by @burlacio in #253
feat: replace table and all variants (RTAS, CORT, CORTAS, etc) by @hamersaw in #251
feat: improve caching by @jackye1995 in #261

Bug Fixes 🐛

fix: load table with correct read options by @bryanck in #224

Documentation 📚

docs: add databricks vendor docs by @hamersaw in #219

Other Changes

refactor: remove param largeVarTypes from LanceArrowUtils.toArrowSchema by @fangbo in #270
refactor: replace newTransactionBuilder with CommitBuilder API by @hamersaw in #271
refactor: set overwrite write mode on replace operation fragment writes by @hamersaw in #262

New Contributors

@burlacio made their first contribution in #253

Full Changelog: v0.2.0...v0.2.0-beta.1

Contributors

fangbo, burlacio, and 3 other contributors

Assets 2

06 Feb 23:48

lance-community

v0.2.0

ae296c0

v0.2.0 Latest

Latest

What's Changed

Breaking Changes 🛠

refactor!: update to use org.lance domain and related lance packages by @jackye1995 in #133

New Features 🎉

feat: add support for distributed compaction by @fangbo in #105
feat: add the data storage version config in write options by @xloya in #123
feat: support dataframe load path API by @jiaoew1991 in #131
feat: support count optimization without filter by @jackye1995 in #139
feat: bump lance to 2.0.0-beta.3 by @jackye1995 in #145
feat: support read and write large varchar column by @jackye1995 in #146
feat: add support for cleanup by @fangbo in #132
feat: add experimental support for queued write buffer by @jackye1995 in #148
feat(spark): support Java 21 for Spark 4.0 by @zhangyue19921010 in #160
feat: support to create btree index distributedly by @fangbo in #150
feat: support distributed FTS index by @jtuglu1 in #171
feat: bump lance to 2.0.0-beta.9 and lance-namespace to 0.4.5 by @jackye1995 in #177
feat: auto-update lance version by @jackye1995 in #180
feat: support refreshing vended credentials in worker by @jackye1995 in #175
feat: optimizer for fragment aware join by @jiaoew1991 in #161
feat: support and optimize Spark MERGE INTO by @zhangyue19921010 in #172
feat: support vector search by @nyl3532016 in #189
feat: support time travel by @zhangyue19921010 in #201
feat: support UPDATE COLUMNS FROM syntax for column-level updates by @jiaoew1991 in #194

Bug Fixes 🐛

fix: support read list of fixed-size-list by @jiaoew1991 in #93
fix: work around for nested datatype pushdown failed by @jiaoew1991 in #96
fix: uint8 uint16 uint32 convert failed by @jiaoew1991 in #125
fix: support select uint8, uint16 from table by @jiaoew1991 in #129
fix: add timeout for LanceDataWriter.abort by @c21 in #142
fix: ensure spark runtime options are properly merged by @jackye1995 in #147
fix: use manifest summary for stats reporting by @jackye1995 in #151
fix: properly handle array in struct by @jackye1995 in #155
fix: make allocator config simpler by @jackye1995 in #157
fix: set classloader for Kryo by @bryanck in #178
fix: move Spark vectorized classes under org.lance by @bryanck in #188
fix: set indexedDefined for StructWriter by @fangbo in #186
fix: update docker-build to use spark 3.5.8 and support docker compose v2 by @hamersaw in #183
fix: unify OPTIMIZE Kryo serialization and fix classloader for executor decode by @majian1998 in #193
fix: spark dataframe saveAsTable fails by @hfutatzhanghb in #173
fix: strip storage. prefix from catalog storage options by @majian1998 in #179
fix: correctly set write mode when insert overwrite table by @fangbo in #204
fix: use deregisterTable for dropTable and dropTable for purgeTable by @jackye1995 in #209

Documentation 📚

docs: add performance guide by @jackye1995 in #149
docs: update doc for time travel, drop, insert overwrite by @jackye1995 in #215

Other Changes

refactor: use OPTIMIZE and VACUUM for compaction and cleanup by @jackye1995 in #152
refactor: cleanup configs and improve namespace handling by @jackye1995 in #156

New Contributors

@hfutatzhanghb made their first contribution in #112
@xloya made their first contribution in #123
@c21 made their first contribution in #142
@zhangyue19921010 made their first contribution in #160
@jtuglu1 made their first contribution in #171
@lance-community made their first contribution in #182
@bryanck made their first contribution in #178
@nyl3532016 made their first contribution in #189
@hamersaw made their first contribution in #183
@majian1998 made their first contribution in #193

Full Changelog: v0.0.15...v0.2.0

Contributors

fangbo, jiaoew1991, and 11 other contributors

Assets 2

04 Feb 02:56

lance-community

v0.1.3-beta.8

6cd050c

v0.1.3-beta.8 Pre-release

Pre-release

What's Changed

New Features 🎉

feat(spark): support Java 21 for Spark 4.0 by @zhangyue19921010 in #160
feat: support to create btree index distributedly by @fangbo in #150
feat: support distributed FTS index by @jtuglu1 in #171
feat: bump lance to 2.0.0-beta.9 and lance-namespace to 0.4.5 by @jackye1995 in #177
feat: auto-update lance version by @jackye1995 in #180
feat: support refreshing vended credentials in worker by @jackye1995 in #175
feat: optimizer for fragment aware join by @jiaoew1991 in #161
feat: support and optimize Spark MERGE INTO by @zhangyue19921010 in #172
feat: support vector search by @nyl3532016 in #189

Bug Fixes 🐛

fix: properly handle array in struct by @jackye1995 in #155
fix: make allocator config simpler by @jackye1995 in #157
fix: set classloader for Kryo by @bryanck in #178
fix: move Spark vectorized classes under org.lance by @bryanck in #188
fix: set indexedDefined for StructWriter by @fangbo in #186
fix: update docker-build to use spark 3.5.8 and support docker compose v2 by @hamersaw in #183
fix: unify OPTIMIZE Kryo serialization and fix classloader for executor decode by @majian1998 in #193

Other Changes

refactor: use OPTIMIZE and VACUUM for compaction and cleanup by @jackye1995 in #152
refactor: cleanup configs and improve namespace handling by @jackye1995 in #156