v0.3.2

What's new in 0.3.2 (2023-06-13)

These are the changes in xorbits v0.3.2.

New features

• FEAT: Support Applymap Operand by @Hank0626 in #445
• FEAT: Add JuiceFs as an alternative storage by @Zhou1213CN in #453
• FEAT: groupby rolling by @UranusSeven in #448
• FEAT: support pivot op by @Hank0626 in #460
• FEAT: rolling cov by @UranusSeven in #450
• FEAT: implement jax-based runtime graph fusion optimizer by @RandomY-2 in #440
• FEAT: support pivot_table op by @Hank0626 in #474
• FEAT: create gpu feature on dashboard by @RayJi01 in #477

Enhancements

• ENH: Support numpy.ndarray fallback to numpy by @Zhou1213CN in #434
• ENH: update pivot_table op by @Hank0626 in #485
• ENH: update tokenizing numpy by @Hank0626 in #486
• ENH: update pivot op by @Hank0626 in #483
• ENH: update warning msg for NPM absence by @UranusSeven in #509
• ENH: Make memory allocation more aggressive on local by @aresnow1 in #514
• BLD: Restrict numpy version in docker for python 3.8 and 3.9 by @ChengjieLi28 in #449
• BLD: Change conda numpy and scipy channel in docker build for python 3.8 by @ChengjieLi28 in #452
• BLD: add doc deps to setup.cfg by @UranusSeven in #472
• BLD: fix doc build by @UranusSeven in #481

Bug fixes

• BUG: Value Error when using dataframes' results to initialize a new dataframe by @Zhou1213CN in #436
• BUG: series.tolist()[0] get Attribute Error by @Zhou1213CN in #430
• BUG: pd.pivot by @UranusSeven in #480
• BUG: fix read pickle bug and add UT by @aa452948257 in #484
• BUG: fix sort_index bug when level attribute is None by @pangyoki in #496
• BUG: Try to fix UCX communication backend argument parser by @luweizheng in #490
• BUG: Fix the GPU display issue in worker and dashboard page by @RayJi01 in #512
• BUG: Fix errors for WSL machines with GPU. by @yifeis7 in #498

Documentation

• DOC: fix the docs issue link by @UranusSeven in #451
• DOC: correct dev doc by @UranusSeven in #455
• DOC: git usage zh_cn by @UranusSeven in #457
• DOC: update pot files by @UranusSeven in #459
• DOC: Check doc building in CI by @aresnow1 in #463
• DOC: translate documentation development by @UranusSeven in #458
• DOC: remove fuzzy statements by @UranusSeven in #468
• DOC: correct & translate creating dev env by @UranusSeven in #470
• DOC: translation for the contributing doc by @RayJi01 in #469
• DOC: Fix docker development doc by @ChengjieLi28 in #473
• DOC: remove the readthedocs related content from dev docs by @shark-21 in #476
• DOC: Update Xorbits build and installation process by @yifeis7 in #495
• DOC: Adds guide to use -f flag to modify default config. by @shark-21 in #508

Others

• CHORE: Modify mergify by @aresnow1 in #467

New Contributors

• @Hank0626 made their first contribution in #445
• @RayJi01 made their first contribution in #469
• @shark-21 made their first contribution in #476
• @pangyoki made their first contribution in #496
• @yifeis7 made their first contribution in #495

Full Changelog: v0.3.1...v0.3.2

v0.3.1

What's new in 0.3.1 (2023-05-16)

These are the changes in xorbits v0.3.1.

Highlights

Xorbits is now able to pass all TPC-H queries for v0.3.1 on a single NVIDIA GPU card.

New features

• FEAT: implement sklearn.model_selection.ParameterGrid by @RandomY-2 in #421

Enhancements

• CLN: Remove k8s deploy in _mars by @ChengjieLi28 in #431

Bug fixes

• BUG: AttributeError when calling df.col by @Zhou1213CN in #410
• BUG: Wrong dataframe id value by @Zhou1213CN in #422
• BUG: Fix agg('size') execute map result, which should be a series rather than a df by @ChengjieLi28 in #427
• BUG: groupby same agg funcs leads to wrong results by @UranusSeven in #433
• BUG: Fix groupby apply on GPU by @ChengjieLi28 in #425
• BUG: generate proper code for custom aggregation func by @UranusSeven in #428

Others

• CHORE: Fix pyarrow version by @aresnow1 in #423
• CHORE: Compatible with Arrow 12.0.0 by @aresnow1 in #426

Full Changelog: v0.3.0...v0.3.1

v0.3.0

What's new in 0.3.0 (2023-04-28)

These are the changes in xorbits v0.3.0.

Highlights

For v0.3.0, Xorbits is generally available on a single NVIDIA GPU card. For TPC-H benchmark, Xorbits is able to pass almost all queries except for q8 and q12. Multiple GPUs support will be added in the coming v0.3.x versions.

New features

• FEAT: np.tri by @aa452948257 in #338
• FEAT: add alluxio storage lib by @Zhou1213CN in #392

Enhancements

• ENH: Support Scale Up on K8S by @Zhou1213CN in #311
• ENH: GPU benchmark script by @UranusSeven in #342
• ENH: lazy init for pandas fallback by @jiayini1119 in #350
• ENH: implement synchronous SGD for Logistic Regression by @RandomY-2 in #323
• ENH: Support DF and Series agg on GPU by @ChengjieLi28 in #346
• ENH: Compatible with pandas 2.0 by @aresnow1 in #328
• ENH: remove global vars in adapters by @jiayini1119 in #372
• ENH: Support arrow list type for read_parquet by @aresnow1 in #379
• ENH: Move the logic for handling this n_io_process to xorbits by @aresnow1 in #401
• CLN: remove mutable service by @qianduoduo0904 in #374

Bug fixes

• BUG: Fix local init for multiple GPU devices by @aresnow1 in #325
• BUG: Int Auto Conversion by @Zhou1213CN in #336
• BUG: unexpected error from DataFrame.__mars_tensor__ by @UranusSeven in #340
• BUG: No permission to chmod when multi users on the same machine by @ChengjieLi28 in #326
• BUG: equality assertion on lazy loaded cupy by @UranusSeven in #247
• BUG: Xorbits.numpy raises TypeError by @Zhou1213CN in #343
• BUG: fetch vram data from cluster by @UranusSeven in #355
• BUG: fetch vram data from local cluster by @UranusSeven in #357
• BUG: Fix pd.__dir__() by @jiayini1119 in #370
• BUG: Fix df.groupby.size on GPU by @ChengjieLi28 in #354
• BUG: fetching from GPU by @UranusSeven in #381
• BUG: destory session in xorbits.shutdown by @ChengjieLi28 in #388
• BUG: DataFrameBinOp tile output dtype is wrong by @ChengjieLi28 in #367
• BUG: GPU DataFrame init from tensors or pandas dataframes by @UranusSeven in #385
• BUG: Fix batch put and fetch data on GPU by @ChengjieLi28 in #383
• BUG: Fix estimate_size of DataFrameReadParquet by @aresnow1 in #398
• BUG: Fix DataFrame.set_axis on GPU by @UranusSeven in #368
• BUG: add attribute 'array' for mars dataframe by @Bojun-Feng in #282
• BUG: series initializer on GPU by @UranusSeven in #414
• BUG: GPU index initializer by @UranusSeven in #409
• BUG: Fix merge and join on GPU by @ChengjieLi28 in #411
• BUG: gpu dataframe iloc by @UranusSeven in #419

Tests

• TST: Fix pandas version by @aresnow1 in #322
• TST: refine ut and add 'convert_to_abspath' ut by @aa452948257 in #320
• TST: fix GPU tests by @UranusSeven in #373
• TST: reuse existing tests for GPU testing by @UranusSeven in #386

Documentation

• DOC: add deployment doc for slurm by @luweizheng in #317
• DOC: added SLURM doc links by @qianduoduo0904 in #332

New Contributors

• @luweizheng made their first contribution in #317
• @Zhou1213CN made their first contribution in #336
• @Bojun-Feng made their first contribution in #282

Full Changelog: v0.2.2...v0.3.0

v0.2.2

What's new in 0.2.2 (2023-04-03)

These are the changes in xorbits v0.2.2.

New features

• FEAT: impl rolling.corr by @UranusSeven in #313

Enhancements

• ENH: globally enable task info collection by @UranusSeven in #228
• ENH: Support xorbits.numpy.random fallback to numpy by @RandomY-2 in #244
• ENH: dump runtime stats to cloud storage systems by @UranusSeven in #243
• ENH: Session id is prefixed with namespace in the k8s environment by @ChengjieLi28 in #259
• ENH: Use get_index_and_uuid to get device uuid by @aresnow1 in #242
• ENH: logging improvements by @UranusSeven in #257
• ENH: do not use global variable for profiling by @qianduoduo0904 in xorbitsai/xoscar#11
• ENH: remove tblib by @qianduoduo0904 in #293
• ENH: use a single tempdir by @UranusSeven in #296
• ENH: init extension entry point when importing by @qianduoduo0904 in #299
• ENH: support numpy fallback by @RandomY-2 in #284
• BLD: Fixes cython compilation failures in Python 3.11. by @sighingnow in #252
• BLD: Fix requirements for building wheel by @aresnow1 in #263
• BLD: Add python cp311 wheel support & drop cp37 wheel build by @aresnow1 in #272
• BLD: Drop Python 3.7 and support Python 3.11 for docker and CI by @ChengjieLi28 in #274
• BLD: Xoscar in Xorbits base image by @ChengjieLi28 in #281
• BLD: Trigger docker cd manually by @ChengjieLi28 in #286
• CLN: Remove _is_exiting_future by @aresnow1 in #230
• CLN: Remove code related to caching function by @aresnow1 in #239
• CLN: Remove read_raydataset by @aresnow1 in #307
• REF: use Xoscar as dependency for actor layer by @qianduoduo0904 in #279

Bug fixes

• BUG: add sleep when waiting for endpoint ready in K8s by @ChengjieLi28 in #241
• BUG: delete node_modules dir in docker by @ChengjieLi28 in #265
• BUG: Fix np.einsum error when computing trace by @RandomY-2 in #262
• BUG: handle interactive exec exception by @UranusSeven in #270
• BUG: fix error when inputting 1d array for np.tril by @jiayini1119 in #283
• BUG: default logging not applied in interactive mode by @UranusSeven in #287
• BUG: Stop uploading node info after interpreter shutdown by @aresnow1 in #291
• BUG: runtime error caused by read_csv and read_parquet with relative path by @aa452948257 in #292

Tests

• TST: GPU CI by @ChengjieLi28 in #240
• TST: Add "_mars" directory to coverage by @aresnow1 in #305

Documentation

• DOC: logging by @UranusSeven in #306
• DOC: K8s drops support for 3.7 and adds support for 3.11 by @ChengjieLi28 in #314

Others

• Chore(deps-dev): Bump webpack from 5.75.0 to 5.76.0 in /python/xorbits/web/ui by @dependabot in #275
• CHORE: Move gpu workflow into python.yaml by @aresnow1 in #256
• CHORE: Fix conditions of mergify by @aresnow1 in #260
• CHORE: Fix url in setup.cfg by @aresnow1 in #261
• CHORE: Update codespell by @aresnow1 in #264
• CHORE: Fix openssl version in k8s by @ChengjieLi28 in #266
• CHORE: Add permissions for mergify by @aresnow1 in #294
• CHORE: Revert "Add permissions for mergify (#294)" by @aresnow1 in #302

New Contributors

• @jiayini1119 made their first contribution in #283
• @aa452948257 made their first contribution in #292

Full Changelog: v0.2.1...v0.2.2

v0.2.1

What's new in 0.2.1 (2023-03-17)

These are the changes in xorbits v0.2.1.

Enhancements

• ENH: globally enable task info collection by @UranusSeven in #228
• ENH: Support xorbits.numpy.random fallback to numpy by @RandomY-2 in #244
• ENH: dump runtime stats to cloud storage systems by @UranusSeven in #243
• ENH: Session id is prefixed with namespace in the k8s environment by @ChengjieLi28 in #259
• ENH: Use get_index_and_uuid to get device uuid by @aresnow1 in #242
• ENH: logging improvements by @UranusSeven in #257
• BLD: Fixes cython compilation failures in Python 3.11. by @sighingnow in #252
• BLD: Fix requirements for building wheel by @aresnow1 in #263
• BLD: Add python cp311 wheel support & drop cp37 wheel build by @aresnow1 in #272
• BLD: Drop Python 3.7 and support Python 3.11 for docker and CI by @ChengjieLi28 in #274
• CLN: Remove _is_exiting_future by @aresnow1 in #230
• CLN: Remove code related to caching function by @aresnow1 in #239

Bug fixes

• BUG: add sleep when waiting for endpoint ready in K8s by @ChengjieLi28 in #241
• BUG: delete node_modules dir in docker by @ChengjieLi28 in #265
• BUG: Fix np.einsum error when computing trace by @RandomY-2 in #262
• BUG: handle interactive exec exception by @UranusSeven in #270

Tests

• TST: GPU CI by @ChengjieLi28 in #240

Others

• Chore(deps-dev): Bump webpack from 5.75.0 to 5.76.0 in /python/xorbits/web/ui by @dependabot in #275
• CHORE: Move gpu workflow into python.yaml by @aresnow1 in #256
• CHORE: Fix conditions of mergify by @aresnow1 in #260
• CHORE: Fix url in setup.cfg by @aresnow1 in #261
• CHORE: Update codespell by @aresnow1 in #264
• CHORE: Fix openssl version in k8s by @ChengjieLi28 in #266

New Contributors

• @sighingnow made their first contribution in #252

Full Changelog: v0.2.0...v0.2.1

v0.2.0

What's new in 0.2.0 (2023-02-21)

Xorbits v0.2.0 mainly focuses on enhancing the user experience.

In terms of deployment, the new version includes a significant improvement to the xorbits.deploy.kubernetes.client.new_cluster() interface, which now automatically detects the local Python version and selects the corresponding Xorbits image when deploying a cluster. Additionally, to avoid dependency conflicts, the new version also automatically aligns with the specific version of certain dependencies, such as cloudpickle.

Regarding interactive user experience, the new version has optimized the delayed execution strategy, avoiding potential duplicate calculations during the analysis process and improving the performance of interactive analysis.

These enhancements should provide users with a more seamless and efficient experience when working with Xorbits.

Enhancements

• ENH: runtime stats by @UranusSeven in #202
• ENH: execute refs in user ns eagerly by @UranusSeven in #217
• ENH: Support read_csv with passing skiprows by @aresnow1 in #218
• ENH: Automatically install some version-sensitive packages and select image for user by @ChengjieLi28 in #192
• REF: Move mars into xorbits by @aresnow1 in #195

Bug fixes

• BUG: group dataframe by anonymous series by @UranusSeven in #205
• BUG: Fix wrong length of DataFrame by @aresnow1 in #223

Tests

• TST: Add asv benchmarks by @aresnow1 in #200
• TST: Move Mars CI into xorbits by @aresnow1 in #207
• TST: added more CI by @qianduoduo0904 in #209
• TST: Add checks for _mars module by @aresnow1 in #219

Documentation

• DOC: Translate Kubernetes deployment docs by @ChengjieLi28 in #190
• DOC: add development doc by @qianduoduo0904 in #189
• DOC: fix links for README by @qianduoduo0904 in #194
• DOC: add links for index doc by @qianduoduo0904 in #197
• DOC: Add twitter link to README.md by @qianduoduo0904 in #201
• DOC: update benchmark links by @qianduoduo0904 in #224

Others

• FIX: scipy requires at least 1.4.0 for voigt_profile by @matrixji in #208
• CHORE: Update license by @aresnow1 in #199

New Contributors

• @matrixji made their first contribution in #208

Full Changelog: v0.1.2...v0.2.0

v0.1.2

What's new in 0.1.2 (2023-02-02)

These are the changes in xorbits v0.1.2.

Enhancements

• BLD: Docker image supports multi python versions by @ChengjieLi28 in #170

Bug fixes

• BUG: fix all the slack links by @qianduoduo0904 in #168
• BUG: update slack links by @qianduoduo0904 in #171
• BUG: Incompatible versions of Minikube and Kubernetes by @ChengjieLi28 in #172
• BUG: Workers pod may turn to running status before supervisor by @ChengjieLi28 in #177

Documentation

• DOC: Add note about installing requirements for 10 minutes to xorbits.numpy by @RandomY-2 in #176
• DOC: fix index and update po by @UranusSeven in #178
• DOC: add make html_zh_cn by @qianduoduo0904 in #179
• DOC: add zh_CN translation for index by @qianduoduo0904 in #180
• DOC: fix translation by @qianduoduo0904 in #182
• DOC: Translate part of deployment documentations into zh_CN by @aresnow1 in #181
• DOC: Fix ref link in deployment section by @aresnow1 in #184
• DOC: zh_CN for Installation by @aresnow1 in #185
• Doc: User guide by @aresnow1 in #186
• DOC: getting started translation by @UranusSeven in #187

Others

• Fix: change slack link by @qianduoduo0904 in #167
• CHORE: Update black by @aresnow1 in #183
• Chore: update submodule by @XprobeBot in #188

New Contributors

• @RandomY-2 made their first contribution in #176

Full Changelog: v0.1.1...v0.1.2

v0.1.1

What's new in 0.1.1 (2023-01-17)

These are the changes in xorbits v0.1.1.

Enhancements

• ENH: Make sure only one local cluster exists by @aresnow1 in #150
• ENH: K8s deployment supports installing pip and conda packages by @ChengjieLi28 in #155
• ENH: Support aws and azure install options by @ChengjieLi28 in #164
• BLD: Xorbits CD for Dockerhub by @ChengjieLi28 in #146

Bug fixes

• BUG: Show xorbits release version and commit id on Web UI by @ChengjieLi28 in #165

Tests

• TST: Fix test_cluster hanging by @aresnow1 in #149

Documentation

• DOC: Use one supervisoras example for cluster deployment by @qianduoduo0904 in #154
• DOC: Add i18n support by @qianduoduo0904 in #161

Others

• Chore(deps): Bump json5 from 2.2.1 to 2.2.3 in /python/xorbits/web/ui by @dependabot in #148
• Chore: update submodule by @XprobeBot in #163
• Chore: update submodule by @XprobeBot in #166

New Contributors

• @dependabot made their first contribution in #148
• @XprobeBot made their first contribution in #163

Full Changelog: v0.1.0...v0.1.1

v0.1.0

These are the changes in xorbits v0.1.0.

New features

• FEAT: support starting via python -m xorbits.{supervisor, worker} by @qianduoduo0904 in #110

Enhancements

• ENH: pandas Series and Index fallbacks by @UranusSeven in #117
• ENH: Make Xorbits Data iterable by @aresnow1 in #118
• ENH: version disclaimers by @UranusSeven in ENH: version disclaimers by UranusSeven · Pull Request #126 · xprobe-inc/xorbits
• ENH: Create mergify configuration by @aresnow1 in #127
• ENH: Support kubernetes deploy by @ChengjieLi28 in #85
• ENH: replace head2head compare about API with gif for readme by @qianduoduo0904 in #134
• ENH: Stop executing DataRef when pycharm etc tries to evaluate properties in debug by @qianduoduo0904 in #141

Bug fixes

• BUG: Add cls method fallback warning by @hoarjour in #124
• BUG: handle at and iloc by @hoarjour in #116
• BUG: Set use_arrow_dtype's default to False in TPCH benchmark by @aresnow1 in #128
• BUG: Set MARS_USE_CGROUP_STAT environment when creating k8s cluster by @aresnow1 in BUG: Set MARS_USE_CGROUP_STAT environment when creating k8s cluster by aresnow1 · Pull Request #14

Tests

• TST: enhance tests stability by @qianduoduo0904 in #125
• TST: Add codecov configuration by @aresnow1 in #137

Documentation

• DOC: installation by @UranusSeven in #114
• DOC: add deployment doc by @qianduoduo0904 in DOC: add deployment doc by qianduoduo0904 · Pull Request #122 · xprobe-inc/xorbits
• DOC: Added API implemented version and plan by @qianduoduo0904 in #135
• DOC: documents about K8s deployment by @ChengjieLi28 in #131
• DOC: Added benchmark for readme & doc index by @qianduoduo0904 in #136
• DOC: refine README and doc index by @qianduoduo0904 in #145

Others

• Fix mergify configurations by @aresnow1 in #129
• Switch merge method to squash by @aresnow1 in Switch merge method to squash by aresnow1 · Pull Request #130 · xprobe-inc/xorbits
• Automatically update branch by @aresnow1 in #132
• FIX: correct license by @UranusSeven in #143
• Chore: update submodule by @qianduoduo0904 in Chore: update submodule by qianduoduo0904 · Pull Request #123 · xprobe-inc/xorbits
• CHORE: Bump copyright year by @aresnow1 in #138
• CHORE: Update pandas_vs_xorbits.gif by @lipengsh in CHORE: Update pandas_vs_xorbits.gif by lipengsh · Pull Request #140 · xprobe-inc/xorbits

New Contributors

• @lipengsh made their first contribution in CHORE: Update pandas_vs_xorbits.gif by lipengsh · Pull Request #140 · xprobe-inc/xorbits

Full Changelog: v0.0.1...v0.1.0

v0.0.1

ENH: Suppress Numpy warnings (#120)