Skip to content

Commit 35ffc16

Browse files
chaokunyangchaokunyang
authored
[Ray] Refine mars on ray usability (#2580)
1 parent 9f6b857 commit 35ffc16

File tree

12 files changed

+418
-19
lines changed

12 files changed

+418
-19
lines changed

‎README.rst‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,48 @@ Mars can scale in to a single machine, and scale out to a cluster with thousands
242242
It's fairly simple to migrate from a single machine to a cluster to
243243
process more data or gain a better performance.
244244

245+
Mars on Ray
246+
------------
247+
Mars also has deep integration with Ray and can run on `Ray <https://docs.ray.io/en/latest/>` efficiently and
248+
interact with the large ecosystem of machine learning and distributed systems built on top of the core Ray.
249+
250+
Starting a new Mars on Ray runtime locally via:
251+
252+
.. code-block:: python
253+
254+
import ray
255+
ray.init()
256+
import mars
257+
mars.new_ray_session(worker_num=2)
258+
import mars.tensor as mt
259+
mt.random.RandomState(0).rand(1000_0000, 5).sum().execute()
260+
261+
Or connecting to a Mars on Ray cluster which is already initialized.
262+
263+
.. code-block:: python
264+
265+
import mars
266+
mars.new_ray_session('http://<web_ip>:<ui_port>')
267+
# perform computation
268+
269+
Interact with Ray Dataset:
270+
271+
.. code-block:: python
272+
273+
import mars.tensor as mt
274+
import mars.dataframe as md
275+
df = md.DataFrame(
276+
mt.random.rand(1000_0000, 4),
277+
columns=list('abcd'))
278+
# Convert mars dataframe to ray dataset
279+
ds = md.to_ray_dataset(df)
280+
print(ds.schema(), ds.count())
281+
ds.filter(lambda row: row["a"] > 0.5).show(5)
282+
# Convert ray dataset to mars dataframe
283+
df2 = md.read_ray_dataset(ds)
284+
print(df2.head(5).execute())
285+
286+
Refer to `Mars on Ray`_ for more information.
245287

246288
Bare Metal Deployment
247289
`````````````````````
@@ -310,6 +352,7 @@ Thank you in advance for your contributions!
310352
.. _`pull requests`: https://github.com/mars-project/mars/pulls
311353
.. _`Documentation`: https://docs.pymars.org
312354
.. _`中文文档`: https://docs.pymars.org/zh_CN/latest/
355+
.. _`Mars on Ray`: https://docs.pymars.org/en/latest/installation/ray.html
313356
.. _`Run on Kubernetes`: https://docs.pymars.org/en/latest/installation/kubernetes.html
314357
.. _`Run on Yarn`: https://docs.pymars.org/en/latest/installation/yarn.html
315358
.. _`DASK on Mars`: https://docs.pymars.org/en/latest/user_guide/contrib/dask.html

‎docs/source/index.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,11 @@ DASK on Mars
168168

169169
Refer to :ref:`DASK on Mars <integrate_dask>`.
170170

171+
Mars on Ray
172+
------------
173+
174+
Refer to :ref:`Mars on Ray <mars_ray>`.
175+
171176
Easy to scale in and scale out
172177
------------------------------
173178

@@ -180,6 +185,7 @@ Mars can run in a few ways:
180185

181186
- :ref:`Local scheduling <local>`
182187
- :ref:`Run on cluster <deploy>`
188+
- :ref:`Run on Ray <mars_ray>`
183189
- :ref:`Run on Kubernetes <k8s>`
184190
- :ref:`Run on Yarn <mars_yarn>`
185191

‎docs/source/installation/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,5 +15,6 @@ You can simply install Mars via pip:
1515

1616
install
1717
deploy
18+
ray
1819
kubernetes
1920
yarn

‎docs/source/installation/ray.rst‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
.. _mars_ray:
2+
3+
Run on Ray
4+
=================
5+
6+
Mars also has deep integration with Ray and can run on `Ray <https://docs.ray.io/en/latest/>`_ efficiently and natively.
7+
8+
Basic steps
9+
-----------
10+
Install Ray locally:
11+
12+
.. code-block:: bash
13+
14+
pip install ray>=1.7.0
15+
16+
Start a Ray cluster:
17+
18+
.. code-block:: python
19+
20+
import ray
21+
ray.init()
22+
23+
Or connecting to a existing Ray cluster using `Ray client <https://docs.ray.io/en/latest/cluster/ray-client.html>`_:
24+
25+
.. code-block:: python
26+
27+
import ray
28+
ray.init(address="ray://<head_node_host>:10001")
29+
30+
Creating a Mars on Ray runtime in the Ray cluster and do the computing:
31+
32+
.. code-block:: python
33+
34+
import mars
35+
import mars.tensor as mt
36+
import mars.dataframe as md
37+
session = mars.new_ray_session(worker_num=2, worker_mem=2 * 1024 ** 3)
38+
mt.random.RandomState(0).rand(1000_0000, 5).sum().execute()
39+
df = md.DataFrame(
40+
mt.random.rand(1000_0000, 4, chunk_size=500_0000),
41+
columns=list('abcd'))
42+
print(df.sum().execute())
43+
print(df.describe().execute())
44+
# Convert mars dataframe to ray dataset
45+
ds = md.to_ray_dataset(df)
46+
print(ds.schema(), ds.count())
47+
ds.filter(lambda row: row["a"] > 0.5).show(5)
48+
# Convert ray dataset to mars dataframe
49+
df2 = md.read_ray_dataset(ds)
50+
print(df2.head(5).execute())
51+
52+
53+
Create a Mars on Ray runtime independently in the Ray cluster:
54+
55+
.. code-block:: python
56+
57+
import mars
58+
import mars.tensor as mt
59+
cluster = mars.new_cluster_in_ray(worker_num=2, worker_mem=2 * 1024 ** 3)
60+
61+
Connect to the created Mars on Ray runtime and do the computing:
62+
63+
.. code-block:: python
64+
65+
import mars
66+
import mars.tensor as mt
67+
session = mars.new_ray_session(address="http://ip:port", session_id="abcd", default=True)
68+
session.execute(mt.random.RandomState(0).rand(100, 5).sum())
69+
70+
Stop the created Mars on Ray runtime:
71+
72+
.. code-block:: python
73+
74+
cluster.stop()
75+
76+
77+
Customizing cluster
78+
-------------------
79+
``new_ray_session``/``new_cluster_in_ray`` function provides several keyword arguments for users to define
80+
the cluster.
81+
82+
Arguments for supervisors:
83+
84+
+----------------------+-----------------------------------------------------------+
85+
| Argument | Description |
86+
+======================+===========================================================+
87+
| supervisor_mem | Memory size for supervisor in the cluster, in bytes. |
88+
+----------------------+-----------------------------------------------------------+
89+
90+
Arguments for workers:
91+
92+
+--------------------+-----------------------------------------------------------------+
93+
| Argument | Description |
94+
+====================+=================================================================+
95+
| worker_num | Number of workers in the cluster, 1 by default. |
96+
+--------------------+-----------------------------------------------------------------+
97+
| worker_cpu | Number of CPUs for every worker, 2 by default. |
98+
+--------------------+-----------------------------------------------------------------+
99+
| worker_mem | Memory size for workers in the cluster, in bytes, 2G by default.|
100+
+--------------------+-----------------------------------------------------------------+
101+
102+
For instance, if you want to create a Mars cluster with 100 workers,
103+
each worker has 4 cores and 16GB memory, you can use the code below:
104+
105+
.. code-block:: python
106+
107+
import mars
108+
import mars.tensor as mt
109+
cluster = mars.new_cluster_in_ray(worker_num=100, worker_cpu=4, worker_mem=16 * 1024 ** 3)

‎docs/source/locale/zh_CN/LC_MESSAGES/index.po‎

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,14 @@ msgid ""
88
msgstr ""
99
"Project-Id-Version: mars \n"
1010
"Report-Msgid-Bugs-To: \n"
11-
"POT-Creation-Date: 2021-08-29 00:57+0800\n"
11+
"POT-Creation-Date: 2021-11-17 17:55+0800\n"
1212
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
1313
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
1414
"Language-Team: LANGUAGE <[email protected]>\n"
1515
"MIME-Version: 1.0\n"
1616
"Content-Type: text/plain; charset=utf-8\n"
1717
"Content-Transfer-Encoding: 8bit\n"
18-
"Generated-By: Babel 2.8.0\n"
18+
"Generated-By: Babel 2.9.1\n"
1919

2020
#: ../../source/index.rst:5
2121
msgid "Mars Documentation"
@@ -160,10 +160,18 @@ msgid "Refer to :ref:`DASK on Mars <integrate_dask>`."
160160
msgstr "参考 :ref:`DASK on Mars <integrate_dask>`。"
161161

162162
#: ../../source/index.rst:172
163+
msgid "Mars on Ray"
164+
msgstr ""
165+
166+
#: ../../source/index.rst:174
167+
msgid "Refer to :ref:`Mars on Ray <mars_ray>`."
168+
msgstr "参考 :ref:`Mars on Ray <mars_ray>`。"
169+
170+
#: ../../source/index.rst:177
163171
msgid "Easy to scale in and scale out"
164172
msgstr "适应各种数据规模"
165173

166-
#: ../../source/index.rst:174
174+
#: ../../source/index.rst:179
167175
msgid ""
168176
"Mars can scale in to a single machine, and scale out to a cluster with "
169177
"hundreds of machines. Both the local and distributed version share the "
@@ -174,23 +182,27 @@ msgstr ""
174182
"两种环境下可使用相同的代码。因此，Mars 可以方便地从单台机器迁移到集群，以"
175183
"处理更多数据或者获得更好的性能。"
176184

177-
#: ../../source/index.rst:179
185+
#: ../../source/index.rst:184
178186
msgid "Mars can run in a few ways:"
179187
msgstr "Mars 能以若干种方式运行："
180188

181-
#: ../../source/index.rst:181
189+
#: ../../source/index.rst:186
182190
msgid ":ref:`Local scheduling <local>`"
183191
msgstr ":ref:`本地执行 <local>`"
184192

185-
#: ../../source/index.rst:182
193+
#: ../../source/index.rst:187
186194
msgid ":ref:`Run on cluster <deploy>`"
187195
msgstr ":ref:`在集群中运行 <deploy>`"
188196

189-
#: ../../source/index.rst:183
197+
#: ../../source/index.rst:188
198+
msgid ":ref:`Run on Ray <mars_ray>`"
199+
msgstr ":ref:`在 Ray 中运行 Mars <mars_ray>`"
200+
201+
#: ../../source/index.rst:189
190202
msgid ":ref:`Run on Kubernetes <k8s>`"
191203
msgstr ":ref:`在 Kubernetes 中部署 <k8s>`"
192204

193-
#: ../../source/index.rst:184
205+
#: ../../source/index.rst:190
194206
msgid ":ref:`Run on Yarn <mars_yarn>`"
195207
msgstr ":ref:`在 Yarn 中部署 <k8s>`"
196208

‎docs/source/locale/zh_CN/LC_MESSAGES/installation/ray.po‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# SOME DESCRIPTIVE TITLE.
2+
# Copyright (C) 1999-2020, The Alibaba Group Holding Ltd.
3+
# This file is distributed under the same license as the mars package.
4+
# FIRST AUTHOR <EMAIL@ADDRESS>, 2020.
5+
#
6+
#, fuzzy
7+
msgid ""
8+
msgstr ""
9+
"Project-Id-Version: mars 0.8.0rc1\n"
10+
"Report-Msgid-Bugs-To: \n"
11+
"POT-Creation-Date: 2021-11-17 20:47+0800\n"
12+
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
13+
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
14+
"Language-Team: LANGUAGE <[email protected]>\n"
15+
"MIME-Version: 1.0\n"
16+
"Content-Type: text/plain; charset=utf-8\n"
17+
"Content-Transfer-Encoding: 8bit\n"
18+
"Generated-By: Babel 2.9.1\n"
19+
20+
#: ../../source/installation/ray.rst:4
21+
msgid "Run on Ray"
22+
msgstr "在 Ray 中运行 Mars"
23+
24+
#: ../../source/installation/ray.rst:6
25+
msgid ""
26+
"Mars also has deep integration with Ray and can run on `Ray "
27+
"<https://docs.ray.io/en/latest/>`_ efficiently and natively."
28+
msgstr ""
29+
"Mars 与 `Ray <https://docs.ray.io/en/latest/>`_ 进行了深度集成，并可以"
30+
"高效原生地运行在 Ray 上。"
31+
32+
#: ../../source/installation/ray.rst:9
33+
msgid "Basic steps"
34+
msgstr "基本步骤"
35+
36+
#: ../../source/installation/ray.rst:10
37+
msgid "Install Ray locally:"
38+
msgstr "在本地安装 Ray ："
39+
40+
#: ../../source/installation/ray.rst:16
41+
msgid "Start a Ray cluster:"
42+
msgstr "启动 Ray 集群："
43+
44+
#: ../../source/installation/ray.rst:23
45+
msgid ""
46+
"Or connecting to a existing Ray cluster using `Ray client "
47+
"<https://docs.ray.io/en/latest/cluster/ray-client.html>`_:"
48+
msgstr ""
49+
"或者使用 `Ray Client <https://docs.ray.io/en/latest/cluster/ray-client."
50+
"html>`_ 连接到一个已有的集群："
51+
52+
#: ../../source/installation/ray.rst:30
53+
msgid "Creating a Mars on Ray runtime in the Ray cluster and do the computing:"
54+
msgstr "创建 Mars on Ray 运行时并执行计算："
55+
56+
#: ../../source/installation/ray.rst:53
57+
msgid "Create a Mars on Ray runtime independently in the Ray cluster:"
58+
msgstr "在 Ray 集群里面独立创建 Mars on Ray运行时："
59+
60+
#: ../../source/installation/ray.rst:61
61+
msgid "Connect to the created Mars on Ray runtime and do the computing:"
62+
msgstr "连接到创建的 Mars on Ray 运行时并执行计算："
63+
64+
#: ../../source/installation/ray.rst:70
65+
msgid "Stop the created Mars on Ray runtime:"
66+
msgstr "停止 Mars on Ray 运行时："
67+
68+
#: ../../source/installation/ray.rst:78
69+
msgid "Customizing cluster"
70+
msgstr "自定义集群"
71+
72+
#: ../../source/installation/ray.rst:79
73+
msgid ""
74+
"``new_ray_session``/``new_cluster_in_ray`` function provides several "
75+
"keyword arguments for users to define the cluster."
76+
msgstr ""
77+
"``new_ray_session``/``new_cluster_in_ray`` 函数提供了一些用于自定义集群的"
78+
"关键字参数。"
79+
80+
#: ../../source/installation/ray.rst:82
81+
msgid "Arguments for supervisors:"
82+
msgstr "Supervisor 相关参数："
83+
84+
#: ../../source/installation/ray.rst:85 ../../source/installation/ray.rst:93
85+
msgid "Argument"
86+
msgstr "参数"
87+
88+
#: ../../source/installation/ray.rst:85 ../../source/installation/ray.rst:93
89+
msgid "Description"
90+
msgstr "描述"
91+
92+
#: ../../source/installation/ray.rst:87
93+
msgid "supervisor_mem"
94+
msgstr ""
95+
96+
#: ../../source/installation/ray.rst:87
97+
msgid "Memory size for supervisor in the cluster, in bytes."
98+
msgstr "Supervisor 的内存大小，单位是字节"
99+
100+
#: ../../source/installation/ray.rst:90
101+
msgid "Arguments for workers:"
102+
msgstr "Worker 相关参数："
103+
104+
#: ../../source/installation/ray.rst:95
105+
msgid "worker_num"
106+
msgstr ""
107+
108+
#: ../../source/installation/ray.rst:95
109+
msgid "Number of workers in the cluster, 1 by default."
110+
msgstr "集群中 Worker 的数目，默认为 1"
111+
112+
#: ../../source/installation/ray.rst:97
113+
msgid "worker_cpu"
114+
msgstr ""
115+
116+
#: ../../source/installation/ray.rst:97
117+
msgid "Number of CPUs for every worker, 2 by default."
118+
msgstr "每个 Worker 的 CPU 数目，默认为 2"
119+
120+
#: ../../source/installation/ray.rst:99
121+
msgid "worker_mem"
122+
msgstr ""
123+
124+
#: ../../source/installation/ray.rst:99
125+
msgid "Memory size for workers in the cluster, in bytes, 2G by default."
126+
msgstr "每个 Worker 的内存大小，单位是字节，默认2G"
127+
128+
#: ../../source/installation/ray.rst:102
129+
msgid ""
130+
"For instance, if you want to create a Mars cluster with 100 workers, each"
131+
" worker has 4 cores and 16GB memory, you can use the code below:"
132+
msgstr ""
133+
"比如你想创建一个100个 Worker 的 Mars 集群，每个 Worker 拥有 4 核 16GB "
134+
"内存，你可以使用下面的代码："

0 commit comments

Comments
 (0)