Skip to content

Commit b8b2e89

Browse files
wanglei828wangkuiyi
authored andcommitted
Scripts: refactor paddle bash shell scripts. (#10038)
* Scripts: refactor paddle bash shell scripts. * Fix indent. * Modify readme documents.
1 parent 4c8ff72 commit b8b2e89

7 files changed

+642
-59
lines changed

paddle/scripts/docker/README.md renamed to paddle/scripts/README.md

Lines changed: 45 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -13,40 +13,49 @@ We want to make the building procedures:
1313
1. Build docker images with PaddlePaddle pre-installed, so that we can run
1414
PaddlePaddle applications directly in docker or on Kubernetes clusters.
1515

16-
To achieve this, we created a repo: https://github.com/PaddlePaddle/buildtools
17-
which gives several docker images that are `manylinux1` sufficient. Then we
18-
can build PaddlePaddle using these images to generate corresponding `whl`
19-
binaries.
16+
To achieve this, we maintain a dockerhub repo:https://hub.docker.com/r/paddlepaddle/paddle
17+
which provides pre-built environment images to build PaddlePaddle and generate corresponding `whl`
18+
binaries.(**We strongly recommend building paddlepaddle in our pre-specified Docker environment.**)
2019

21-
## Run The Build
20+
## Development Workflow
21+
22+
Here we describe how the workflow goes on. We start from considering our daily development environment.
23+
24+
Developers work on a computer, which is usually a laptop or desktop:
25+
26+
<img src="doc/paddle-development-environment.png" width=500 />
27+
28+
or, they might rely on a more sophisticated box (like with GPUs):
29+
30+
<img src="doc/paddle-development-environment-gpu.png" width=500 />
31+
32+
A principle here is that source code lies on the development computer (host) so that editors like Eclipse can parse the source code to support auto-completion.
33+
34+
## Build With Docker
2235

2336
### Build Environments
2437

25-
The pre-built build environment images are:
38+
The lastest pre-built build environment images are:
2639

2740
| Image | Tag |
2841
| ----- | --- |
29-
| paddlepaddle/paddle_manylinux_devel | cuda7.5_cudnn5 |
30-
| paddlepaddle/paddle_manylinux_devel | cuda8.0_cudnn5 |
31-
| paddlepaddle/paddle_manylinux_devel | cuda7.5_cudnn7 |
32-
| paddlepaddle/paddle_manylinux_devel | cuda9.0_cudnn7 |
42+
| paddlepaddle/paddle | latest-dev |
43+
| paddlepaddle/paddle | latest-dev-android |
3344

3445
### Start Build
3546

36-
Choose one docker image that suit your environment and run the following
37-
command to start a build:
38-
3947
```bash
4048
git clone https://github.com/PaddlePaddle/Paddle.git
4149
cd Paddle
42-
docker run --rm -v $PWD:/paddle -e "WITH_GPU=OFF" -e "WITH_AVX=ON" -e "WITH_TESTING=OFF" -e "RUN_TEST=OFF" -e "PYTHON_ABI=cp27-cp27mu" paddlepaddle/paddle_manylinux_devel /paddle/paddle/scripts/docker/build.sh
50+
./paddle/scripts/paddle_docker_build.sh build
4351
```
4452

4553
After the build finishes, you can get output `whl` package under
4654
`build/python/dist`.
4755

48-
This command mounts the source directory on the host into `/paddle` in the container, then run the build script `/paddle/paddle/scripts/docker/build.sh`
49-
in the container. When it writes to `/paddle/build` in the container, it writes to `$PWD/build` on the host indeed.
56+
This command will download the most recent dev image from docker hub, start a container in the backend and then run the build script `/paddle/paddle/scripts/paddle_build.sh build` in the container.
57+
The container mounts the source directory on the host into `/paddle`.
58+
When it writes to `/paddle/build` in the container, it writes to `$PWD/build` on the host indeed.
5059

5160
### Build Options
5261

@@ -68,7 +77,6 @@ Users can specify the following Docker build arguments with either "ON" or "OFF"
6877
| `WITH_DOC` | OFF | Build docs after build binaries. |
6978
| `WOBOQ` | OFF | Generate WOBOQ code viewer under `build/woboq_out` |
7079

71-
7280
## Docker Images
7381

7482
You can get the latest PaddlePaddle docker images by
@@ -144,59 +152,37 @@ docker push
144152
kubectl ...
145153
```
146154

147-
## Docker Images for Developers
148-
149-
We have a special docker image for developers:
150-
`paddlepaddle/paddle:<version>-dev`. This image is also generated from
151-
https://github.com/PaddlePaddle/buildtools
152-
153-
This a development image contains only the
154-
development tools and standardizes the building procedure. Users include:
155-
156-
- developers -- no longer need to install development tools on the host, and can build their current work on the host (development computer).
157-
- release engineers -- use this to build the official release from certain branch/tag on Github.com.
158-
- document writers / Website developers -- Our documents are in the source repo in the form of .md/.rst files and comments in source code. We need tools to extract the information, typeset, and generate Web pages.
159-
160-
Of course, developers can install building tools on their development computers. But different versions of PaddlePaddle might require different set or version of building tools. Also, it makes collaborative debugging easier if all developers use a unified development environment.
161-
162-
The development image contains the following tools:
163-
164-
- gcc/clang
165-
- nvcc
166-
- Python
167-
- sphinx
168-
- woboq
169-
- sshd
170-
171-
Many developers work on a remote computer with GPU; they could ssh into the computer and `docker exec` into the development container. However, running `sshd` in the container allows developers to ssh into the container directly.
172-
173-
174-
### Development Workflow
175-
176-
Here we describe how the workflow goes on. We start from considering our daily development environment.
155+
### Reading source code with woboq codebrowser
177156

178-
Developers work on a computer, which is usually a laptop or desktop:
157+
For developers who are interested in the C++ source code, you can build C++ source code into HTML pages using [Woboq codebrowser](https://github.com/woboq/woboq_codebrowser).
179158

180-
<img src="doc/paddle-development-environment.png" width=500 />
159+
- The following command builds PaddlePaddle, generates HTML pages from C++ source code, and writes HTML pages into `$HOME/woboq_out` on the host:
181160

182-
or, they might rely on a more sophisticated box (like with GPUs):
161+
```bash
162+
./paddle/scripts/paddle_docker_build.sh html
163+
```
183164

184-
<img src="doc/paddle-development-environment-gpu.png" width=500 />
165+
- You can open the generated HTML files in your Web browser. Or, if you want to run a Nginx container to serve them for a wider audience, you can run:
185166

186-
A principle here is that source code lies on the development computer (host) so that editors like Eclipse can parse the source code to support auto-completion.
167+
```
168+
docker run -v $HOME/woboq_out:/usr/share/nginx/html -d -p 8080:80 nginx
169+
```
187170

188-
### Reading source code with woboq codebrowser
171+
## More Options
189172

190-
For developers who are interested in the C++ source code, please use -e "WOBOQ=ON" to enable the building of C++ source code into HTML pages using [Woboq codebrowser](https://github.com/woboq/woboq_codebrowser).
173+
### Build Without Docker
191174

192-
- The following command builds PaddlePaddle, generates HTML pages from C++ source code, and writes HTML pages into `$HOME/woboq_out` on the host:
175+
Follow the *Dockerfile* in the paddlepaddle repo to set up your local dev environment and run:
193176

194177
```bash
195-
docker run -v $PWD:/paddle -v $HOME/woboq_out:/woboq_out -e "WITH_GPU=OFF" -e "WITH_AVX=ON" -e "WITH_TESTING=ON" -e "WOBOQ=ON" paddlepaddle/paddle:latest-dev
178+
./paddle/scripts/paddle_build.sh build
196179
```
197180

198-
- You can open the generated HTML files in your Web browser. Or, if you want to run a Nginx container to serve them for a wider audience, you can run:
181+
### Additional Tasks
199182

200-
```
201-
docker run -v $HOME/woboq_out:/usr/share/nginx/html -d -p 8080:80 nginx
183+
You can get the help menu for the build scripts by running with no options:
184+
185+
```bash
186+
./paddle/scripts/paddle_build.sh
187+
or ./paddle/scripts/paddle_docker_build.sh
202188
```

0 commit comments

Comments
 (0)