Skip to content
This repository was archived by the owner on Jul 1, 2023. It is now read-only.

Commit 68266f9

Browse files
texasmichelletexasmichelle
authored
Adjust structure of developer guide (#1164)
* Adjust structure of developer guide Make it clearer when to use each set of instructions. * Incorporate #1161 & #1165 * Address review feedback
1 parent b78c6af commit 68266f9

File tree

1 file changed

+188
-123
lines changed

1 file changed

+188
-123
lines changed

‎Documentation/Development.md

Lines changed: 188 additions & 123 deletions
Original file line numberDiff line numberDiff line change
@@ -1,41 +1,167 @@
1-
# Swift for TensorFlow Deep Learning Library
1+
# Swift for TensorFlow Developer Guide
22

3-
## Development
3+
## Requirements
44

5-
### Requirements
5+
* The latest Swift toolchain snapshot from [swift.org][swift]
66

7-
* The latest Swift toolchain snapshot from [swift.org](https://swift.org/download/#sanpshots).
7+
Swift for TensorFlow APIs can be built with either:
88

9-
### Building
9+
* CMake 3.16 or later from [cmake.org][cmake]
10+
* Swift Package Manager (included in the above toolchain)
1011

11-
### X10
12+
## Components
1213

13-
X10 provides the underlying lazy tensor implementation that is used for the
14-
tensor computation in the TensorFlow Swift APIs. This library builds atop of
15-
the TensorFlow library, using the XLA compiler to perform optimizations over
16-
the tensor computation.
14+
Building Swift for TensorFlow APIs involves two distinct components:
1715

18-
In the case that a prebuilt option is not available for your platform, note that
19-
by default the CMake based build will build a copy of X10 if it does not find
20-
one.
16+
1. X10
17+
2. Swift APIs
2118

22-
#### (Option 1) Use a prebuilt version of X10
19+
<details>
20+
<summary>What is X10?</summary>
21+
22+
> X10 provides the underlying lazy tensor implementation that is used for
23+
> tensor computations in the Swift for TensorFlow APIs. This library builds on
24+
> top of the TensorFlow library, using the XLA compiler to perform
25+
> optimizations.
26+
>
27+
> The X10 implementation consists of two halves:
28+
>
29+
> 1. [XLA Client](Sources/x10/xla_client): provides an abstract interface for
30+
> dispatching XLA computations on devices
31+
> 2. [X10](Sources/x10/xla_tensor): Lowering rules and tracing support for
32+
> Tensors
33+
34+
35+
</details>
36+
37+
38+
The two components can be built together (CMake only) or separately.
39+
40+
Follow the instructions below based on your preferred build tool (CMake or
41+
SwiftPM).
42+
43+
## Building With CMake
2344

24-
You can use a prebuilt version of the X10 library for building the TensorFlow
25-
Swift APIs package if you do not need to make changes to the X10 library
26-
implementation.
45+
With CMake, X10 and Swift APIs can be built either together or separately.
46+
47+
To determine the appropriate build path, first consider your platform and
48+
whether you are modifying X10.
49+
50+
* If you are modifying files in any subdirectory within **Sources/CX10** or
51+
**Sources/x10**, you are modifying X10 and must follow
52+
[**Option 1**](#option-1-build-x10-and-swift-apis-together).
53+
* If you are not modifying X10 and are on Windows or macOS or have previously
54+
built X10, use [**Option 2**](#option-2-use-a-prebuilt-version-of-x10).
55+
* Otherwise, use [**Option 1**](#option-1-build-x10-and-swift-apis-together).
56+
57+
> **Note:** In-tree builds are not supported.
58+
59+
> **Note:** To enable CUDA support, run `export TF_NEED_CUDA=1` before the
60+
steps below.
61+
62+
> **Note:** If `swiftc` is not in your `PATH`, you must specify the path to it
63+
using `-D CMAKE_Swift_COMPILER=`.
64+
65+
### Option 1: Build X10 and Swift APIs together
66+
67+
This command builds both X10 and Swift APIs in sequence. Ensure that you
68+
do not have the X10 modules in the toolchain that you are using to develop here
69+
(more explicitly, use a stock toolchain rather than a TensorFlow toolchain).
70+
71+
```shell
72+
cmake -B out -G Ninja -S swift-apis -D CMAKE_BUILD_TYPE=Release
73+
cmake --build out
74+
```
75+
76+
### Option 2: Use a prebuilt version of X10
77+
78+
If you are not modifying X10 or have already built it yourself using Option 1
79+
above, you can inform CMake where to find the build time components of the
80+
library (SDK contents).
2781

2882
There are prebuilt versions of the X10 library for certain platforms. If a
2983
prebuilt library is unavailable for your desired platform, you can build X10
30-
from source following the instructions below.
84+
from source following the instructions in [Option 1](#option-2-build-x10).
85+
86+
- [Windows 10 (x64)][windows10]
87+
- [macOS (x64)][macOS]
88+
89+
Note the location where you extract the prebuilt library. The path to these
90+
libraries is not fixed and depends on your machine setup.
91+
You should substitute the paths with the appropriate values. In the example
92+
commands below, we assume that the library is packaged in a traditional Unix
93+
style layout and placed in `/Library/tensorflow-2.4.0`.
94+
95+
Because the library name differs based on the platform, the following examples
96+
may help identify what the flags should look like for the target that you are
97+
building for.
98+
99+
macOS:
100+
101+
```shell
102+
cmake -B out -G Ninja -S swift-apis -D CMAKE_BUILD_TYPE=Release \
103+
-D X10_LIBRARY=/Library/tensorflow-2.4.0/usr/lib/libx10.dylib \
104+
-D X10_INCLUDE_DIRS=/Library/tensorflow-2.4.0/usr/include
105+
cmake --build out
106+
```
107+
108+
Windows:
109+
110+
```shell
111+
cmake -B out -G Ninja -S swift-apis -D CMAKE_BUILD_TYPE=Release \
112+
-D X10_LIBRARY=/Library/tensorflow-2.4.0/usr/lib/x10.lib \
113+
-D X10_INCLUDE_DIRS=/Library/tensorflow-2.4.0/usr/include
114+
cmake --build out
115+
```
116+
117+
Other Unix systems (e.g. Linux, BSD, Android, etc):
118+
119+
```shell
120+
cmake -B out -G Ninja -S swift-apis -D CMAKE_BUILD_TYPE=Release \
121+
-D X10_LIBRARY=/Library/tensorflow-2.4.0/usr/lib/libx10.so \
122+
-D X10_INCLUDE_DIRS=/Library/tensorflow-2.4.0/usr/include
123+
cmake --build out
124+
```
125+
126+
### Running tests
127+
128+
To run tests:
129+
130+
> **Note:** To view failure output, run `export CTEST_OUTPUT_ON_FAILURE=1`
131+
before running tests.
132+
133+
```shell
134+
cmake --build out --target test
135+
```
136+
137+
#### macOS
138+
139+
On macOS, passing `-D BUILD_TESTING=NO` is currently necessary to skip building
140+
tests. This avoids an error: `cannot load underlying module for 'XCTest'`.
141+
142+
```shell
143+
cmake -B out -G Ninja -S swift-apis -D CMAKE_BUILD_TYPE=Release \
144+
-D BUILD_TESTING=NO
145+
cmake --build out
146+
```
147+
148+
## Building With Swift Package Manager
31149

32-
- [Windows 10 (x64)](https://artprodeus21.artifacts.visualstudio.com/A8fd008a0-56bc-482c-ba46-67f9425510be/3133d6ab-80a8-4996-ac4f-03df25cd3224/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2NvbXBuZXJkL3Byb2plY3RJZC8zMTMzZDZhYi04MGE4LTQ5OTYtYWM0Zi0wM2RmMjVjZDMyMjQvYnVpbGRJZC80Mzc2OC9hcnRpZmFjdE5hbWUvdGVuc29yZmxvdy13aW5kb3dzLXg2NA2/content?format=zip)
33-
- [macOS (x64)](https://artprodeus21.artifacts.visualstudio.com/A8fd008a0-56bc-482c-ba46-67f9425510be/3133d6ab-80a8-4996-ac4f-03df25cd3224/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2NvbXBuZXJkL3Byb2plY3RJZC8zMTMzZDZhYi04MGE4LTQ5OTYtYWM0Zi0wM2RmMjVjZDMyMjQvYnVpbGRJZC80Mzc2OC9hcnRpZmFjdE5hbWUvdGVuc29yZmxvdy1kYXJ3aW4teDY00/content?format=zip)
150+
Building with SwiftPM involves building X10 and Swift APIs separately.
34151

35-
The location where you extract the prebuilt library is important to remember
36-
as it will be required in building the TensorFlow Swift APIs.
152+
### Building X10
37153

38-
#### (Option 2) Building X10
154+
To determine the appropriate build path, first consider your platform and
155+
whether you are modifying X10.
156+
157+
* If you are modifying files in any subdirectory within **Sources/CX10** or
158+
**Sources/x10**, you are modifying X10 and must follow
159+
[**Option 1**](#option-1-build-x10).
160+
* If you are on Windows or macOS and are not modifying X10, use
161+
[**Option 2**](#option-2-use-a-prebuilt-version-of-x10-1).
162+
* Otherwise, use [**Option 1**](#option-1-build-x10).
163+
164+
#### Option 1: Build X10
39165

40166
Although using the prebuilt libraries is more convenient, there may be some
41167
situations where you may need to build X10 yourself. These include:
@@ -44,21 +170,15 @@ situations where you may need to build X10 yourself. These include:
44170
2. You are attempting to make changes to the X10 implementation itself, and must
45171
therefore build a new version.
46172

47-
#### Requirements
173+
##### Requirements
48174

49175
* Bazel. This can be installed [manually][bazel] or with
50176
[Bazelisk][bazelisk]. You will need a version supported by TensorFlow
51177
(between `_TF_MIN_BAZEL_VERSION` and `_TF_MAX_BAZEL_VERSION` as specified in
52178
[tensorflow/configure.py][configure.py]).
53179
* Python3 with [numpy][numpy].
54180

55-
#### Building
56-
57-
The X10 implementation is distributed as part of the Swift for TensorFlow APIs
58-
repository. It consists of two halves:
59-
60-
1. [XLA Client](Sources/x10/xla_client): provides an abstract interface for dispatching XLA computations on devices
61-
2. [X10](Sources/x10/xla_tensor): Lowering rules and tracing support for Tensors
181+
##### Building
62182

63183
The library is designed to be built as part of the
64184
[tensorflow](https://github.com/tensorflow/tensorflow) build. As such, in
@@ -92,9 +212,12 @@ libraries.
92212
<summary>Windows Build Script</summary>
93213

94214

95-
*Note: This must be executed in the x64 Native Developer Command Prompt*
215+
*Note: This must be executed in the x64 Native Developer Command Prompt.*
96216

97-
*Note: You will either need to be running with elevated privilleges, have rights to create symbolic links and junctions, or have enabled developer mode to successfully create the directory junctions. You may alternatively copy the sources instead of creating a junction.*
217+
*Note: You will either need to be running with elevated privilleges, have
218+
rights to create symbolic links and junctions, or have enabled developer mode
219+
to successfully create the directory junctions. You may alternatively copy the
220+
sources instead of creating a junction.*
98221

99222
```cmd
100223
:: clone swift-apis
@@ -163,7 +286,9 @@ copy tensorflow\bazel-out\%VSCMD_ARG_TGT_ARCH%_windows-opt\bin\tensorflow\tensor
163286
<details>
164287
<summary>macOS/Linux Build Script</summary>
165288

166-
*Note: If you are unable to run bazel on macOS due to an error about an unverified developer due to System Integrity Protection (SIP), you can use `xattr -dr com.apple.quarantine bazel`*
289+
> **Note:** If you are unable to run bazel on macOS due to an error about an
290+
> unverified developer due to System Integrity Protection (SIP), you can use
291+
> `xattr -dr com.apple.quarantine bazel` to designate it as trusted.
167292
168293
```bash
169294
# clone swift-apis
@@ -222,19 +347,31 @@ cp swift-apis/Sources/x10/swift_bindings/xla_tensor_wrapper.h ${DESTDIR}/usr/inc
222347
```
223348
</details>
224349

225-
### (TensorFlow) Swift APIs
350+
#### Option 2: Use a prebuilt version of X10
351+
352+
You can use a prebuilt version of the X10 library for building the Swift for
353+
TensorFlow APIs package if you are on a supported platform and do not need to
354+
make changes to the X10 library implementation.
355+
356+
There are prebuilt versions of the X10 library for certain platforms. If a
357+
prebuilt library is unavailable for your desired platform, you can build X10
358+
from source following the instructions in [Option 1](#option-1-build-x10).
359+
360+
- [Windows 10 (x64)][windows10]
361+
- [macOS (x64)][macOS]
226362

227-
#### SwiftPM
363+
Note the location where you extract the prebuilt library, since it is required
364+
for building Swift for TensorFlow APIs.
228365

229-
*Note: Building with SwiftPM does not include changes to X10 modules.*
366+
### Building Swift APIs
230367

231-
Because X10 is a required component for building the TensorFlow Swift APIs,
232-
and the project does not build with Swift Package Manager, we must pass along
233-
the necessary information to the build. We need to pass two items to the
234-
build:
368+
> **Note:** This step requires pre-built X10 binaries. Building with SwiftPM
369+
> does not include changes to X10.
235370
236-
1. the location of the X10 & TensorFlow headers
237-
2. the location of the X10 & TensorFlow libraries
371+
SwiftPM requires two items:
372+
373+
1. The location of the X10 & TensorFlow headers.
374+
2. The location of the X10 & TensorFlow libraries.
238375

239376
The path to these libraries is not fixed and depends on your machine setup.
240377
You should substitute the paths with the appropriate values. In the example
@@ -245,107 +382,35 @@ style layout and placed in `/Library/tensorflow-2.4.0`.
245382
$ swift build -Xcc -I/Library/tensorflow-2.4.0/usr/include -Xlinker -L/Library/tensorflow-2.4.0/usr/lib
246383
```
247384

248-
```shell
249-
$ swift test -Xcc -I/Library/tensorflow-2.4.0/usr/include -Xlinker -L/Library/tensorflow-2.4.0/usr/lib
250-
```
385+
#### macOS
251386

252387
On macOS, in order to select the proper toolchain, the `TOOLCHAINS` environment
253388
variable can be used to modify the selected Xcode toolchain temporarily. The
254-
macOS (Xcode) toolchain distributed from [swift.org](https://swift.org) has a
389+
macOS (Xcode) toolchain distributed from [swift.org][swift] has a
255390
bundle identifier which can uniquely identify the toolchain to the system. The
256391
following attempts to determine the latest toolchain snapshot and extract the
257392
identifier for it.
258393

259394
```shell
260395
xpath 2>/dev/null $(find /Library/Developer/Toolchains ~/Library/Developer/Toolchains -type d -depth 1 -regex '.*/swift-DEVELOPMENT-SNAPSHOT-.*.xctoolchain | sort -u | tail -n 1)/Info.plist "/plist/dict/key[. = 'CFBundleIdentifier']/following-sibling::string[1]//text()"
261396
```
262-
263397
This allows one to build the package as:
264-
265398
```shell
266399
TOOLCHAINS=$(xpath 2>/dev/null $(find /Library/Developer/Toolchains ~/Library/Developer/Toolchains -type d -depth 1 -regex '.*/swift-DEVELOPMENT-SNAPSHOT-.*.xctoolchain | sort -u | tail -n 1)/Info.plist "/plist/dict/key[. = 'CFBundleIdentifier']/following-sibling::string[1]//text()") swift build -Xswiftc -DTENSORFLOW_USE_STANDARD_TOOLCHAIN -Xcc -I/Library/tensorflow-2.4.0/usr/include -Xlinker -L/Library/tensorflow-2.4.0/usr/lib
267400
```
268401

269-
#### CMake
270-
271-
*Note: In-tree builds are not supported.*
272-
273-
*Note: To enable CUDA support, run `export TF_NEED_CUDA=1` before the steps below.*
274-
275-
*Note: If `swiftc` is not in your `PATH`, you must specify the path to it using
276-
`-D CMAKE_Swift_COMPILER=`.*
277-
278-
By default, CMake will build X10 if it is not informed of where the library is
279-
available. If you wish to build X10 as part of the CMake build, ensure that you
280-
do not have the X10 modules in the toolchain that you are using to develop here
281-
(more explicitly, use a stock toolchain rather than a TensorFlow toolchain).
282-
283-
```shell
284-
cmake -B out -G Ninja -S swift-apis -D CMAKE_BUILD_TYPE=Release
285-
cmake --build out
286-
```
287-
288-
If you have a prebuilt version of the X10 library that you wish to use (i.e. you
289-
are using one of the prebuilt libraries referenced above or have built it
290-
yourself above), you can inform CMake where to find the build time components of
291-
the library (SDK contents) and avoid building X10.
292-
293-
The path to these libraries is not fixed and depends on your machine setup.
294-
You should substitute the paths with the appropriate values. In the example
295-
commands below, we assume that the library is packaged in a traditional Unix
296-
style layout and placed in `/Library/tensorflow-2.4.0`.
297-
298-
Because the library name differs based on the platform, the following examples
299-
may help identify what the flags should look like for the target that you are
300-
building for.
301-
302-
macOS:
303-
304-
```shell
305-
-D X10_LIBRARY=/Library/tensorflow-2.4.0/usr/lib/libx10.dylib -D X10_INCLUDE_DIRS=/Library/tensorflow-2.4.0/usr/include
306-
```
307-
308-
Windows:
309-
310-
```shell
311-
-D X10_LIBRARY=/Library/tensorflow-2.4.0/usr/lib/x10.lib -D X10_INCLUDE_DIRS=/Library/tensorflow-2.4.0/usr/include
312-
```
313-
314-
Other Unix systems (e.g. Linux, BSD, Android, etc):
315-
316-
```shell
317-
-D X10_LIBRARY=/Library/tensorflow-2.4.0/usr/lib/libx10.so -D X10_INCLUDE_DIRS=/Library/tensorflow-2.4.0/usr/include
318-
```
402+
### Running tests
319403

320404
To run tests:
321405

322-
*Note: To view failure output, run `export CTEST_OUTPUT_ON_FAILURE=1` before
323-
running tests.*
324-
325406
```shell
326-
cmake --build out --target test
327-
```
328-
329-
If you are not intending to develop X10, you can reduce the build times by
330-
using the bundled X10 in the toolchain using
331-
`-D USE_BUNDLED_X10=YES -D USE_BUNDLED_CTENSORFLOW=YES`:
332-
333-
```shell
334-
cmake -B out -D USE_BUNDLED_CTENSORFLOW=YES -D USE_BUNDLED_X10=YES -G Ninja -S swift-apis
335-
cmake --build out
336-
cmake --build out --target test
337-
```
338-
339-
#### macOS
340-
341-
On macOS, passing `-D BUILD_TESTING=NO` is currently necessary to skip building
342-
tests. This avoids an error: `cannot load underlying module for 'XCTest'`.
343-
344-
```shell
345-
cmake -B out -D USE_BUNDLED_CTENSORFLOW=YES -D USE_BUNDLED_X10=YES -D BUILD_TESTING=NO -G Ninja -S swift-apis
346-
cmake --build out
407+
$ swift test -Xcc -I/Library/tensorflow-2.4.0/usr/include -Xlinker -L/Library/tensorflow-2.4.0/usr/lib
347408
```
348409

410+
[swift]: https://swift.org/download/#snapshots
411+
[cmake]: https://www.cmake.org/download
412+
[windows10]: https://artprodeus21.artifacts.visualstudio.com/A8fd008a0-56bc-482c-ba46-67f9425510be/3133d6ab-80a8-4996-ac4f-03df25cd3224/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2NvbXBuZXJkL3Byb2plY3RJZC8zMTMzZDZhYi04MGE4LTQ5OTYtYWM0Zi0wM2RmMjVjZDMyMjQvYnVpbGRJZC80Mzc2OC9hcnRpZmFjdE5hbWUvdGVuc29yZmxvdy13aW5kb3dzLXg2NA2/content?format=zip
413+
[macOS]: https://artprodeus21.artifacts.visualstudio.com/A8fd008a0-56bc-482c-ba46-67f9425510be/3133d6ab-80a8-4996-ac4f-03df25cd3224/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2NvbXBuZXJkL3Byb2plY3RJZC8zMTMzZDZhYi04MGE4LTQ5OTYtYWM0Zi0wM2RmMjVjZDMyMjQvYnVpbGRJZC80Mzc2OC9hcnRpZmFjdE5hbWUvdGVuc29yZmxvdy1kYXJ3aW4teDY00/content?format=zip
349414
[bazel]: https://docs.bazel.build/versions/master/install.html
350415
[bazelisk]: https://github.com/bazelbuild/bazelisk
351416
[configure.py]: https://github.com/tensorflow/tensorflow/blob/master/configure.py

0 commit comments

Comments
 (0)