Skip to content

Commit 89d15da

Browse files
chipkentchipkent
authored
Use Deephaven as a Python Library (#63)
* Progress towards using DHaaL * Trying to build whl files * Trying to build whl files * Working to fix sphinx * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * added deephaven_server to docs. * Refactored local docker. * Updated documentation. * Updated documentation. * Updated documentation. * Renamed the wheel version number for dev branches to make twine happy. * Set pypi publishing credential. * Updated readme with run instructions * Updated readme with better run instructions * Updated readme * Fixed readme bugs * Fixed docs on port exposure * Handle blocking contract error conditions properly. * Better error reporting on contract details problems. * Fixed example code.
1 parent 8860c46 commit 89d15da

23 files changed

+405
-321
lines changed

‎.github/workflows/Dockerfile‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
2+
FROM ubuntu:22.04
3+
4+
RUN apt update && \
5+
apt install -y openjdk-11-jdk python3-pip
6+
7+
ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/
8+
9+
COPY ./wheels /wheels
10+
11+
RUN pip3 install /wheels/*.whl && \
12+
rm -rf /wheels/
13+
14+
CMD python3

‎.github/workflows/build-and-publish.yml‎

Lines changed: 126 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,16 @@
1-
name: Build and publish to PyPI
1+
name: Build & Publish
22

3-
on: push
3+
on:
4+
pull_request:
5+
branches:
6+
- main
7+
push:
8+
branches:
9+
- main
410

511
jobs:
6-
build-and-publish:
7-
name: Build and publish to PyPI
12+
build-whl:
13+
name: Build WHL
814
runs-on: ubuntu-20.04
915
steps:
1016
- uses: actions/checkout@master
@@ -22,7 +28,8 @@ jobs:
2228
TAG_NAME=${{ github.event.release.tag_name }}
2329
if [ -z "${TAG_NAME}" ]
2430
then
25-
echo "::set-output name=version::development"
31+
PR_NUMBER=$(echo $GITHUB_REF | awk 'BEGIN { FS = "/" } ; { print $3 }')
32+
echo "::set-output name=version::0.0.0.dev${PR_NUMBER}"
2633
else
2734
echo "::set-output name=version::${{ github.event.release.tag_name }}"
2835
fi
@@ -31,15 +38,122 @@ jobs:
3138
DH_IB_VERSION: ${{ steps.version.outputs.version }}
3239
run: |
3340
python -m build
34-
- name: Archive production artifacts
41+
- name: Archive build artifacts
3542
uses: actions/upload-artifact@v2
3643
with:
37-
name: Artifacts
44+
name: wheels
3845
path: |
3946
dist/*
40-
#TODO: publish
41-
# - name: Publish
42-
# if: xxxx
43-
# python -m twine upload --repository testpypi dist/*
4447
45-
48+
publish-whl:
49+
name: Publish WHL
50+
runs-on: ubuntu-20.04
51+
needs: [build-whl]
52+
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
53+
steps:
54+
- name: Download Build Artifacts
55+
uses: actions/download-artifact@v3
56+
with:
57+
name: build-artifacts
58+
path: wheel/
59+
- name: Publish WHL to PyPi
60+
uses: pypa/gh-action-pypi-publish@release/v1
61+
with:
62+
user: __token__
63+
password: ${{ secrets.DEEPHAVENIB_PYPI_TOKEN }}
64+
packages_dir: wheel/
65+
66+
67+
build-sphinx:
68+
name: Build Sphinx
69+
runs-on: ubuntu-20.04
70+
needs: [build-whl]
71+
steps:
72+
- uses: actions/checkout@v1
73+
- name: Apt installs
74+
run: |
75+
sudo apt update
76+
sudo apt install -y openjdk-11-jdk
77+
- name: Pip installs
78+
run: pip3 install --upgrade sphinx==4.2.0 sphinx-autodoc-typehints furo==2021.10.9
79+
- name: Download wheels
80+
uses: actions/download-artifact@v3
81+
with:
82+
name: wheels
83+
- name: Install Whl
84+
run: pip3 install *.whl
85+
- name: Run Sphinx
86+
working-directory: ./sphinx
87+
env:
88+
JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
89+
run: |
90+
make html
91+
touch build/html/.nojekyll
92+
- name: Archive Sphinx artifacts
93+
uses: actions/upload-artifact@v1
94+
with:
95+
name: documentation-html
96+
path: sphinx/build/html/
97+
98+
publish-sphinx:
99+
name: Publish Sphinx
100+
runs-on: ubuntu-20.04
101+
needs: [build-sphinx]
102+
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
103+
steps:
104+
- name: Download Sphinx Artifacts
105+
uses: actions/download-artifact@v3
106+
with:
107+
name: documentation-html
108+
path: html/
109+
- name: Deploy Sphinx docs to gh-pages
110+
uses: JamesIves/github-pages-deploy-action@v4.2.3
111+
with:
112+
branch: gh-pages
113+
folder: html/
114+
115+
docker:
116+
name: Build and Publish Docker
117+
runs-on: ubuntu-20.04
118+
needs: [build-whl]
119+
permissions:
120+
contents: read
121+
packages: write
122+
steps:
123+
- name: Checkout repository
124+
uses: actions/checkout@v2
125+
- name: Get branch name
126+
id: branch-name
127+
uses: tj-actions/branch-names@v5.2
128+
- name: Download wheels
129+
uses: actions/download-artifact@v3
130+
with:
131+
name: wheels
132+
path: wheels/
133+
- name: Log in to the Container registry
134+
uses: docker/login-action@v1.10.0
135+
with:
136+
registry: ghcr.io
137+
username: ${{ github.actor }}
138+
password: ${{ secrets.GITHUB_TOKEN }}
139+
- name: Extract metadata (tags, labels) for Docker
140+
id: meta-base
141+
uses: docker/metadata-action@v4.0.1
142+
with:
143+
images: ghcr.io/${{ github.repository }}
144+
tags: |
145+
type=schedule
146+
type=ref,event=branch
147+
type=ref,event=tag
148+
type=ref,event=pr
149+
type=raw,${{ steps.branch-name.outputs.current_branch }}
150+
- name: Build and push Docker image
151+
uses: docker/build-push-action@v2.7.0
152+
with:
153+
context: .
154+
file: ./.github/workflows/Dockerfile
155+
push: true
156+
tags: ${{ steps.meta-base.outputs.tags }}
157+
labels: ${{ steps.meta-base.outputs.labels }}
158+
env:
159+
IMAGE_NAME: ${{ github.repository }}

‎.github/workflows/docker-cleanup.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Delete Docker images after PR merge
2+
#
3+
4+
name: 'Clean up Docker images from PR'
5+
6+
on:
7+
pull_request:
8+
types: [closed]
9+
10+
jobs:
11+
purge-image:
12+
name: Delete images from ghcr.io
13+
runs-on: ubuntu-20.04
14+
steps:
15+
- name: Delete image (Base)
16+
uses: chipkent/action-cleanup-package@v1.0.2
17+
with:
18+
package-name: ${{ github.event.repository.name }}-base
19+
tag: pr-${{ github.event.pull_request.number }}
20+
github-token: ${{ secrets.CI_ACTION_TOKEN }}
21+
- name: Delete image (Downloader)
22+
uses: chipkent/action-cleanup-package@v1.0.2
23+
with:
24+
package-name: ${{ github.event.repository.name }}-downloader
25+
tag: pr-${{ github.event.pull_request.number }}
26+
github-token: ${{ secrets.CI_ACTION_TOKEN }}

‎.github/workflows/sphinx.yml‎

Lines changed: 0 additions & 42 deletions
This file was deleted.

‎.gitignore‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
11
.idea
22
venv
33
dist
4-
src/deephaven_ib.egg-info
4+
src/deephaven_ib.egg-info
5+
docker/data
6+
docker/*/data
7+
docker/*/build

‎README.md‎

Lines changed: 63 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
21
# deephaven-ib
32

43
![Deephaven Data Labs Logo](docs/assets/Deephaven-Logo-Wordmark-Community-OnLight.png)
@@ -174,33 +173,80 @@ upper right corner. ![](docs/assets/config-gear.png)
174173
1) [For Paper Trading] Log into the [Interactive Brokers Web Interface](https://interactivebrokers.com/).
175174
1) [For Paper Trading] In the [Interactive Brokers Web Interface](https://interactivebrokers.com/), navigate to `Account->Settings->Paper Trading Account` and make sure that "Share real-time market data subscriptions with paper trading account?" is set to true.
176175

176+
177177
## Launch
178178
To launch the system:
179179

180+
### Launch with Docker
181+
182+
This is the most tested way to launch.
183+
180184
1) Launch [IB Trader Workstation (TWS)](https://www.interactivebrokers.com/en/trading/tws.php).
181-
1) Accept incoming connections to [IB Trader Workstation (TWS)](https://www.interactivebrokers.com/en/trading/tws.php). (May not be required for all sessions.)
185+
2) Accept incoming connections to [IB Trader Workstation (TWS)](https://www.interactivebrokers.com/en/trading/tws.php). (May not be required for all sessions.)
182186
![](docs/assets/allow-connections.png)
183-
1) Build the Docker images:
187+
3) Create a directory for your data and scripts
188+
```bash
189+
mkdir data
190+
```
191+
4) Launch the system (Option 1):
192+
* On Mac:
184193
```bash
185-
./docker/deephaven_ib_docker.sh build --dh-version <deephaven_version>
194+
git clone git@github.com:deephaven-examples/deephaven-ib.git
195+
cd deephaven-ib/docker/dev/build.sh
196+
# Set jvm_args to the desired JVM memory for Deephaven
197+
docker run -it -v data:/data -p 10000:10000 deephaven-examples/deephaven-ib:dev python3 -i -c "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"
186198
```
187-
1) Launch the system:
199+
* On other platforms:
188200
```bash
189-
./docker/deephaven_ib_docker.sh up --dh-version <deephaven_version>
201+
# Set jvm_args to the desired JVM memory for Deephaven
202+
docker run -it -v data:/data -p 10000:10000 ghcr.io/deephaven-examples/deephaven-ib python3 -i -c "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"
190203
```
191-
1) Launch the [Deephaven IDE](https://github.com/deephaven/deephaven-core/blob/main/README.md#run-deephaven-ide) by navigating to [http://localhost:10000/ide/](http://localhost:10000/ide/) in a browser.
204+
5) Launch the system and execute a custom script (Option 2):
205+
* On Mac:
206+
```bash
207+
git clone git@github.com:deephaven-examples/deephaven-ib.git
208+
cd deephaven-ib/docker/dev/build.sh
209+
# your_script.py must begin with: "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"
210+
# Set jvm_args to the desired JVM memory for Deephaven
211+
cp path/to/your_script.py data/your_script.py
212+
docker run -it -v data:/data -p 10000:10000 deephaven-examples/deephaven-ib:dev python3 -i /data/your_script.py
213+
```
214+
* On other platforms:
215+
```bash
216+
# your_script.py must begin with: "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"
217+
# Set jvm_args to the desired JVM memory for Deephaven
218+
cp path/to/your_script.py data/your_script.py
219+
docker run -it -v data:/data -p 10000:10000 ghcr.io/deephaven-examples/deephaven-ib python3 -i /data/your_script.py
220+
```
221+
7) Launch the [Deephaven IDE](https://github.com/deephaven/deephaven-core/blob/main/README.md#run-deephaven-ide) by navigating to [http://localhost:10000/ide/](http://localhost:10000/ide/) in a browser.
192222

193-
## Shutdown
194-
To shut down the system:
195-
```bash
196-
./docker/deephaven_ib_docker.sh down --dh-version <deephaven_version>
197-
```
223+
### Launch with a local installation (No Docker)
198224

199-
## Help
200-
To get help on running the system:
201-
```bash
202-
./docker/deephaven_ib_docker.sh help
203-
```
225+
> **_NOTE:_** Deephaven pip install does not yet supported on all architectures. This launch should work on Linux (AMD64 and ARM64) and Windows WSL. It is not yet supported on Windows without WSL or Mac. For these architectures, you should use the Docker installation. As soon as Deephaven supports these architectures for pip, [deephaven-ib](https://github.com/deephaven-examples/deephaven-ib) will work.
226+
227+
It is possible to use [deephaven-ib](https://github.com/deephaven-examples/deephaven-ib) without docker, but this is a
228+
new feature and has not been well tested. To do this:
229+
1) Launch [IB Trader Workstation (TWS)](https://www.interactivebrokers.com/en/trading/tws.php).
230+
2) Accept incoming connections to [IB Trader Workstation (TWS)](https://www.interactivebrokers.com/en/trading/tws.php). (May not be required for all sessions.)
231+
![](docs/assets/allow-connections.png)
232+
3) Install [deephaven-ib](https://github.com/deephaven-examples/deephaven-ib):
233+
```bash
234+
pip3 install deephaven-ib
235+
```
236+
4) Install Java 11 and set the appropriate `JAVA_HOME` environment variable.
237+
5) Launch the system (Option 1):
238+
```bash
239+
# Set jvm_args to the desired JVM memory for Deephaven
240+
python3 -i -c "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"
241+
```
242+
6) Launch the system and execute a custom script (Option 2):
243+
```bash
244+
# your_script.py must begin with: "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"
245+
# Set jvm_args to the desired JVM memory for Deephaven
246+
python3 -i /data/your_script.py
247+
```
248+
7) Launch the [Deephaven IDE](https://github.com/deephaven/deephaven-core/blob/main/README.md#run-deephaven-ide) by navigating to [http://localhost:10000/ide/](http://localhost:10000/ide/) in a browser.
249+
8) Use `host=localhost` for the hostname in the examples
204250

205251
# Use deephaven-ib
206252

‎docker/Dockerfile‎

Lines changed: 0 additions & 24 deletions
This file was deleted.

0 commit comments

Comments
 (0)