Skip to content

Commit 0dc8d56

Browse files
jvddjvdd
authored
🕵️ testing + 🏗️ build & 🚀 release (#3)
* 🕵️ first tests * 🙈 * 🙏 add build * 🌔 nightly rust toolchain * 🙈 typo * 🚀 add release * 🙃 bumb version * 🧹 update readme * 🖊️ update readme
1 parent b83380d commit 0dc8d56

File tree

9 files changed

+330
-21
lines changed

9 files changed

+330
-21
lines changed

‎.github/workflows/ci-tsdownsample.yml‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,4 +40,160 @@ jobs:
4040
- run: make lint # Lint Python & Rust
4141
- run: make mypy # Type check Python
4242

43+
Test:
44+
runs-on: ${{ matrix.os }}
45+
strategy:
46+
fail-fast: false
47+
matrix:
48+
os: ['windows-latest', 'macOS-latest', 'ubuntu-latest']
49+
rust: ['nightly'] # ['stable', 'beta']
50+
python-version: ['3.7', '3.8', '3.9', '3.10', '3.11']
51+
52+
env:
53+
PYTHON: ${{ matrix.python-version }}
54+
55+
steps:
56+
- uses: actions/checkout@v3
57+
- uses: actions/setup-python@v4
58+
with:
59+
python-version: ${{ matrix.python-version }}
60+
- run: pip install -r tests/requirements.txt
61+
62+
- name: Install Rust toolchain
63+
uses: actions-rs/toolchain@v1
64+
with:
65+
profile: minimal
66+
toolchain: nightly
67+
components: clippy, rustfmt
68+
- name: Setup Rust
69+
run: |
70+
rustup update nightly --no-self-update
71+
rustup default nightly
72+
- name: Cache rust
73+
uses: Swatinem/rust-cache@v2
74+
75+
- name: install develop version
76+
run: make install
77+
# python setup.py develop
78+
79+
- run: pip install -r tests/requirements.txt
80+
81+
- run: pip freeze
82+
83+
- run: make test # Test Python
84+
85+
- name: Upload coverage to Codecov
86+
uses: codecov/codecov-action@v3
87+
88+
Build:
89+
# Perhaps smth more in line with this https://github.com/messense/crfs-rs/blob/main/.github/workflows/Python.yml
90+
name: build on ${{ matrix.os }} (${{ matrix.target }} - ${{ matrix.manylinux || 'auto' }})
91+
# only run on push to main and on release
92+
if: "success() && (startsWith(github.ref, 'refs/tags/') || github.ref == 'refs/heads/main' || contains(github.event.pull_request.labels.*.name, 'Full Build'))"
93+
strategy:
94+
fail-fast: false
95+
matrix:
96+
os: [ubuntu, macos, windows]
97+
target: [x86_64, aarch64]
98+
manylinux: [auto]
99+
include:
100+
- os: windows
101+
ls: dir
102+
- os: windows
103+
ls: dir
104+
target: i686
105+
python-architecture: x86
106+
- os: macos
107+
target: aarch64
108+
- os: ubuntu
109+
target: i686
110+
# GCC 4.8.5 in manylinux2014 container doesn't support c11 atomic
111+
# we use manylinux_2_24 container for aarch64 and armv7 targets instead,
112+
- os: ubuntu
113+
target: aarch64
114+
container: messense/manylinux_2_24-cross:aarch64
115+
- os: ubuntu
116+
target: armv7
117+
container: messense/manylinux_2_24-cross:armv7
118+
- os: ubuntu
119+
target: ppc64le
120+
container: messense/manylinux_2_24-cross:ppc64le
121+
- os: ubuntu
122+
target: s390x
123+
container: messense/manylinux_2_24-cross:s390x
124+
# musllinux
125+
- os: ubuntu
126+
target: x86_64
127+
manylinux: musllinux_1_1
128+
- os: ubuntu
129+
target: aarch64
130+
manylinux: musllinux_1_1
131+
exclude:
132+
# this fails
133+
- os: windows
134+
target: aarch64
135+
136+
runs-on: ${{ matrix.os }}-latest
137+
steps:
138+
- uses: actions/checkout@v3
139+
140+
- name: set up python
141+
uses: actions/setup-python@v4
142+
# with:
143+
# python-version: '3.11'
144+
# architecture: ${{ matrix.python-architecture || 'x64' }}
145+
146+
- name: build sdist
147+
if: ${{ matrix.os == 'ubuntu' && matrix.target == 'x86_64' && matrix.manylinux == 'auto' }}
148+
uses: PyO3/maturin-action@v1
149+
with:
150+
command: sdist
151+
args: --out dist
152+
153+
- name: build wheels
154+
uses: PyO3/maturin-action@v1
155+
with:
156+
rust-toolchain: nightly
157+
target: ${{ matrix.target }}
158+
manylinux: ${{ matrix.manylinux || 'auto' }}
159+
container: ${{ matrix.container }}
160+
args: --release --out dist --interpreter ${{ matrix.interpreter || '3.7 3.8 3.9 3.10 3.11' }}
161+
162+
- run: ${{ matrix.ls || 'ls -lh' }} dist/
163+
164+
- uses: actions/upload-artifact@v3
165+
with:
166+
name: pypi_files
167+
path: dist
168+
169+
Release:
170+
needs: [Lint_and_Check, Test, Build]
171+
if: "success() && startsWith(github.ref, 'refs/tags/')"
172+
runs-on: ubuntu-latest
173+
174+
steps:
175+
- uses: actions/checkout@v3
176+
177+
- name: set up python
178+
uses: actions/setup-python@v4
179+
# with:
180+
# python-version: '3.10'
181+
182+
- run: pip install -U twine
183+
184+
- name: get dist artifacts
185+
uses: actions/download-artifact@v3
186+
with:
187+
name: pypi_files
188+
path: dist
189+
190+
- run: twine check dist/*
191+
192+
- name: upload to pypi
193+
run: twine upload dist/*
194+
env:
195+
TWINE_USERNAME: __token__
196+
TWINE_PASSWORD: ${{ secrets.pypi_token }}
197+
43198
# https://github.com/samuelcolvin/rtoml/blob/main/.github/workflows/ci.yml
199+
# https://github.com/messense/rjmespath-py/blob/main/.github/workflows/CI.yml

‎Makefile‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
isort = isort tsdownsample tests
33
black = black tsdownsample tests
44

5+
install:
6+
pip install -r requirements.txt
7+
pip install -e .
58

69
.PHONY: format
710
format:
@@ -29,8 +32,14 @@ lint: lint-python lint-rust
2932
mypy:
3033
mypy tsdownsample
3134

35+
36+
.PHONY: test
37+
test:
38+
pytest --cov=tsdownsample --cov-report=term-missing --cov-report=html --cov-report=xml
39+
40+
3241
.PHONY: all
33-
all: lint mypy testcov
42+
all: lint mypy test
3443

3544
.PHONY: clean
3645
clean:

‎README.md‎

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -9,17 +9,31 @@
99

1010
## Features ✨
1111

12-
* **Fast**: written in rust with pyo3 bindings
12+
* **Fast**: written in rust with PyO3 bindings
1313
- leverages optimized [argminmax](https://github.com/jvdd/argminmax) - which is SIMD accelerated with runtime feature detection
1414
- scales linearly with the number of data points
15-
- scales multi-threaded with rayon (rust)
15+
- multithreaded with Rayon (in Rust)
16+
<details>
17+
<summary><i>Why we do not use Python multiprocessing</i></summary>
18+
Citing the <a href="https://pyo3.rs/v0.17.3/parallelism.html">PyO3 docs on parallelism</a>:<br>
19+
<blockquote>
20+
CPython has the infamous Global Interpreter Lock, which prevents several threads from executing Python bytecode in parallel. This makes threading in Python a bad fit for CPU-bound tasks and often forces developers to accept the overhead of multiprocessing.
21+
</blockquote>
22+
In Rust - which is a compiled language - there is no GIL, so CPU-bound tasks can be parallelized (with <a href="https://github.com/rayon-rs/rayon">Rayon</a>) with little to no overhead.
23+
</details>
1624
* **Efficient**: memory efficient
1725
- works on views of the data (no copies)
1826
- no intermediate data structures are created
1927
* **Flexible**: works on any type of data
20-
- supported datatypes are `f16`, `f32`, `f64`, `i16`, `i32`, `i64`, `u16`, `u32`, `u64`
21-
*!! 🚀 `f16` [argminmax](https://github.com/jvdd/argminmax) is 200-300x faster than numpy*
22-
* **Easy to use**: simple API
28+
- supported datatypes are `f16`, `f32`, `f64`, `i16`, `i32`, `i64`, `u16`, `u32`, `u64`
29+
<details>
30+
<summary><i>!! 🚀 <code>f16</code> <a href="https://github.com/jvdd/argminmax">argminmax</a> is 200-300x faster than numpy</i></summary>
31+
In contrast with all other data types above, <code>f16</code> is *not* hardware supported (i.e., no instructions for f16) by most modern CPUs!! <br>
32+
🐌 Programming languages facilitate support for this datatype by either (i) upcasting to `f32` or (ii) using a software implementation. <br>
33+
💡 As for argminmax, only comparisons are needed - and thus no arithmetic operations - creating a <ins>symmetrical ordinal mapping from <code>f16</code> to <code>i16</code></ins> is sufficient. This mapping allows to use the hardware supported scalar and SIMD <code>i16</code> instructions - while not producing any memory overhead 🎉 <br>
34+
<i>More details are described in <a href="https://github.com/jvdd/argminmax/pull/1">argminmax PR #1</a>.</i>
35+
</details>
36+
* **Easy to use**: simple & flexible API
2337

2438
## Install
2539

@@ -33,15 +47,15 @@ pip install tsdownsample
3347
## Usage
3448

3549
```python
36-
import tsdownsample as tsds
50+
from tsdownsample import MinMaxLTTBDowsampler
3751
import pandas as pd; import numpy as np
3852

3953
# Create a time series
4054
y = np.random.randn(10_000_000)
4155
s = pd.Series(y)
4256

4357
# Downsample to 1000 points
44-
s_ds = tsds.minmaxlttb(s, n_out=1000)
58+
s_ds = MinMaxLTTBDownsampler.downsample(s, n_out=1000)
4559
```
4660

4761
---

‎pyproject.toml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
[build-system]
2-
requires = ["setuptools", "wheel", "setuptools-rust"]
2+
requires = ["setuptools", "wheel", "setuptools-rust"] #, "maturin"]
3+
# build-backend = "maturin" # TODO: this works as well!
34

45
[project]
56
name = "tsdownsample"
6-
version = "0.1.0a1"
7+
version = "0.1.0a2"
78
requires-python = ">=3.7"
89
description = "Time series downsampling in rust"
910
authors = [{name = "Jeroen Van Der Donckt"}]
@@ -15,7 +16,6 @@ classifiers = [
1516
'License :: OSI Approved :: MIT License',
1617
'Intended Audience :: Developers',
1718
'Programming Language :: Python :: 3',
18-
'Programming Language :: Python :: 3.6', # TODO?
1919
'Programming Language :: Python :: 3.7',
2020
'Programming Language :: Python :: 3.8',
2121
'Programming Language :: Python :: 3.9',

‎requirements.txt‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
numpy
22
pandas
33
# Build dependencies
4-
setuptools_rust
4+
setuptools_rust
5+
# maturin

‎tests/requirements.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
pytest
2+
pytest-cov

0 commit comments

Comments
 (0)