Skip to content
This repository was archived by the owner on Sep 19, 2023. It is now read-only.

Commit cc3ba34

Browse files
patkivikrampatkivikram
authored
Merge pull request #1 from wbarnha/major-update
Major update with changes from other python-rocksdb forks
2 parents e6bf958 + 634322d commit cc3ba34

40 files changed

+2754
-716
lines changed

‎.github/workflows/build.yml‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# vim:ts=2:sw=2:et:ai:sts=2
2+
name: 'Build'
3+
4+
on:
5+
pull_request:
6+
push:
7+
branches:
8+
- 'main'
9+
- 'force_ci/all/**' # For development, forcing all workflows to run.
10+
- 'force_ci/build/**' # For debugging and/or only forcing this workflow.
11+
12+
jobs:
13+
# Build the RocksDB C library and cache the result.
14+
librocksdb:
15+
name: 'Build librocksdb'
16+
runs-on: ${{ matrix.os }}
17+
env:
18+
LIBROCKSDB_PATH: /opt/rocksdb-${{ matrix.rocksdb_ver }}
19+
strategy:
20+
matrix:
21+
os: [ubuntu-latest]
22+
rocksdb_ver: ['v6.29.3', 'v6.25.3', 'v6.11.4']
23+
24+
steps:
25+
- uses: actions/cache@v2
26+
id: cache-librocksdb
27+
with:
28+
key: ${{ matrix.os }}-librocksdb-${{ matrix.rocksdb_ver }}
29+
path: ${{ env.LIBROCKSDB_PATH }}
30+
31+
- name: 'Install dependencies'
32+
if: steps.cache-librocksdb.outputs.cache-hit != 'true'
33+
run: >
34+
sudo apt install -y libsnappy-dev libbz2-dev liblz4-dev libz-dev
35+
libgflags-dev libzstd-dev
36+
37+
- name: 'Clone & build RocksDB ${{ matrix.rocksdb_ver }}'
38+
if: steps.cache-librocksdb.outputs.cache-hit != 'true'
39+
run: >
40+
git clone https://github.com/facebook/rocksdb --depth 1
41+
--branch ${{ matrix.rocksdb_ver }} ${{ env.LIBROCKSDB_PATH }} &&
42+
pushd ${{ env.LIBROCKSDB_PATH }} &&
43+
CXXFLAGS='-flto -Os -s' PORTABLE=1 make shared_lib -j 4 &&
44+
popd
45+
46+
test:
47+
name: 'Build and test python-rocksdb'
48+
needs: librocksdb
49+
runs-on: ${{ matrix.os }}
50+
env:
51+
LIBROCKSDB_PATH: /opt/rocksdb-${{ matrix.rocksdb_ver }}
52+
strategy:
53+
matrix:
54+
os: [ubuntu-latest]
55+
py_ver: ['3.7', '3.8', '3.9']
56+
rocksdb_ver: ['v6.29.3', 'v6.25.3', 'v6.11.4']
57+
58+
steps:
59+
- uses: actions/checkout@v2
60+
name: 'Checkout source repository'
61+
62+
- uses: actions/setup-python@v2
63+
name: 'Set up Python ${{ matrix.py_ver }}'
64+
with:
65+
python-version: ${{ matrix.py_ver }}
66+
67+
- name: 'Install C libraries'
68+
# XXX(Tina): The non-development versions are sufficient, but package
69+
# names are difficult to predict.
70+
run: >
71+
sudo apt install -y libsnappy-dev libbz2-dev liblz4-dev libz-dev
72+
libgflags-dev libzstd-dev
73+
74+
# Recover the pre-built C library.
75+
- uses: actions/cache@v2
76+
id: cache-librocksdb
77+
with:
78+
key: ${{ matrix.os }}-librocksdb-${{ matrix.rocksdb_ver }}
79+
path: ${{ env.LIBROCKSDB_PATH }}
80+
81+
- name: 'Install RocksDB ${{ matrix.rocksdb_ver }}'
82+
if: steps.cache-librocksdb.outputs.cache-hit == 'true'
83+
# DO NOT FORGET to call `ldconfig`!
84+
run: |
85+
pushd ${{ env.LIBROCKSDB_PATH }} &&
86+
sudo make install-shared &&
87+
sudo ldconfig &&
88+
popd
89+
90+
- name: Build and install python-rocksdb
91+
# Use `pip install` instead of `setup.py` so build-dependencies from
92+
# `pyproject.toml` are installed, in particular `Cython`, without which
93+
# the build fails in confusing ways.
94+
run: |
95+
python3 -m pip install '.[test]'
96+
97+
- name: Run tests
98+
# Use `--pyargs` to interpret parameter as module to import, not as a
99+
# path, and do not use `python3 -m pytest`. This way we prevent
100+
# importing the module from the current directory instead of the
101+
# installed package, and failing when it cannot find the shared
102+
# library.
103+
run: |
104+
pytest --pyargs rocksdb

‎.github/workflows/debian.yml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
name: debian
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches:
7+
- 'main'
8+
- 'force_ci/all/**' # For development, forcing all workflows to run.
9+
- 'force_ci/debian/**' # For debugging and/or only forcing this workflow.
10+
11+
jobs:
12+
debian-build:
13+
name: ${{ matrix.dist }}
14+
runs-on: ubuntu-latest
15+
container: debian:${{ matrix.dist }}-slim
16+
strategy:
17+
fail-fast: false
18+
matrix:
19+
dist: [bullseye, bookworm]
20+
steps:
21+
- uses: actions/checkout@v2
22+
23+
- name: Install build-dependencies
24+
# TODO(dato): find out why setup.py links to compression libraries
25+
# by hand (and hence their development packages needed here).
26+
run: |
27+
apt-get update
28+
apt-get install --no-install-recommends -y \
29+
build-essential librocksdb-dev cython3 python3-pkgconfig \
30+
python3-dev python3-pip python3-pytest \
31+
libsnappy-dev libbz2-dev liblz4-dev libz-dev
32+
33+
- name: Build pyrocksdb
34+
# TODO(dato): consider using pypa/build --no-isolaiton, to
35+
# build the package using a tool specifically designed for
36+
# that, rather than trusting it to a tool that does a lot
37+
# more (pip).
38+
run: |
39+
python3 -m pip install --no-build-isolation -v '.[test]'
40+
41+
- name: Run tests
42+
run: |
43+
pytest-3 --pyargs rocksdb

‎.github/workflows/dist.yml‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# vim:ts=2:sw=2:et:ai:sts=2
2+
name: 'Build distribution'
3+
4+
on:
5+
# Only run when pushing to main/merging PRs.
6+
push:
7+
branches:
8+
- main
9+
- 'force_ci/all/**' # For development, forcing all workflows to run.
10+
- 'force_ci/dist/**' # For debugging and/or only forcing this workflow.
11+
12+
jobs:
13+
build_wheels:
14+
name: 'Build wheels'
15+
runs-on: ${{ matrix.os }}
16+
env:
17+
LIBROCKSDB_PATH: /opt/rocksdb-${{ matrix.rocksdb_ver }}
18+
strategy:
19+
matrix:
20+
os: [ubuntu-latest]
21+
rocksdb_ver: ['v6.14.6']
22+
23+
steps:
24+
- uses: actions/checkout@v2
25+
name: 'Checkout source repository'
26+
27+
- uses: actions/setup-python@v2
28+
name: 'Set up Python 3.9'
29+
with:
30+
python-version: '3.9'
31+
32+
- name: 'Install cibuildwheel'
33+
run: |
34+
python3 -m pip install cibuildwheel==1.7.1
35+
36+
- name: 'Build wheels'
37+
run: |
38+
python3 -m cibuildwheel --output-dir dist
39+
env:
40+
CIBW_MANYLINUX_X86_64_IMAGE: 'manylinux2014'
41+
CIBW_BUILD: 'cp3*'
42+
CIBW_SKIP: '*-manylinux_i686'
43+
# Install python package and test-deps.
44+
CIBW_TEST_REQUIRES: '.[test] pytest'
45+
# Use `--pyargs` to interpret parameter as module to import, not as a
46+
# path, and do not use `python3 -m pytest`. This way we prevent
47+
# importing the module from the current directory instead of the
48+
# installed package, and failing when it cannot find the shared
49+
# library.
50+
CIBW_TEST_COMMAND: 'pytest --pyargs rocksdb'
51+
# Avoid re-building the C library in every iteration by testing for
52+
# the build directory.
53+
CIBW_BEFORE_BUILD: >
54+
yum install -y bzip2-devel lz4-devel snappy-devel zlib-devel
55+
python3-Cython && (
56+
test -d ${{ env.LIBROCKSDB_PATH }} || (
57+
git clone https://github.com/facebook/rocksdb --depth 1
58+
--branch ${{ matrix.rocksdb_ver }} ${{ env.LIBROCKSDB_PATH }} &&
59+
cd ${{ env.LIBROCKSDB_PATH }} &&
60+
CXXFLAGS='-flto -Os -s' PORTABLE=1 make shared_lib -j 4
61+
)) &&
62+
pushd ${{ env.LIBROCKSDB_PATH }} &&
63+
make install-shared &&
64+
ldconfig &&
65+
popd
66+
67+
- uses: actions/upload-artifact@v2
68+
name: 'Upload build artifacts'
69+
with:
70+
path: 'dist/*.whl'
71+
72+
build_sdist:
73+
name: 'Build source distribution'
74+
runs-on: 'ubuntu-latest'
75+
steps:
76+
- uses: actions/checkout@v2
77+
name: 'Checkout source repository'
78+
79+
- uses: actions/setup-python@v2
80+
name: 'Set up Python 3.9'
81+
with:
82+
python-version: '3.9'
83+
84+
- name: 'Build sdist'
85+
run: |
86+
python3 setup.py sdist
87+
88+
- uses: actions/upload-artifact@v2
89+
name: 'Upload build artifacts'
90+
with:
91+
path: 'dist/*.tar.gz'
92+
93+
94+
# upload_pypi:
95+
# name: 'Upload packages'
96+
# needs: ['build_wheels', 'build_sdist']
97+
# runs-on: 'ubuntu-latest'
98+
# if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/v')
99+
# steps:
100+
# - uses: actions/download-artifact@v2
101+
# name: 'Download artifacts'
102+
# with:
103+
# name: 'artifact'
104+
# path: 'dist'
105+
#
106+
# - uses: pypa/gh-action-pypi-publish@master
107+
# name: 'Publish built packages'
108+
# with:
109+
# user: '__token__'
110+
# password: '${{ secrets.PYPI_API_TOKEN }}'

‎Dockerfile‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM ubuntu:18.04
1+
FROM ubuntu:20.04
22
ENV SRC /home/tester/src
33
ENV DEBIAN_FRONTEND noninteractive
44

@@ -10,7 +10,7 @@ RUN apt-get update -y && apt-get install -qy \
1010
python3 \
1111
python-dev \
1212
python3-dev \
13-
python-pip \
13+
python3-pip \
1414
librocksdb-dev \
1515
libsnappy-dev \
1616
zlib1g-dev \
@@ -24,10 +24,19 @@ RUN update-locale
2424
RUN locale-gen $LANG
2525

2626
#NOTE(sileht): Upgrade python dev tools
27-
RUN pip install -U pip tox virtualenv
27+
RUN pip3 install -U pip tox virtualenv setuptools pytest Cython
2828

29-
RUN groupadd --gid 2000 tester
30-
RUN useradd --uid 2000 --gid 2000 --create-home --shell /bin/bash tester
31-
USER tester
29+
# Set username same as generic default username. Allows output build to be available to same user
30+
ENV USER_NAME ubuntu
3231

33-
WORKDIR $SRC
32+
ARG host_uid=1001
33+
ARG host_gid=1001
34+
RUN groupadd -g $host_gid $USER_NAME && \
35+
useradd -g $host_gid -m -s /bin/bash -u $host_uid $USER_NAME
36+
37+
USER $USER_NAME
38+
39+
ENV BUILD_INPUT_DIR /home/$USER_NAME/workspace
40+
RUN mkdir -p $BUILD_INPUT_DIR
41+
42+
WORKDIR $BUILD_INPUT_DIR

LICENSE.md renamed to LICENSE

File renamed without changes.

‎README.rst‎

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ The original pyrocksdb (https://pypi.python.org/pypi/pyrocksdb/0.4) has not been
44

55
News (2019/04/18)
66
=========
7-
Currently I am refactoring the code, and more features like TTL are coming soon. And the installation with cmake will be much more easily.
7+
Currently I am refactoring the code, and more features like TTL are coming soon. And the installation with cmake will be much more easily.
88

99
News (2019/04/19)
1010
=========
@@ -16,8 +16,9 @@ pyrocksdb
1616
=========
1717

1818
Python bindings for RocksDB.
19-
See http://python-rocksdb.readthedocs.io/en/latest/ for a more comprehensive install and usage description.
2019

20+
See https://rocksdb-tina.readthedocs.io/ for a more comprehensive install and
21+
usage description.
2122

2223
Quick Install
2324
-------------
@@ -47,10 +48,28 @@ Quick install for debian/ubuntu like linux distributions.
4748
Quick Usage Guide
4849
-----------------
4950

50-
.. code-block:: pycon
51+
.. code-block:: python
5152
5253
>>> import rocksdb
53-
>>> db = rocksdb.DB("test.db", rocksdb.Options(create_if_missing=True))
54+
>>> db = rocksdb.DB('test.db', rocksdb.Options(create_if_missing=True))
5455
>>> db.put(b'a', b'data')
55-
>>> print db.get(b'a')
56+
>>> print(db.get(b'a'))
5657
b'data'
58+
59+
60+
Acknowledgements
61+
----------------
62+
63+
This project attempts to collect the efforts put into different forks of the
64+
`pyrocksdb`_ project that was originally written by `stephan-hof`_, as sadly
65+
none seems to be actively maintained. In particular, the `python-rocksdb`_ fork
66+
created by `twmht`_, but it also incorporates changes from other forks and
67+
unfinished pull requests. The current state of this project has pulled all of
68+
these commits from `NightTsarina`_'s `fork`_.
69+
70+
.. _python-rocksdb: https://github.com/twmht/python-rocksdb
71+
.. _twmht: https://github.com/twmht
72+
.. _pyrocksdb: https://github.com/stephan-hof/pyrocksdb
73+
.. _stephan-hof: https://github.com/stephan-hof
74+
.. _NightTsarina: https://github.com/NightTsarina
75+
.. _fork: https://github.com/NightTsarina/python-rocksdb

0 commit comments

Comments
 (0)