Skip to content

parallelize tests #509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PastelStorm merged 15 commits into from
Jun 14, 2025
Merged

parallelize tests #509

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
654ec65
parallelize tests
jiajun-unstructured Jun 5, 2025
deafbe3
try out ci change
jiajun-unstructured Jun 10, 2025
e3006b0
add make install test
jiajun-unstructured Jun 11, 2025
fd4e9e0
update test.txt
jiajun-unstructured Jun 11, 2025
d7212f0
change test runner to xlarge
jiajun-unstructured Jun 11, 2025
a711e0f
switch back to ubuntu-latest-m
jiajun-unstructured Jun 11, 2025
42b7cdd
remove absolute path
jiajun-unstructured Jun 12, 2025
e66cbb3
run make install in gha
jiajun-unstructured Jun 12, 2025
2abf77e
add test.txt in the cache key for ci.yml
jiajun-unstructured Jun 12, 2025
c6c2491
remove make install-nltk-models
jiajun-unstructured Jun 12, 2025
996a0ad
add cache hit miss to ci
jiajun-unstructured Jun 12, 2025
d7f339f
remove cache hit miss backup
jiajun-unstructured Jun 12, 2025
a4c87c1
Bump Python to 3.11, rebuild deps, revert CI changes
PastelStorm Jun 12, 2025
7c81aa8
downgrade unstructured to 0.16.17; fix test
jiajun-unstructured Jun 13, 2025
591c24c
skip the test where json became processable given the unstructured ve…
jiajun-unstructured Jun 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ on:
branches: [ main ]

env:
PYTHON_VERSION: "3.10"
PYTHON_VERSION: "3.12"
PIPELINE_FAMILY: "general"

jobs:
Expand All @@ -20,7 +20,7 @@ jobs:
with:
path: |
.venv
key: ci-venv-${{ env.PIPELINE_FAMILY }}-${{ hashFiles('requirements/base.txt') }}
key: ci-venv-${{ env.PIPELINE_FAMILY }}-${{ hashFiles('requirements/*.txt') }}
- name: Set up Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@v5
with:
Expand All @@ -42,7 +42,7 @@ jobs:
with:
path: |
.venv
key: ci-venv-${{ env.PIPELINE_FAMILY }}-${{ hashFiles('requirements/base.txt') }}
key: ci-venv-${{ env.PIPELINE_FAMILY }}-${{ hashFiles('requirements/*.txt') }}
- name: Lint
run: |
source .venv/bin/activate
Expand All @@ -65,16 +65,17 @@ jobs:
with:
path: |
.venv
key: ci-venv-${{ env.PIPELINE_FAMILY }}-${{ hashFiles('requirements/base.txt') }}
key: ci-venv-${{ env.PIPELINE_FAMILY }}-${{ hashFiles('requirements/test.txt') }}
- name: Run core tests
run: |
python${{ env.PYTHON_VERSION }} -m venv .venv
source .venv/bin/activate
sudo apt-get update && sudo apt-get install --yes poppler-utils libreoffice
make install-test
make install-pandoc
sudo add-apt-repository -y ppa:alex-p/tesseract-ocr5
sudo apt-get install -y tesseract-ocr tesseract-ocr-kor
tesseract --version
make install-nltk-models
make test
make check-coverage

Expand Down Expand Up @@ -106,10 +107,12 @@ jobs:
with:
path: |
.venv
key: ci-venv-${{ env.PIPELINE_FAMILY }}-${{ hashFiles('requirements/base.txt') }}
key: ci-venv-${{ env.PIPELINE_FAMILY }}-${{ hashFiles('requirements/test.txt') }}
- name: Test Dockerfile
run: |
python${{ env.PYTHON_VERSION }} -m venv .venv
source .venv/bin/activate
make install-test
make docker-build
make docker-test
# - name: Scan image
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docker-publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ env:
PACKAGE: "unstructured-api"
PIPELINE_FAMILY: "general"
PIP_VERSION: "25.1.1"
PYTHON_VERSION: "3.10"
PYTHON_VERSION: "3.12"

jobs:
setup:
Expand Down
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ run-web-app:
## test: runs core tests
.PHONY: test
test:
PYTHONPATH=. pytest -v test_${PIPELINE_PACKAGE} --cov=${PACKAGE_NAME} --cov-report term-missing
PYTHONPATH=. pytest -n auto -v test_${PIPELINE_PACKAGE} --cov=${PACKAGE_NAME} --cov-report term-missing

# Setting a low bar here - need more tests!
.PHONY: check-coverage
Expand Down
5 changes: 3 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,12 +289,13 @@ curl -X 'POST'
* Using `pyenv` to manage virtualenv's is recommended
* Mac install instructions. See [here](https://github.com/Unstructured-IO/community#mac--homebrew) for more detailed instructions.
* `brew install pyenv-virtualenv`
* `pyenv install 3.10.12`
* `pyenv install 3.12`
* Linux instructions are available [here](https://github.com/Unstructured-IO/community#linux).

* Create a virtualenv to work in and activate it, e.g. for one named `document-processing`:

`pyenv virtualenv 3.10.12 unstructured-api` <br />
`pyenv virtualenv 3.12
unstructured-api` <br />
`pyenv activate unstructured-api`

See the [Unstructured Quick Start](https://github.com/Unstructured-IO/unstructured#eight_pointed_black_star-quick-start) for the many OS dependencies that are required, if the ability to process all file types is desired.
Expand Down
5 changes: 1 addition & 4 deletions requirements/base.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
-c constraints.in
unstructured[all-docs]
# Pinning click due to a unicode issue in black
# can remove after black drops support for Python 3.6
# ref: https://github.com/psf/black/issues/2964
click==8.2.1
click
fastapi
uvicorn
ratelimit
Expand Down
Loading