Skip to content

Kevinjqliu/run ci windows #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Kevinjqliu/run ci windows #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8ea69cc
use command from python
kevinjqliu Sep 16, 2025
92007bc
install python before poetry
kevinjqliu Sep 16, 2025
f6e09fa
run ci with windows
kevinjqliu Sep 17, 2025
522771c
dont install kerberos for windows
kevinjqliu Sep 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/python-ci-docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,11 @@ jobs:

steps:
- uses: actions/checkout@v5
- name: Install poetry
run: make install-poetry
- uses: actions/setup-python@v6
with:
python-version: 3.12
- name: Install poetry
run: make install-poetry
- name: Install
run: make docs-install
- name: Build docs
Expand Down
129 changes: 78 additions & 51 deletions .github/workflows/python-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,66 +43,93 @@ concurrency:
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
lint-and-test:
runs-on: ubuntu-22.04
# lint-and-test:
# runs-on: ubuntu-22.04
# strategy:
# matrix:
# python: ['3.9', '3.10', '3.11', '3.12']

# steps:
# - uses: actions/checkout@v5
# - uses: actions/setup-python@v6
# with:
# python-version: ${{ matrix.python }}
# - name: Install poetry
# run: make install-poetry
# - name: Install system dependencies
# run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
# - name: Install
# run: make install-dependencies
# - name: Run linters
# run: make lint
# - name: Run unit tests with coverage
# run: COVERAGE=1 make test
# - name: Generate coverage report (85%) # Coverage threshold should only increase over time — never decrease it!
# run: COVERAGE_FAIL_UNDER=85 make coverage-report

# integration-test:
# runs-on: ubuntu-22.04
# strategy:
# matrix:
# python: ['3.9', '3.10', '3.11', '3.12']

# steps:
# - uses: actions/checkout@v5
# - name: Install system dependencies
# run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
# - name: Install
# run: make install

# - name: Run integration tests with coverage
# run: COVERAGE=1 make test-integration
# - name: Show debug logs
# if: ${{ failure() }}
# run: docker compose -f dev/docker-compose.yml logs

# - name: Run s3 integration tests with coverage
# run: COVERAGE=1 make test-s3
# - name: Show debug logs
# if: ${{ failure() }}
# run: docker compose -f dev/docker-compose.yml logs

# - name: Run adls integration tests with coverage
# run: COVERAGE=1 make test-adls
# - name: Show debug logs
# if: ${{ failure() }}
# run: docker compose -f dev/docker-compose-azurite.yml logs

# - name: Run gcs integration tests with coverage
# run: COVERAGE=1 make test-gcs
# - name: Show debug logs
# if: ${{ failure() }}
# run: docker compose -f dev/docker-compose-gcs-server.yml logs

# - name: Generate coverage report (75%) # Coverage threshold should only increase over time — never decrease it!
# run: COVERAGE_FAIL_UNDER=75 make coverage-report

make-test-windows:
runs-on: windows-2022
strategy:
matrix:
python: ['3.9', '3.10', '3.11', '3.12']
python: ['3.12']

steps:
- uses: actions/checkout@v5
- name: Install poetry
run: make install-poetry
- name: Install Make
run: choco install make
shell: powershell
- uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python }}
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
- name: Install
- name: Install poetry
run: make install-poetry
shell: bash
- name: Install dependencies
run: make install-dependencies
- name: Run linters
run: make lint
shell: bash
- name: Run unit tests with coverage
run: COVERAGE=1 make test
- name: Generate coverage report (85%) # Coverage threshold should only increase over time — never decrease it!
shell: bash
- name: Generate coverage report (85%)
run: COVERAGE_FAIL_UNDER=85 make coverage-report

integration-test:
runs-on: ubuntu-22.04
strategy:
matrix:
python: ['3.9', '3.10', '3.11', '3.12']

steps:
- uses: actions/checkout@v5
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
- name: Install
run: make install

- name: Run integration tests with coverage
run: COVERAGE=1 make test-integration
- name: Show debug logs
if: ${{ failure() }}
run: docker compose -f dev/docker-compose.yml logs

- name: Run s3 integration tests with coverage
run: COVERAGE=1 make test-s3
- name: Show debug logs
if: ${{ failure() }}
run: docker compose -f dev/docker-compose.yml logs

- name: Run adls integration tests with coverage
run: COVERAGE=1 make test-adls
- name: Show debug logs
if: ${{ failure() }}
run: docker compose -f dev/docker-compose-azurite.yml logs

- name: Run gcs integration tests with coverage
run: COVERAGE=1 make test-gcs
- name: Show debug logs
if: ${{ failure() }}
run: docker compose -f dev/docker-compose-gcs-server.yml logs

- name: Generate coverage report (75%) # Coverage threshold should only increase over time — never decrease it!
run: COVERAGE_FAIL_UNDER=75 make coverage-report
shell: bash
4 changes: 2 additions & 2 deletions .github/workflows/python-release-docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,11 @@ jobs:

steps:
- uses: actions/checkout@v5
- name: Install poetry
run: make install-poetry
- uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python }}
- name: Install poetry
run: make install-poetry
- name: Install docs
run: make docs-install
- name: Build docs
Expand Down
37 changes: 20 additions & 17 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,15 @@ COVERAGE ?= 0 # Set COVERAGE=1 to enable coverage: make test COVERAGE=1
COVERAGE_FAIL_UNDER ?= 85 # Minimum coverage % to pass: make coverage-report COVERAGE_FAIL_UNDER=70
KEEP_COMPOSE ?= 0 # Set KEEP_COMPOSE=1 to keep containers after integration tests

PIP = python -m pip

POETRY_VERSION = 2.1.4
POETRY = python -m poetry

ifeq ($(COVERAGE),1)
TEST_RUNNER = poetry run coverage run --parallel-mode --source=pyiceberg -m
TEST_RUNNER = $(POETRY) run coverage run --parallel-mode --source=pyiceberg -m
else
TEST_RUNNER = poetry run
TEST_RUNNER = $(POETRY) run
endif

ifeq ($(KEEP_COMPOSE),1)
Expand All @@ -35,8 +40,6 @@ else
CLEANUP_COMMAND = docker compose -f dev/docker-compose-integration.yml down -v --remove-orphans 2>/dev/null || true
endif

POETRY_VERSION = 2.1.4

# ============
# Help Section
# ============
Expand All @@ -53,21 +56,21 @@ help: ## Display this help message
##@ Setup

install-poetry: ## Ensure Poetry is installed at the specified version
@if ! command -v poetry &> /dev/null; then \
@if ! command -v ${POETRY} &> /dev/null; then \
echo "Poetry not found. Installing..."; \
pip install --user poetry==$(POETRY_VERSION); \
${PIP} install --user poetry==$(POETRY_VERSION); \
else \
INSTALLED_VERSION=$$(pip show poetry | grep Version | awk '{print $$2}'); \
INSTALLED_VERSION=$$(${PIP} show poetry | grep Version | awk '{print $$2}'); \
if [ "$$INSTALLED_VERSION" != "$(POETRY_VERSION)" ]; then \
echo "Updating Poetry to version $(POETRY_VERSION)..."; \
pip install --user --upgrade poetry==$(POETRY_VERSION); \
${PIP} install --user --upgrade poetry==$(POETRY_VERSION); \
else \
echo "Poetry version $(POETRY_VERSION) already installed."; \
fi; \
fi

install-dependencies: ## Install all dependencies including extras
poetry install --all-extras
$(POETRY) install --all-extras

install: install-poetry install-dependencies ## Install Poetry and dependencies

Expand All @@ -81,7 +84,7 @@ check-license: ## Check license headers
./dev/check-license

lint: ## Run code linters via pre-commit
poetry run pre-commit run --all-files
$(POETRY) run pre-commit run --all-files

# ===============
# Testing Section
Expand Down Expand Up @@ -132,10 +135,10 @@ test-coverage: COVERAGE=1
test-coverage: test test-integration test-s3 test-adls test-gcs coverage-report ## Run all tests with coverage and report

coverage-report: ## Combine and report coverage
poetry run coverage combine
poetry run coverage report -m --fail-under=$(COVERAGE_FAIL_UNDER)
poetry run coverage html
poetry run coverage xml
${POETRY} run coverage combine
${POETRY} run coverage report -m --fail-under=$(COVERAGE_FAIL_UNDER)
${POETRY} run coverage html
${POETRY} run coverage xml

# ================
# Documentation
Expand All @@ -144,13 +147,13 @@ coverage-report: ## Combine and report coverage
##@ Documentation

docs-install: ## Install docs dependencies
poetry install --with docs
${POETRY} install --with docs

docs-serve: ## Serve local docs preview (hot reload)
poetry run mkdocs serve -f mkdocs/mkdocs.yml
${POETRY} run mkdocs serve -f mkdocs/mkdocs.yml

docs-build: ## Build the static documentation site
poetry run mkdocs build -f mkdocs/mkdocs.yml --strict
${POETRY} run mkdocs build -f mkdocs/mkdocs.yml --strict

# ===================
# Project Maintenance
Expand Down
Loading
Loading