Skip to content

Added CI to run ceph tests on ns aws using minio #9186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
+746 −8
Open

Added CI to run ceph tests on ns aws using minio #9186

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
278b226
Added CI to run ceph tests on ns aws using minio
achouhan09 Aug 7, 2025
95a5292
Moved bucket creation to config, proper casing and some efficient cha…
achouhan09 Aug 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/workflows/ceph-ns-aws-s3-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Ceph NS AWS S3 Tests
on: [workflow_call]

jobs:
ceph-ns-aws-s3-tests:
name: Ceph NS AWS S3 Tests
runs-on: ubuntu-latest
timeout-minutes: 90
permissions:
actions: read # download-artifact
contents: read # required for actions/checkout
steps:
- name: Checkout noobaa-core
uses: actions/checkout@v4
with:
repository: 'noobaa/noobaa-core'
path: 'noobaa-core'

- name: Download artifact
uses: actions/download-artifact@v4
with:
name: noobaa-tester
path: /tmp

- name: Load image
run: docker load --input /tmp/noobaa-tester.tar

- name: Run Ceph NS AWS s3-tests
run: |
set -x
cd ./noobaa-core
mkdir -p logs/ceph-ns-aws-test-logs
chmod 777 logs/ceph-ns-aws-test-logs
make test-cephs3-ns-aws -o tester
4 changes: 4 additions & 0 deletions .github/workflows/run-pr-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ jobs:
needs: build-noobaa-image
uses: ./.github/workflows/ceph-s3-tests.yaml

ceph-ns-aws-s3-tests:
needs: build-noobaa-image
uses: ./.github/workflows/ceph-ns-aws-s3-tests.yaml

ceph-nsfs-s3-tests:
needs: build-noobaa-image
uses: ./.github/workflows/ceph-nsfs-s3-tests.yaml
Expand Down
44 changes: 44 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,17 @@ MINT_MOCK_ACCESS_KEY="aaaaaaaaaaaaaEXAMPLE"
MINT_MOCK_SECRET_KEY="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaEXAMPLE"
MINT_NOOBAA_HTTP_ENDPOINT_PORT=6001

####################
# MINIO VARIABLES #
###################

MINIO_IMAGE?="quay.io/minio/minio"

MINIO_PORT?=9000
MINIO_USER?="miniotest"
MINIO_PASSWORD?="miniotest123"
MINIO_TEST_BUCKET?="minio-test-bucket"

###############
# BUILD LOCAL #
###############
Expand Down Expand Up @@ -352,6 +363,19 @@ test-cephs3: tester
@$(call remove_docker_network)
.PHONY: test-cephs3

test-cephs3-ns-aws: tester
@echo "\033[1;34mRunning ceph-s3 tests with namespace store pointing to s3 bucket in Minio\033[0m"
@$(call create_docker_network)
@$(call run_minio)
@$(call run_postgres)
@echo "\033[1;34mRunning tests\033[0m"
$(CONTAINER_ENGINE) run $(CPUSET) --privileged --user root --network noobaa-net --name noobaa_$(GIT_COMMIT)_$(NAME_POSTFIX) --env "SUPPRESS_LOGS=$(SUPPRESS_LOGS)" --env "POSTGRES_HOST=coretest-postgres-$(GIT_COMMIT)-$(NAME_POSTFIX)" --env "POSTGRES_USER=noobaa" --env "DB_TYPE=postgres" --env "POSTGRES_DBNAME=coretest" --env "USE_S3_NAMESPACE_RESOURCE=true" --env "MINIO_ENDPOINT=http://coretest-minio-$(GIT_COMMIT)-$(NAME_POSTFIX):$(MINIO_PORT)" --env "MINIO_USER=$(MINIO_USER)" --env "MINIO_PASSWORD=$(MINIO_PASSWORD)" --env "MINIO_TEST_BUCKET=$(MINIO_TEST_BUCKET)" -v $(PWD)/logs:/logs $(TESTER_TAG) "./src/test/external_tests/ceph_s3_tests/run_ceph_test_on_test_container.sh"
@$(call stop_noobaa)
@$(call stop_postgres)
@$(call stop_minio)
@$(call remove_docker_network)
Comment on lines +373 to +376
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a previous step fails, I believe it causes the entire make to fail, which means those cleanup steps will not run.
We need to make sure that the cleanup runs even if we encounter unexpected, fatal errors

.PHONY: test-cephs3-ns-aws

test-warp: tester
@echo "\033[1;34mRunning warp tests with Postgres.\033[0m"
@$(call create_docker_network)
Expand Down Expand Up @@ -581,3 +605,23 @@ define stop_blob_mock
fi
@echo "\033[1;32mBlob mock server stop done.\033[0m"
endef

#########
# MINIO #
#########

define run_minio
@echo "\033[1;34mRunning Minio container\033[0m"
$(CONTAINER_ENGINE) run -d $(CPUSET) --network noobaa-net --name coretest-minio-$(GIT_COMMIT)-$(NAME_POSTFIX) -p $(MINIO_PORT):$(MINIO_PORT) --env "MINIO_ROOT_USER=$(MINIO_USER)" --env "MINIO_ROOT_PASSWORD=$(MINIO_PASSWORD)" $(MINIO_IMAGE) server /data --address ":$(MINIO_PORT)"
@echo "\033[1;34mWaiting for Minio to start..\033[0m"
sleep 20
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The time period (20 seconds) might be both too short as well as too long. I'd recommend to implement a proper health check with periodic (1 second~) checks instead

@echo "\033[1;32mRun Minio done.\033[0m"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the startup fails? It'd be good to add health checks and proper error handling (possibly with retry logic)

endef

define stop_minio
@echo "\033[1;34mStopping/removing Minio container\033[0m"
$(call disconnect_container_from_noobaa_network, coretest-minio-$(GIT_COMMIT)-$(NAME_POSTFIX))
$(CONTAINER_ENGINE) stop coretest-minio-$(GIT_COMMIT)-$(NAME_POSTFIX)
$(CONTAINER_ENGINE) rm coretest-minio-$(GIT_COMMIT)-$(NAME_POSTFIX)
@echo "\033[1;32mStop Minio done.\033[0m"
endef
8 changes: 7 additions & 1 deletion src/test/external_tests/ceph_s3_tests/run_ceph_test_on_test_container.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,13 @@ export MGMT_ADDR=wss://${NOOBAA_MGMT_SERVICE_HOST:-localhost}:${NOOBAA_MGMT_SERV
export BG_ADDR=wss://localhost:5445
export HOSTED_AGENTS_ADDR=wss://localhost:5446

export CEPH_TEST_LOGS_DIR=/logs/ceph-test-logs
# set logs directory based on test type
# later we will add more test types and logs directories on the basis of s3 namespace resource type
if [ "${USE_S3_NAMESPACE_RESOURCE}" = "true" ]; then
export CEPH_TEST_LOGS_DIR=/logs/ceph-ns-aws-test-logs
else
export CEPH_TEST_LOGS_DIR=/logs/ceph-test-logs
fi

export CONFIG_JS_OBJECT_SDK_BUCKET_CACHE_EXPIRY_MS=0 # Needed for disabling cache for ceph cors test and maybe some more
export CONFIG_JS_allow_anonymous_access_in_test=true # Needed for allowing anon access for tests using ACL='public-read-write'
Expand Down
Loading
Loading