Exit code 0 even though tests fail #1112
Description
We're seeing Ceedling return exit code 0 even though there are failing tests with ceedling test:all:
-----------------------
14 Tests 4 Failures 0 Ignored
FAIL
Ceedling operations completed in 2.56 seconds
root@docker-desktop:/workspaces/myproject# echo $?
0
Ceedling version:
# ceedling version
🌱 Welcome to Ceedling!
Ceedling => 1.0.1-fb1ce6c
----------------------
/var/lib/gems/3.2.0/gems/ceedling-1.0.1/
Build Frameworks
----------------------
CMock => 2.6.0
Unity => 2.6.1
CException => 1.3.4
Our project.yml:
---
---
# Notes:
# Sample project C code is not presently written to produce a release artifact.
# As such, release build options are disabled.
# This sample, therefore, only demonstrates running a collection of unit tests.
:project:
:use_exceptions: FALSE
:use_test_preprocessor: :all
:use_auxiliary_dependencies: TRUE
:build_root: build
# :release_build: TRUE
:test_file_prefix: test_
:which_ceedling: gem
:ceedling_version: 1.0.1
:default_tasks:
- test:all
- gcov:all
#:test_build:
# :use_assembly: TRUE
#:release_build:
# :output: MyApp.out
# :use_assembly: FALSE
:environment:
:extension:
:executable: .out
# Add required files to source and include paths.
:paths:
:test:
- +:test/**
:source:
- +:src/**
# - +:/opt/esp/idf/components/**
- +:/opt/esp/idf/components/esp_system/**
- +:/opt/esp/idf/components/esp_driver_gpio/**
- +:/opt/esp/idf/components/esp_hw_support/**
- +:/opt/esp/idf/components/soc/esp32c6/**
- +:/opt/esp/idf/components/hal/**
- +:/opt/esp/idf/components/esp_rom/**
:include:
- +:include/**
# - +:/opt/esp/idf/components/**
- +:/opt/esp/idf/components/esp_common/**
- +:/opt/esp/idf/components/esp_driver_gpio/**
- +:/opt/esp/idf/components/esp_hw_support/**
- +:/opt/esp/idf/components/soc/esp32c6/**
- +:/opt/esp/idf/components/hal/**
- +:/opt/esp/idf/components/esp_rom/**
- +:/opt/esp/idf/components/log/**
- +:/opt/esp/idf/components/efuse/**
- +:/opt/esp/idf/components/ieee802154/**
:support:
- +:test/custom_mocks/**
:libraries: []
# ADD all project definitions to the common_defines array for unit testing
:defines:
:common:
&common_defines
[
CONFIG_HARDWARE_VERSION=1
]
:test:
- *common_defines
- CONFIG_MOCK_BOARD_ID=0
:test_preprocess:
- *common_defines
- CONFIG_MOCK_BOARD_ID=0
:cmock:
:mock_prefix: mock_
:when_no_prototypes: :warn
:enforce_strict_ordering: TRUE
:plugins:
- :ignore
- :ignore_arg
- :callback
- :expect_any_args
:treat_as:
uint8: HEX8
uint16: HEX16
uint32: UINT32
int8: INT8
bool: UINT8
:ignore:
- esp_log.h
- esp_efuse.h
# Add -gcov to the plugins list to make sure of the gcov plugin
# You will need to have gcov and gcovr both installed to make it work.
# For more information on these options, see docs in plugins/gcov
:gcov:
# :html_report_type: detailed
:reports:
- HtmlDetailed
:gcovr:
:html_medium_threshold: 75
:html_high_threshold: 90
#:tools:
# Ceedling defaults to using gcc for compiling, linking, etc.
# As [:tools] is blank, gcc will be used (so long as it's in your system path)
# See documentation to configure a given toolchain for use
:flags:
:test:
:compile:
- -std=c2x
:gcov:
:compile:
- -std=c2x
# LIBRARIES
# These libraries are automatically injected into the build process. Those specified as
# common will be used in all types of builds. Otherwise, libraries can be injected in just
# tests or releases. These options are MERGED with the options in supplemental yaml files.
:libraries:
:placement: :end
:flag: "-l${1}"
:path_flag: "-L ${1}"
:system: [] # for example, you might list 'm' to grab the math library
:test: []
:release: []
:plugins:
:enabled:
- gcov
# - xml_tests_report
# - stdout_pretty_tests_report
- module_generator
What's going on here?