[Bug]: Equivalence Ind. Samples T-Test upper bound for Cohen's d #3457
Description
JASP Version
0.19.3
Commit ID
No response
JASP Module
Equivalence T-Tests
What analysis are you seeing the problem on?
Equivalence independent samples t-test (the frequentist one)
What OS are you seeing the problem on?
Windows 11
Bug Description
EDIT: I already found the error in the source code, so for a tl;dr, skip to my comment below
I noticed that the upper bounds for "Raw" and for "Cohen's d" where the same in the Equivalence Bounds table. First, I thought "well that is a funny coincidence". But, next analysis: it happened again! Suspecting it could be a bug in the older version I'm using on a company environment (0.18.3 on Windows 10), I tried it again on 0.19.3 on Windows 11, the problem sustained. Same with 0.19.3 on macOS (Apple Silicon). It seems that the upper bound for "Raw" is used for the upper bound of "Cohen's d" as well.
I decided to make a replicable example which shows the problem as well. Simulated some data and performing a TOST using the following R code:
set.seed(42)
x <- rnorm(400, 10, 2)
g <- sample(rep(c(1,2), each=200)) # Don't know why I felt the need to use "sample", but whatever :)
library(TOSTER)
tsum_TOST(m1 = mean(x[g==1]),
sd1 = sd(x[g==1]),
m2 = mean(x[g==2]),
sd2 = sd(x[g==2]),
n1 = sum(g==1),
n2 = sum(g==2),
eqbound_type = "SMD",
eqb = c(-0.3, 0.3),
bias_correction = FALSE,
var.equal = TRUE)
write.csv(data.frame(x,g), "equivTest.csv", row.names = FALSE)
The function tsum_TOST gives us:
Two Sample t-test
The equivalence test was significant, t(398) = 2.767, p = 2.96e-03
The null hypothesis test was non-significant, t(398) = -0.233, p = 8.16e-01
NHST: don't reject null significance hypothesis that the effect is equal to zero
TOST: reject null equivalence hypothesis
TOST Results
t df p.value
t-test -0.2331 398 0.816
TOST Lower 2.7669 398 0.003
TOST Upper -3.2331 398 < 0.001
Effect Sizes
Estimate SE C.I. Conf. Level
Raw -0.04480 0.1922 [-0.3617, 0.2721] 0.9
Cohen's d -0.02331 0.1000 [-0.1878, 0.1412] 0.9
Note: SMD confidence intervals are an approximation. See vignette("SMD_calcs").
I attached the data file for your convenience, as well as a print screen of the output using JASP. Here, you can see that the upper bound value for "Cohen's d" is 0.272 (same as for "Raw"), where it should be 0.141. I also performed a "regular" independent samples t-test in JASP and checked the options for the location parameter (and 90% CI) as well as Cohen's d (and 90% CI), because I would expect to see these values in the Equivalence Bounds table. This gave me the right boundaries there.
I suspect the values of the Equivalence Bounds table are used for the Equivalence bounds plot, because this plot reflects the faulty behaviour (not shown in the print screen).
Expected Behaviour
The upper bound of the 90% CI for Cohen's d should have the value that is calculated by TOSTER. Furthermore, the Equivalence Bounds plot should depict this value as the upper bound as well.
Steps to Reproduce
- Load the attached file or simulate it yourself with the R code above
- Perform an Equivalence Independent Samples T-Test
- Look at the Equivalence Bounds table, row "Cohen's d", under "Upper", and compare to the same value for "Raw"
Log (if any)
No response
More Debug Information
-------- Application Info --------
JASP Version: JASP 0.19.3 (Apple Silicon)
Build Branch: 0.19.3
Build Date: Jan 3 2025 12:51:39 (Netherlands)
Last Commit: c8710df5442149527b0692cc1367ea5b3c982d04
-------- Basic Info --------
Operating System: macOS Sonoma (14.5)
Product Version: 14.5
Kernel Type: darwin
Kernel Version: 23.5.0
Architecture: arm64
Install Path: /Applications/JASP.app/Contents/MacOS
Platfotm Name: cocoa
System Local: en_NL
Final Checklist
- I have included a screenshot showcasing the issue, if possible.
- I have included a JASP file (zipped) or data file that causes the crash/bug, if applicable.
- I have accurately described the bug, and steps to reproduce it.