blog/one-proportion-and-goodness-of-fit-test-in-r-and-by-hand/

[giscus[bot]]

blog/one-proportion-and-goodness-of-fit-test-in-r-and-by-hand/

Learn how to perform the one proportion and goodness of fit test (useful to check if a distribution follows a specific known distribution) in R and by hand

https://statsandr.com/blog/one-proportion-and-goodness-of-fit-test-in-r-and-by-hand/

[vtfresh]

The Chi-squared goodness of fit test doesn't scale to high-cardinality variables (10,000+ categories) as it starts to over-reject. Are there similar tests that scale better or are more reliable for many-many categories? Also, since chi-squared requires at least 5 examples per category in the expected vector, it can't factor in differences where some categories are exclusive to either the expected or the observed. Are there alternative tests that can factor in cases where the expected frequency is 0?

[AntoineSoetewey]

That’s a great point @vtfresh! The Chi-squared goodness-of-fit test does have limitations, especially with high-cardinality variables, where it can become unreliable and over-reject null hypotheses.

I haven’t personally tested alternatives for cases with thousands of categories, but from what I’ve found with some research, there seems to be some potential solutions such as:

1. G-test (likelihood ratio test): similar to the Chi-squared test but based on likelihood ratios. Some sources suggest it might handle large numbers of categories better, though I’m not sure how much it improves the situation in practice.
2. Monte Carlo or permutation-based tests: these methods generate empirical $p$-values rather than relying on asymptotic distributions, which could make them more robust for high-cardinality cases.
3. Bayesian approaches: using models like the Dirichlet-multinomial distribution might help when dealing with sparse data, but I haven’t worked with them myself.

As for cases where expected frequencies are zero in some categories, I’ve read that alternatives like KL Divergence (Kullback-Leibler Divergence) or Earth Mover’s Distance (Wasserstein Distance) might be worth exploring, since they measure distributional differences in different ways. However, I’m not sure how they compare in terms of statistical rigor or ease of implementation for this specific problem.

If you’re working with a dataset where this is an issue, I’d love to hear how you’re approaching it! There might be other solutions I haven’t come across yet.