Likelihoods from different datasets are not comparable

[piklprado]

Hi Carl,

If I understood correctly, you calculated likelihood ratios of the same model fitted to different data sets (e.g., your tutorial defines deviances as "-2 times the log likelihood of data fit under A that had been simulated under A, minus the log likelihood of fits under A simulated under B model "). Maybe I am missing something, but I wonder how this can be done, as likelihoods are conditional to data, and so likelihoods from different datasets are not comparable.

[cboettig]

@piklprado We should be careful what we mean by 'different datasets' here though. In this context you should think of these as stochastic realizations of the 'same data', regardless of the model being used.

For instance, if I simulate two different datasets from model A, but both with the same number of observations at the same time points, I think you would agree the likelihoods are comparable, even though not identical. If on the other hand I simulate more points in one of those simulations, these likelihoods are not comparable. The generating model itself is not the issue -- after all, the same data can be generated by multiple models so it cannot make a difference.

Not sure if this heuristic explanation helps you, but maybe? Otherwise it can always be useful to write down a trivial example (say, simulating from two normal distributions). After all, these simulations are just a way to generate distributions predicted by the model by bootstrapping -- if the models are simple enough (such as the case of the normal distribution) we could simply write down the distributions directly. You should be able to show, for instance, that in the case of models being two normal distributions of equal variances and different and that given sufficient replicates this approach converges to a t-test.

[piklprado]

As likelihoods are conditional to observed data, not to the unknow process that generated the data, I stil think they are not comparable, at least through likelihood ratios (See Royall or Edwards books, for example). You can of course think on likelihoods got from multiple realizations of a given process as random variables, as I think you did. In doing so you can ask what is the probability of a given process generating data that makes you choose the wrong model (probability of misleading evidence sensu Royall, [pme]).

[piklprado]

When you look at the distribution of deviances of model A fit to data generated under A and B you are implicitly assuming that process A and B have the same probability. But if you have empirical data you can get the posterior probabilities of each model, and then use then in your ROC curves. You could also assume a different prior for the models, for example weighted by the costs of type I and II errors.