Dependence aware tests for ppc_loo_pit_ecdf

[florence-bockting]

Description

Context

LOO-PIT is used for model checking within the Bayesian workflow. The LOO-PIT values are asymptotically uniform (for continuous data) if the model is calibrated. Within this approach, each data point is iteratively held out; the model is then conditioned on the remaining data and the corresponding LOO predictive distribution is compared to the held-out point to test for departures from uniformity. A corresponding graphical uniformity test was developed by Säilynoja et al. (2022) and is implemented in the ppc_loo_pit_ecdf function in the bayesplot package. This function visualizes the empirical cumulative distribution function (ECDF) of the LOO-PITs, overlaid with simultaneous confidence intervals (creating an envelope) for a standard uniform sample.

Issue

The current approach assumes independence of LOO-PIT values which is not valid (Marhunenda et al., 2005). The corresponding graphical test yields an envelope that is too wide, reducing the test's ability to reveal model miscalibration.

Suggested solution (Content of this PR)

Tesso & Vehtari (2026, see preprint) propose three testing procedures that can handle any dependent uniform values and provide an updated graphical representation that uses color coding to indicate influential regions or most influential points of the ECDF. This PR implements the new development, by replacing the current ppc_loo_pit_ecdf implementation.

TODOs

• updated ppc_loo_pit_ecdf() function in ppc-loo.R
• add new helper functions for computing uniformity tests in helpers-ppc.R
• add unittests for plotting and helper functions in test-helpers-ppc.R and test-ppc-loo.R
• add visual regression tests for ppc_loo_pit_ecdf() in test-ppc-loo.R
• update documentation
• new or updated vignette with explanation of differences between new and old method (TODO)
• deprecation suggestion of old method
  • P1: old method (default) and new method available via method argument
  • P2: new method (default) and old method available via method argument
  • P3: old method is removed

[jgabry]

Because this copies a lot of code from stan-dev/posterior#435 and other parts of the posterior package, I think we should wait to merge it until that posterior PR is merged so we can remove the extra code before adding it to bayesplot.

Related to this, I spoke with @avehtari and we think that it could be useful to have all the uniformity test code in posterior also because it could be used without making a plot. So that code could be added to stan-dev/posterior#435 and then used in bayesplot.

[florence-bockting]

Because this copies a lot of code from stan-dev/posterior#435 and other parts of the posterior package, I think we should wait to merge it until that posterior PR is merged so we can remove the extra code before adding it to bayesplot.

Yes, I am totally with you.

Related to this, I spoke with @avehtari and we think that it could be useful to have all the uniformity test code in posterior also because it could be used without making a plot. So that code could be added to stan-dev/posterior#435 and then used in bayesplot.

Alright. I moved the respective code from this PR to stan-dev/posterior#435