subsampling LOO estimates with diff-est-srs-wor start

[avehtari]

Work in progress.

Tested with

• family="normal" and stats %in% c("elpd", "mlpd", "gmpd", "mse", "rmse")
• family="bernoulli" and stats %in% c("acc","pctcorr")
• family="poisson" usings trad and latent approaches
• tests pass expect some plot tests fail with svglite: ... Graphics API version mismatch

Notes

• n_loo matters only if validate_search = TRUE (with FALSE there is no speed advantage)
• in cv_varsel if nloo<n and fast PSIS-LOO result is not yet available, fast PSIS-LOO result is computed
• in cv_varsel if nloo<n, fast PSIS-LOO result is stored in slot $summaries_fast
• the subsampling indices are stored in slot $loo_inds
• the actual subsampling estimating happens in summary_funs.R get_stat()
• removed some NA checking and need to recheck if those need to put back
• improved quantiles for "mse" if the value is close to 0 (can't get negative lq)
• "auc" is not supported (complicated)

Next

• add support for incrementally increasing nloo?

tagging @n-kall

[avehtari]

I wanted to use R2, and as I had rewrote summary stats anyway, added R2 and made all mse, rmse, and R2 to use only normal approximation with as much shared computation as possible

With the added R2 support, this PR will close also #483

[n-kall]

I can take a look

[fweber144]

I have added some comments, but I'm not done with the review yet.

Besides, I think documentation needs to be updated (at least re-roxygenized, but also progressr should be mentioned), the vignettes perhaps as well, and I haven't run R CMD check (including the unit tests) yet. The NEWS file would also need to be updated.

[fweber144]

Tests are fine now (just pushed commits fcee136 to 458012e). Merging now.

[avehtari]

Wohoo! Thank you!

[fweber144]

I'll try to merge the changes from branches misc_from_fix-subsampling and mixed_deltas_plot as soon as possible, but some of them first need some adaptations (see comments above).

[fweber144]

Wohoo! Thank you!

Thank you, this was mostly your work!