Add LOO Difference Plot

[ParadaCarleton]

Implement plot from Bayesian workflow paper

[ParadaCarleton]

Closes #127

[codecov-commenter]

Codecov Report

❌ Patch coverage is 95.65217% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.94%. Comparing base (a59519c) to head (2fb210d).

Files with missing lines	Patch %	Lines
R/loo_difference_plot.R	95.65%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #178      +/-   ##
==========================================
+ Coverage   92.86%   92.94%   +0.08%     
==========================================
  Files          31       32       +1     
  Lines        3041     3133      +92     
==========================================
+ Hits         2824     2912      +88     
- Misses        217      221       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jgabry]

Hey Carlos, thanks for working on this! There are a bunch of small things I put in review comments (most of it related to things that matter for package development but not necessarily when writing code outside of package development), but this is a really really great start. Once we sort out some of smaller issues I commented on we can then get some other people, e.g. @avehtari or the other loo package authors to also take a look and see what they think about the plot itself and the different options provided.

[ParadaCarleton]

@avehtari I've added an example of a plot using the IQ dataset from rstanarm:

The first model only has a constant intercept; the second model also includes a term for the mother's IQ. At the edges, the constant-based model is more accurate than the full model, while the IQ-based model is stronger in the center. This seems counterintuitive, until you realize that this reveals there is significant measurement error in IQ tests: extreme values are probably caused by mismeasurement and need to be adjusted for regression to the mean.

Other examples:

[avehtari]

The plots looks great! Thanks for making this PR!

[ParadaCarleton]

The plots looks great! Thanks for making this PR!

Thanks! Do you happen to know if the spatial data from the Bayesian visualization paper can be added to the loo package? I think it's a good example, but if it can't, I can remove it.

[jgabry]

Sorry for the delay in responding to this. Regarding the issue of including the data from the visualization paper, I think we should avoid adding it to the package but I think there's still a way we can still use it. Basically, we should use a different example in the Examples section in the documentation, but then we can add an example using the visualization paper data in one of the vignettes. This works because in the vignette we're allowed to download data that isn't included in the package. For example, we do this already in one of the vignettes with some other data:

loo/vignettes/loo2-with-rstan.Rmd

Lines 95 to 96 in 0117858

	url <- "http://stat.columbia.edu/~gelman/arm/examples/arsenic/wells.dat"
	wells <- read.table(url)

So we could do something similar and download the visualization paper data from https://github.com/jgabry/bayes-vis-paper. @ParadaCarleton What do you think about that option?

[ParadaCarleton]

Sorry for the delay in responding to this. Regarding the issue of including the data from the visualization paper, I think we should avoid adding it to the package but I think there's still a way we can still use it. Basically, we should use a different example in the Examples section in the documentation, but then we can add an example using the visualization paper data in one of the vignettes. This works because in the vignette we're allowed to download data that isn't included in the package. For example, we do this already in one of the vignettes with some other data:

loo/vignettes/loo2-with-rstan.Rmd

Lines 95 to 96 in 0117858

	url <- "http://stat.columbia.edu/~gelman/arm/examples/arsenic/wells.dat"
	wells <- read.table(url)

So we could do something similar and download the visualization paper data from https://github.com/jgabry/bayes-vis-paper. @ParadaCarleton What do you think about that option?

Sounds good! I've removed the example. Where do you think I should put this -- should I add it to an existing vignette or create a new one?

[jgabry]

Where do you think I should put this -- should I add it to an existing vignette or create a new one?

Hmm, on the one hand I think it could be nice to have it in the main introduction vignette https://mc-stan.org/loo/articles/loo2-example.html. On the other hand, this would be using a totally different data set than the example in that vignette, so that's not ideal and maybe a new vignette is preferable.

I think if we go the route of a new vignette then it shouldn't just be about this one plot but rather about visualizing loo output in general (e.g., this plot, the Pareto k diagnostic plot, perhaps some of the loo related plots in bayesplot, etc.). But that would be more work than just adding this to an existing vignette so it depends how much you feel like working on this (no pressure!).

[ParadaCarleton]

@jgabry btw, I think we should merge this, unless there's some change you'd like to see. I might be able to get to building a vignette before classes start again, but given that I'm focused on adding new features to ParetoSmooth.jl I'm not sure I'll be able to. We can probably add this to some other vignette later.

[avehtari]

@ParadaCarleton and @jgabry , what is the status of this PR?

[VisruthSK]

My changes were assisted by ChatGPT (almost exclusively for writing code I didn't want to type out, not for ideas):

Renamed args to loo_1 and loo_2, and other small renames.
Removed vertical jitter and made labels jitter with points so they would be aligned.
Changed labelling args/semantics, don't call it outlier thresholding.
Added some arg checking.
Added tests.

Outdated