Quarto Book: reference chunk data in another file

[dmenne]

This is a repost of https://community.rstudio.com/t/quarto-book-reference-chunk-data-in-another-file/131967/2 as suggested by cderv

In bookdown , I had a choice between KM and MK rendering approaches. In MK mode, I had access to chapters further up in the book.

Assuming I have a "global" chunk in one chapter, how do make it available to other files? I have tried to reference chunks using dependson, but could not get it to work.

---- Here some additional information to explain my standard procedure to write large reports ---

bookdown was not perfect in MK mode. When you had several chapters, everything was carried along, that's a mess. I typically only want to include only one chunk's content, typically in index.rmd, e.g. with global constants and database pool. My method worked well, but maybe was a bit clunky:

'Index.rmd' had one chunk, with `this_is_the_first_file = TRUE' defined. I use MK strategy.

In bookdown.yml:

before_chapter_script: ["before_chapter_script.R"]
after_chapter_script: ["after_chapter_script.R"]

before_chapter_script.R:

BeforeObjects = ls()

after_chapter_script.R:

if (this_is_the_first_file){
  this_is_the_first_file = FALSE
} else {
  rm(list = setdiff(ls(), BeforeObjects))
}

Jenny-don't-use-rm(list()), please forgive me. It worked.

[jjallaire]

My advice here would be to create a pre-render script (https://quarto.org/docs/projects/quarto-projects.html#pre-and-post-render-scripts) that creates an .RData file with the values you want. I'd then just read the .RData file in the setup chunk of each chapter that wants to consume it.

[dmenne]

That's what I did in the old (pre KM/MK) days. It would be more elegant though if one could use the "dependons" mechanism by namespacing it, something like dependson='index::setup

[dmenne]

Reading from rda is not a good replacement for a common initialization: library(), par() etc are not exported, so I have to split the initialization between data(and functions) and the rest.

[dmenne]

Correction: bad example, par() never leaves a chunk.

[jjallaire]

Would it work for your scenario to put your common initialization code in .Rprofile ?

Another approach would be to source() a common init script in the setup chunk of pages. This is definitely less convenient than adding a before chapter script but would get the job done for now until we implement this.

[dmenne]

Sourcing works technically, but not so much in practice. In my typical case - medical reporting -, I read in "global" data for a report and do data preparation in index.rmd; this often needs a few seconds. Then I work on individual chapters first, but later about 30% of the time I have activated multiple chapters to do overlapping work, always keeping index.rmd included. Sourcing would read index.rmd multiple times.

Reading data from prepared .rda and initialization by source() is ok, I tried it by converting an old report. Despite the nifty features quarto provides, I am not sure if I would switch from bookdown, though. Nevertheless, great work which I will keep in eye as everything since ColdFusion.

[peterejkemp]

same issue here, I'm dealing with a very large dataset, ~20 seconds to load even in .rds format. Having 12 chapters means 20 * 12 seconds of pure loading time, let alone all the packages.

[dmenne]

Not exactly the same issue, because in this case my workaround below (#431 (reply in thread)) does not help. You need "in-memory document globals" , but before-script does not help, because the results are not available to the document.

[cderv]

@peterejkemp Just sharing ideas that may help if this is a data.frame you are storing, there are formats that are more efficient for storage and reading than rds.
See arrow for example;

For large dataset, using an approach of storing in the in-file database to request when needed (made easy with dplyr and dbplyr for example), then in-memory loading would be quick as less data. This is tricks I used for RAM limited environment.

Anyway, current design is for Quarto to render each chapter / document in an independent session so loading shared data in it is required. Optimizing this is a good idea when working with Quarto in this situation for now.

[jjallaire]

So I think what you really need is back to using a single render for the entire book, then split the chapters (as bookdown does). We don't currently support this model unfortunately (it's a non-trivial amount of work so not something we can knock out quickly)

[dmenne]

Probably not worth the effort - as I see the current concept's strong point is independence of the chapters. Some concept like an implicit

#ifndef anton_h
  # include "anton.h"
#endif

would be fine.

[dmenne]

My playground, assume index.rmd (same with qmd)

---
title: 'My cache'
---

# index.rmd

```{r, uncached_intro, echo = FALSE, include=FALSE, cache = FALSE}
# Cannot be cached, when referred to it is simply sourced 
library(nlme)
library(dplyr)
par(bty = "l") # No effect whatever because only within chunk 
```

```{r, cached_intro, echo = FALSE, include=FALSE, cache = TRUE}
options(warn = -1) 
hallo = function() {cat("I am in hallo\n")}

# Assume this is a time-consuming task
od = Orthodont %>%
  as_tibble() # And much more cleanup taske
```

```{r, echo = FALSE}
plot(od$age, od$distance)
lme(distance~age+Sex, random = ~1|Subject, data = od)
warning("This warning is not printed")
```

I other files, refer to these as:

ref.label=c('index:uncached_intro', 'index:uncached_intro')

Uncached chunks are sourced, cached are read from cache. No need to rewrite the code to process all file, nice separation as in quarto works fine.

[dmenne]

This works with rmd, but I should have tested it more carefully with qmd/Quarto. It fails. Hoping for some future feature.

https://community.rstudio.com/t/quarto-book-start-equivalent/144196/2

[dmenne]

As jjallaire mentioned, there is a pre-render and post-render (the original link moved, so I missed it a first)

https://quarto.org/docs/reference/projects/options.html

But, too bad, it is not useful because of this:

https://community.rstudio.com/t/quarto-book-start-equivalent/144196/2

[dmenne]

Here is my workaround (ugly...)

# ----- file common.R -----------------
# This file must be sourced in each qmd
cache_dir = "_common"
this_file = paste0(stringr::str_replace(cache_dir, "_", ""), ".R")
stopifnot(file.exists(this_file))
database_file = "mybook.sqlite"
# Create empty database if it does not exist (can be removed)
if (!file.exists(database_file)) {
  channel = RSQLite::dbConnect(RSQLite::SQLite(), database_file)
  DBI::dbExecute(channel, "CREATE TABLE patient ( patient_id CHAR PRIMARY KEY);")
  DBI::dbWriteTable(channel, "mtcars", mtcars[1:5, ], overwrite = TRUE)
  RSQLite::dbDisconnect(channel)
  rm(channel)
}

dir.create(cache_dir, FALSE)

# Get changed date of relevant files and database
# Add more dependencies here
to_hash = c(file.mtime(database_file),file.mtime(this_file))

hash_file = file.path(cache_dir, paste0(digest::digest(to_hash), ".rdata"))
# The following is never set in 1.3.20 (did I miss something?)
build_all = nzchar(Sys.getenv("QUARTO_PROJECT_RENDER_ALL"))

if (file.exists(hash_file) && !build_all) {
  # Hash file is valid
  load(hash_file)
} else {

  # Recreate hash
  cat("Creating hash file ", hash_file, "\n")
  # Cleanup
  file.remove(list.files(cache_dir, "*.rdata", full.names = TRUE))

  # Add all globals here
  hallo = function() {cat("I am  in hallo\n")}
  library(tibble)
  od = nlme::Orthodont %>%
    as_tibble() # And much more cleanup tasks

  Sys.sleep(10) # Fake time consuming stuff
  suppressWarnings(rm(cache_dir, to_hash, this_file, build_all))
  save.image(hash_file)
}

# Add all transient stuff here
channel = RSQLite::dbConnect(RSQLite::SQLite(), database_file)
# Must call dbDisconnect(channels) in qmd
library(tidyverse)
library(RSQlite)

And in index.qmd:

```{r}
#| echo: false
source("common.R")
DBI::dbReadTable(channel, "mtcars")
DBI::dbDisconnect(channel)
hallo()
head(od)
# This environment variable is NOT set on project
cat(Sys.getenv("QUARTO_PROJECT_RENDER_ALL"), "\n")
```

[follhim]

are there any updates on this? has anyone able to make a global render work?

[bryce-carson]

I don't think so, @follhim. I just tried utilizing dependson, autodep, and cache-globals in a minimal, reproducible example to test exactly this.

The documentation doesn't imply that variable caching doesn't work across documents. As a literate programming system which is much further away from Sweave's noweb heritage than I'd like, it's a bit of a miff that I'll need to implement a custom handler to get the variables from an explicit chunk, let alone implicit as I'd hoped.

---
title: "test-inter-document-dependencies"
---

This is a Quarto website **_specifically written to test
inter-document chunk dependencies_**.

To learn more about knitr caching in Quarto visit
<https://quarto.org/docs/reference/cells/cells-knitr.html#cache>.

😭 It doesn't work.

```{r}
#| label: define-global
#| cache: true
global <- "`global`"
```

---
title: "About"
---

About this site: it is a twenty-line Quarto website project used to
test inter-document dependencies. "Can we effectively use variables
from other documents' R chunks?"

```{r}
#| cache-globals: "global"
#| dependson: "define-global"
sprintf("%s was not defined in about.qmd!", global)
```

[cderv]

@bryce-carson Where did you found about cache-globals?

TO my knowledge, this is not a knitr option and not a Quarto cell option either.

So I would not expect this to work at all if this does not exist.

Where did you find an example of this that gave you the impression that this is working?

@follhim A reminder on how computation works

• For R, knitr will be used. Quarto will open a new R session per document and run knitr to evaluate all the cells based on their configurations. The R session will be closed after, so any state that needs to persist needs to go out of the R workspace memory and be written to disk or sent to a database, or else.

• knitr has a caching mechanism. This is independent of Quarto. The implicit caching is per document. Some documentation are here: https://bookdown.org/yihui/rmarkdown-cookbook/cache.html
  It would be a feature request to knitr to make the implicit caching work across documents. However, it can probably be configured today to maybe work - basically, implicit caching invalidation takes all cell options as a key for the invalidation, and this includes knitr's cells option specific to the document like fig-path.

• knitr also offers explicit caching through the use of xfun::cache_exec() See man page: https://git.yihui.org/xfun/manual.html#sec:man-cache_exec. This is explicit because

  • The default cache ID is based on cell content only
  • ID can be be specified manually
    Documentation for previous xfun::cache_rds() is still relevant and shows examples: https://bookdown.org/yihui/rmarkdown-cookbook/cache-rds.html

  With this explicit approach, you can share content between documents using a global cache path and common IDs.

• For full content reuse, and cell dependency support with caching, I would really give a try to targets: https://books.ropensci.org/targets.

  • This tool offers a way to build a pipeline for the data computation that will handle caching, invalidation, etc...
  • It integrates with tools like Quarto where any step can be made a dependency of a cell: https://books.ropensci.org/targets/literate-programming.html
    Caching will be global; the same data can be reused in multiple documents. Invalidating the data makes targets aware that all dependent documents need to be re-rendered.

I wanted to share this to show that if Quarto may seem to have limitations with this type of feature, this is currently because those needs are answered by external tooling available for the computation engine, not directly in Quarto.

Maybe we'll be able to integrate better in the future, but I believe this works already quite well. It definitely lacks examples and documentation, and I'll remember to provide some.

Hope it helps

[bryce-carson]

@bryce-carson Where did you found about cache-globals?

TO my knowledge, this is not a knitr option and not a Quarto cell option either.

So I would not expect this to work at all if this does not exist.

The Quarto documentation on caching: https://quarto.org/docs/reference/cells/cells-knitr.html#cache. It directly documents the cache-globals cell option, @cderv.

Where did you find an example of this that gave you the impression that this is working?

As I stated, "the [Quarto] documentation doesn't imply that variable caching doesn't work across documents."

A couple years have passed since I last used knitr, as I have been using noweb as my literate programming system. knitr was inspired by Sweave, which was inspired by noweb. I made some unsupported assumptions about Quarto's (and thereby knitr's) capabilities, including how Quarto dealt with its computational engines; the logical connection between the fact it supports either knitr or Jupyter and that Quarto does not processing of its own was not made in my mind.

[bryce-carson]

@follhim A reminder on how computation works

* For R, knitr will be used. Quarto will open a new R session per document and run knitr to evaluate all the cells based on their configurations. The R session will be closed after, so any state that needs to persist needs to go out of the R workspace memory and be written to disk or sent to a database, or else.

* **knitr** has a caching mechanism. This is independent of Quarto. The implicit caching is per document. Some documentation are here: https://bookdown.org/yihui/rmarkdown-cookbook/cache.html
  It would be a feature request to knitr to make the implicit caching work across documents. However, it can probably be configured today to maybe work - basically, implicit caching invalidation takes all cell options as a key for the invalidation, and this includes knitr's cells option specific to the document like `fig-path`.

* **knitr** also offers explicit caching through the use of `xfun::cache_exec()` See man page: https://git.yihui.org/xfun/manual.html#sec:man-cache_exec. This is explicit because
  
  * The default cache ID is based on cell content only
  * ID can be be specified manually
    Documentation for previous `xfun::cache_rds()` is still relevant and shows examples: https://bookdown.org/yihui/rmarkdown-cookbook/cache-rds.html
  
  With this explicit approach, you can share content between documents using a global cache path and common IDs.

* For full content reuse, and cell dependency support with caching, I would really give a try to **targets**: https://books.ropensci.org/targets.
  
  * This tool offers a way to build a pipeline for the data computation that will handle caching, invalidation, etc...
  * It integrates with tools like Quarto where any step can be made a dependency of a cell: https://books.ropensci.org/targets/literate-programming.html
    Caching will be global; the same data can be reused in multiple documents. Invalidating the data makes targets aware that all dependent documents need to be re-rendered.

I wanted to share this to show that if Quarto may seem to have limitations with this type of feature, this is currently because those needs are answered by external tooling available for the computation engine, not directly in Quarto.

Maybe we'll be able to integrate better in the future, but I believe this works already quite well. It definitely lacks examples and documentation, and I'll remember to provide some.

Hope it helps

Thank you, @cderv. Those are good links, especially targets given your claims about it.