Quarto Book: reference chunk data in another file

[dmenne]

This is a repost of https://community.rstudio.com/t/quarto-book-reference-chunk-data-in-another-file/131967/2 as suggested by cderv

In bookdown , I had a choice between KM and MK rendering approaches. In MK mode, I had access to chapters further up in the book.

Assuming I have a "global" chunk in one chapter, how do make it available to other files? I have tried to reference chunks using dependson, but could not get it to work.

---- Here some additional information to explain my standard procedure to write large reports ---

bookdown was not perfect in MK mode. When you had several chapters, everything was carried along, that's a mess. I typically only want to include only one chunk's content, typically in index.rmd, e.g. with global constants and database pool. My method worked well, but maybe was a bit clunky:

'Index.rmd' had one chunk, with `this_is_the_first_file = TRUE' defined. I use MK strategy.

In bookdown.yml:

before_chapter_script: ["before_chapter_script.R"]
after_chapter_script: ["after_chapter_script.R"]

before_chapter_script.R:

BeforeObjects = ls()

after_chapter_script.R:

if (this_is_the_first_file){
  this_is_the_first_file = FALSE
} else {
  rm(list = setdiff(ls(), BeforeObjects))
}

Jenny-don't-use-rm(list()), please forgive me. It worked.

[jjallaire]

My advice here would be to create a pre-render script (https://quarto.org/docs/projects/quarto-projects.html#pre-and-post-render-scripts) that creates an .RData file with the values you want. I'd then just read the .RData file in the setup chunk of each chapter that wants to consume it.

[dmenne]

That's what I did in the old (pre KM/MK) days. It would be more elegant though if one could use the "dependons" mechanism by namespacing it, something like dependson='index::setup

[dmenne]

Reading from rda is not a good replacement for a common initialization: library(), par() etc are not exported, so I have to split the initialization between data(and functions) and the rest.

[dmenne]

Correction: bad example, par() never leaves a chunk.

[jjallaire]

Would it work for your scenario to put your common initialization code in .Rprofile ?

Another approach would be to source() a common init script in the setup chunk of pages. This is definitely less convenient than adding a before chapter script but would get the job done for now until we implement this.

[dmenne]

Sourcing works technically, but not so much in practice. In my typical case - medical reporting -, I read in "global" data for a report and do data preparation in index.rmd; this often needs a few seconds. Then I work on individual chapters first, but later about 30% of the time I have activated multiple chapters to do overlapping work, always keeping index.rmd included. Sourcing would read index.rmd multiple times.

Reading data from prepared .rda and initialization by source() is ok, I tried it by converting an old report. Despite the nifty features quarto provides, I am not sure if I would switch from bookdown, though. Nevertheless, great work which I will keep in eye as everything since ColdFusion.

[peterejkemp]

same issue here, I'm dealing with a very large dataset, ~20 seconds to load even in .rds format. Having 12 chapters means 20 * 12 seconds of pure loading time, let alone all the packages.

[dmenne]

Not exactly the same issue, because in this case my workaround below (#431 (reply in thread)) does not help. You need "in-memory document globals" , but before-script does not help, because the results are not available to the document.

[cderv]

@peterejkemp Just sharing ideas that may help if this is a data.frame you are storing, there are formats that are more efficient for storage and reading than rds.
See arrow for example;

For large dataset, using an approach of storing in the in-file database to request when needed (made easy with dplyr and dbplyr for example), then in-memory loading would be quick as less data. This is tricks I used for RAM limited environment.

Anyway, current design is for Quarto to render each chapter / document in an independent session so loading shared data in it is required. Optimizing this is a good idea when working with Quarto in this situation for now.

[jjallaire]

So I think what you really need is back to using a single render for the entire book, then split the chapters (as bookdown does). We don't currently support this model unfortunately (it's a non-trivial amount of work so not something we can knock out quickly)

[dmenne]

Probably not worth the effort - as I see the current concept's strong point is independence of the chapters. Some concept like an implicit

#ifndef anton_h
  # include "anton.h"
#endif

would be fine.

[dmenne]

My playground, assume index.rmd (same with qmd)

---
title: 'My cache'
---

# index.rmd

```{r, uncached_intro, echo = FALSE, include=FALSE, cache = FALSE}
# Cannot be cached, when referred to it is simply sourced 
library(nlme)
library(dplyr)
par(bty = "l") # No effect whatever because only within chunk 
```

```{r, cached_intro, echo = FALSE, include=FALSE, cache = TRUE}
options(warn = -1) 
hallo = function() {cat("I am in hallo\n")}

# Assume this is a time-consuming task
od = Orthodont %>%
  as_tibble() # And much more cleanup taske
```

```{r, echo = FALSE}
plot(od$age, od$distance)
lme(distance~age+Sex, random = ~1|Subject, data = od)
warning("This warning is not printed")
```

I other files, refer to these as:

ref.label=c('index:uncached_intro', 'index:uncached_intro')

Uncached chunks are sourced, cached are read from cache. No need to rewrite the code to process all file, nice separation as in quarto works fine.

[dmenne]

This works with rmd, but I should have tested it more carefully with qmd/Quarto. It fails. Hoping for some future feature.

https://community.rstudio.com/t/quarto-book-start-equivalent/144196/2

[dmenne]

As jjallaire mentioned, there is a pre-render and post-render (the original link moved, so I missed it a first)

https://quarto.org/docs/reference/projects/options.html

But, too bad, it is not useful because of this:

https://community.rstudio.com/t/quarto-book-start-equivalent/144196/2

[dmenne]

Here is my workaround (ugly...)

# ----- file common.R -----------------
# This file must be sourced in each qmd
cache_dir = "_common"
this_file = paste0(stringr::str_replace(cache_dir, "_", ""), ".R")
stopifnot(file.exists(this_file))
database_file = "mybook.sqlite"
# Create empty database if it does not exist (can be removed)
if (!file.exists(database_file)) {
  channel = RSQLite::dbConnect(RSQLite::SQLite(), database_file)
  DBI::dbExecute(channel, "CREATE TABLE patient ( patient_id CHAR PRIMARY KEY);")
  DBI::dbWriteTable(channel, "mtcars", mtcars[1:5, ], overwrite = TRUE)
  RSQLite::dbDisconnect(channel)
  rm(channel)
}

dir.create(cache_dir, FALSE)

# Get changed date of relevant files and database
# Add more dependencies here
to_hash = c(file.mtime(database_file),file.mtime(this_file))

hash_file = file.path(cache_dir, paste0(digest::digest(to_hash), ".rdata"))
# The following is never set in 1.3.20 (did I miss something?)
build_all = nzchar(Sys.getenv("QUARTO_PROJECT_RENDER_ALL"))

if (file.exists(hash_file) && !build_all) {
  # Hash file is valid
  load(hash_file)
} else {

  # Recreate hash
  cat("Creating hash file ", hash_file, "\n")
  # Cleanup
  file.remove(list.files(cache_dir, "*.rdata", full.names = TRUE))

  # Add all globals here
  hallo = function() {cat("I am  in hallo\n")}
  library(tibble)
  od = nlme::Orthodont %>%
    as_tibble() # And much more cleanup tasks

  Sys.sleep(10) # Fake time consuming stuff
  suppressWarnings(rm(cache_dir, to_hash, this_file, build_all))
  save.image(hash_file)
}

# Add all transient stuff here
channel = RSQLite::dbConnect(RSQLite::SQLite(), database_file)
# Must call dbDisconnect(channels) in qmd
library(tidyverse)
library(RSQlite)

And in index.qmd:

```{r}
#| echo: false
source("common.R")
DBI::dbReadTable(channel, "mtcars")
DBI::dbDisconnect(channel)
hallo()
head(od)
# This environment variable is NOT set on project
cat(Sys.getenv("QUARTO_PROJECT_RENDER_ALL"), "\n")
```