knitr-style code reuse for Jupyter-based documents

[allefeld]

Description

I recently discovered the knitr feature of reusing code from a previous chunk in a later one, see "Embed one chunk in another chunk" – and I'm loving it. The same also works in Quarto qmd-files using R.

My feature request is to implement the same or something similar for qmd-files using Jupyter kernels. That would have to happen before or during conversion of qmd into quarto_ipynb.

Motivation:

This feature is very useful for literate programming.

For example, I often write code to analyse data from a number of participants. With code reuse, I can first develop the analysis code step-by-step in separate chunks, applied to the data of one participant, then pull the code from the different chunks together into a function and apply that function to the data of the other participants. This is from a qmd file using R to preprocess eyetracking data :

```{r}
participant <- "P01"
```

Load the data.

```{r}
#| label: pp1
data <- vroom(
  paste0("raw_data/", participant, ".tsv.gz"),
  col_types = cols(
    KeyPressEventIndex = col_integer(),
    KeyPressEvent = col_character()
  )
)
```

Rows where `MediaName` is empty or where `StudioEvent` is `"ImageStart"` or `"ImageEnd"` are irrelevant. We filter them out.

```{r}
#| label: pp2
data <- filter(
  data,
  ! (is.na(data$MediaName) | data$StudioEvent %in% c("ImageStart", "ImageEnd"))
)
```

and so on, over many chunks.

Then after all steps are complete, I can write:

Put code together in a function.

```{r}
process_participant <- function(participant) {
  <<pp1>>
  <<pp2>>
  <<pp3>>
  <<pp4>>
  <<pp5>>
  <<pp6>>
  <<pp7>>
  <<pp8>>
  <<pp9>>
  <<pp10>>
  <<pp11>>
  <<pp12>>
  <<pp13>>
}
```

Apply function to all participants, combine data frames into one and save it.

```{r}
participants <- c(
  "P01", "P02", "P03" , "P05", "P06", "P07", "P08", "P10", "P11", "P12",
  "P13", "P14", "P15", "P16", "P17", "P18", "P19", "P20", "P21", "P22",
  "P23", "P24", "P25", "P26", "P27", "P28", "P29", "P30", "P31", "P32",
  "P33", "P34", "P35", "P36", "P37", "P38", "P39", "P40", "P41", "P42",
  "P43", "P44", "P45", "P46", "P47", "P48", "P49", "P50", "P51", "P52",
  "P54", "P55", "P56", "P57", "P58", "P59"
)
df <- bind_rows(lapply(participants, process_participant))
write_csv(df, "extracted_data/all.csv")
```

knitr replaces the <<label>> references by the contents of the respective chunks, and executes and shows in the output the resulting code.

Writing the code cumulatively like this is useful to give the output to a student to read, but also when refamiliarizing oneself with code after a pause.

[cscheid]

Thanks for the suggestion. Frankly, we're very unlikely to implement this, because it would make the jupyter execution through Quarto very incompatible with that of existing notebooks. This is a feature that specific kernels should implement themselves (some kind of jupyter magics, maybe?), and then Quarto's execution would inherit the behavior.

[allefeld]

I was afraid that you were going to say this, and I understand.

I also thought to implement this on the side of the kernel (for my own custom one). The problem is that as far as I can tell, the kernel doesn't have access to cell labels. So if the kernel sees a reference like <<pp1>>, it doesn't know what pp1 refers to.

Is that something that could be changed, communicating cell labels to the kernel?

[cscheid]

Is that something that could be changed, communicating cell labels to the kernel?

It's honestly not a very high-priority thing for us, but I think the Jupyter notebook protocol does allow for sideband communications, and this could be done. (IIRC, it's what we used for one of your custom Jupyter kernel requests in the past)

[cscheid]

Can you file an issue about this? Thanks.

[allefeld]

It just occurred to me: Maybe what I want could be achieved with a variant of the embed shortcode?
Differences: It would include from the same document, not another, and it would include not the output or output & complete code cell, but just the contents of the code cell.

[mcanouil]

Maybe you want to take a look at the content shortcode: https://prerelease.quarto.org/docs/authoring/contents.html