Render fails for file in subdirectory for book with freeze

[cwickham]

Bug description

Rendering a file in a subdirectory from the project root fails:

/book-freeze $ quarto render notebooks/r.qmd
Error in rmarkdown:::abs_path(input) : 
  The file 'notebooks/r.qmd' does not exist.
Calls: .main -> execute -> setwd -> dirname -> <Anonymous>
Execution halted
ERROR: Error
    at renderFiles (file:///Users/wickhamc/Documents/posit/quarto-cli/src/command/render/render-files.ts:354:23)
    at eventLoopTick (ext:core/01_core.js:183:11)
    at async render (file:///Users/wickhamc/Documents/posit/quarto-cli/src/command/render/render-shared.ts:102:18)
    at async Command.fn (file:///Users/wickhamc/Documents/posit/quarto-cli/src/command/render/cmd.ts:248:26)
    at async Command.execute (file:///Users/wickhamc/Documents/posit/quarto-cli/src/vendor/deno.land/x/cliffy@v0.25.4/command/command.ts:1790:7)
    at async quarto (file:///Users/wickhamc/Documents/posit/quarto-cli/src/quarto.ts:149:3)
    at async file:///Users/wickhamc/Documents/posit/quarto-cli/src/quarto.ts:163:5
    at async mainRunner (file:///Users/wickhamc/Documents/posit/quarto-cli/src/core/main.ts:35:5)
    at async file:///Users/wickhamc/Documents/posit/quarto-cli/src/quarto.ts:153:3

Works fine if any of the following is changed:

• type: book -> type: default
• notebooks/r.qmd is added to the chapter listing in _quarto.yml
• freeze: auto removed

Steps to reproduce

Repo at: https://github.com/cwickham/book-freeze

Run:

quarto render notebooks/r.qmd

Expected behavior

Should render the same as rendering from inside the subdirectory:

book-freeze/notebooks $ quarto render r.qmd 


processing file: r.qmd
1/3         
2/3 [r-code]
3/3         
output file: r.knit.md

pandoc 
  to: html
  output-file: r.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  
metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  title: R
  
Output created: r.html

Actual behavior

Error as above.

Your environment

• OS: Mac OS Sonoma 14.0

Quarto check output

Quarto 99.9.9
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.1.11: OK
      Dart Sass version 1.69.5: OK
      Deno version 1.37.2: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 99.9.9
      Path: /Users/wickhamc/Documents/posit/quarto-cli/package/dist/bin

[✓] Checking tools....................OK
      TinyTeX: v2023.09
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: TinyTex
      Path: /Users/wickhamc/Library/TinyTeX/bin/universal-darwin
      Version: 2023

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.12.0
      Path: /Library/Frameworks/Python.framework/Versions/3.12/bin/python3
      Jupyter: 5.5.0
      Kernels: ir, julia-1.9, python3

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........OK
      Version: 4.3.2
      Path: /Library/Frameworks/R.framework/Resources
      LibPaths:
        - /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
      knitr: 1.45
      rmarkdown: 2.21

[✓] Checking Knitr engine render......OK

[cderv]

I can reproduce, but removing freeze: auto does not make it works, so I don't think this is related to freeze.

I do think this is related to project render targets for books, as adding to chapters will make it works.

I tried adding to project.render field but it seems render has no impact on this - only files in chapters can be seen as render target for quarto render

project:
  type: book
  render: 
    -  notebooks/r.qmd

book:
  chapters: 
    - index.qmd

I wonder if this should work... Otherwise, how to do make some .qmd resources render for book project if anything needs to be a chapter ? 🤔

Also, I can confirm it happens only in our R code as quarto render notebooks/py.qmd will work. So this seems to be related in how we are passing the working directory for R code. I'll look into it.

[mcanouil]

I wonder if this should work... Otherwise, how to do make some .qmd resources render for book project if anything needs to be a chapter ? 🤔

FYI:

• Preview PDF chapter in quarto book #7762
• Render pdf one file at a time #8895

[cderv]

Thanks @mcanouil - it seems project.render key can't really be used, and only book.chapters allow for render target customization in books... 🤔 That is another issue than this one though

[cderv]

I looked into this a bit more.

• This works ok with 1.4
• This does not work with 1.5

So this is a regression from a recent change in 1.5, possibly in how project context is now computed

So I looked for the breaking change, and this is at a28ac25 from merging PR

• create ProjectContext in single-file mode to guarantee engine and target caching #8771

@cscheid when rendering in a Book project, any file you call quarto render on should be using the Quarto root project as project dir right ?

When doing quarto render notebooks/r.qmd (or an another file notebooks/py.qmd), the context.project.dir will be notebooks/ folder

quarto-cli/src/command/render/render-files.ts

Line 204 in f9cdaa9

projectDir: context.project?.dir,

This is now in my debug testing

> context.project?.dir
'C:\Users\chris\AppData\Local\Temp\quarto\book-freeze\notebooks'

while before it was

> context.project?.dir
undefined

I would like to confirm this is the expected value for you. I would have expected the root directory as the project dir for all files.

Based on this, I'll adapt our R code which is not correct anyway with this new project context change.

We use this information about project dir to set the working directory for R. We relied on undefined before, and now it has changed for R execution from Deno.cwd() to projectDir

quarto-cli/src/execute/rmd.ts

Lines 241 to 243 in e41b4c9

	// establish cwd for our R scripts (the current dir if there is an renv
	// otherwise the project dir if specified)
	const cwd = withinActiveRenv() ? Deno.cwd() : projectDir ?? Deno.cwd();

The input pass to R s context.target.input and is still relative to quarto project root

input:
  'notebooks\\r.qmd'

and so it raises the R error when we do try to "fix" the path in R context

quarto-cli/src/resources/rmd/rmd.R

Lines 53 to 56 in e41b4c9

	# change to input dir and make input relative
	oldwd <- setwd(dirname(rmarkdown:::abs_path(input)))
	on.exit(setwd(oldwd), add = TRUE)
	input <- basename(input)

We can discuss this live if this is easier. Thanks for your input

[cderv]

Adding some notes for later references about behavior I found while trying to understand this.

Issue is we now always have a project, even for single file render. Doing the following would restore the previous behavior

diff --git a/src/command/render/render-files.ts b/src/command/render/render-files.ts
index 35e7b49a7..961752920 100644
--- a/src/command/render/render-files.ts
+++ b/src/command/render/render-files.ts
@@ -223,7 +223,9 @@ export async function renderExecute(
     dependencies: resolveDependencies,
     libDir: context.libDir,
     format: context.format,
-    projectDir: context.project?.dir,
+    projectDir: !context.project.isSingleFile
+      ? context.project?.dir
+      : undefined,
     cwd: flags.executeDir || dirname(normalizePath(context.target.source)),
     params: resolveParams(flags.params, flags.paramsFile),
     quiet: flags.quiet,

So the bottom of the issue is deeper I think

First, R code logic was built around single render file having no project context. We may need to revisit this for 1.5, knowing now any render will have a project context, and the project.dir will be the folder of the input for a single file render

quarto-cli/src/project/types/single-file/single-file.ts

Lines 30 to 31 in a28ac25

	const result: ProjectContext = {
	dir: normalizePath(dirname(source)),

This has impact on where R code should be executed from. Previous it was Deno.cwd() unless in a project. We discussed this previously but we need to settle on this now.

Then, for book project, if quarto render is called on a file not listed as projectInputFiles(), it will be considered outside of a book project.

quarto-cli/src/project/project-context.ts

Lines 252 to 259 in a28ac25

	const { files, engines } = projectInputFiles(dir, projectConfig);
	// if we are attemping to get the projectConext for a file and the
	// file isn't in list of input files then return undefined
	const fullPath = normalizePath(path);
	if (Deno.statSync(fullPath).isFile && !files.includes(fullPath)) {
	return undefined;
	}

This will trigger the new single file context causing the issue here

quarto-cli/src/command/render/render-shared.ts

Lines 102 to 103 in a28ac25

	// NB: singleFileProjectContext is currently not fully-featured
	context = singleFileProjectContext(path, nbContext, options.flags);

So there is the question of this logic for books project, and the potential impact it can have on existing book project. Maybe the impact on R code is the only one.

[cderv]

😞 wrong linking - sorry.

I'll know what to do now and I'll push a fix

[cscheid]

This is fixed on main.