Pandoc memory issue in amd64 docker container emulated on M1 Mac

[jimjam-slam]

Bug description

This might be better directed to @rocker-org, but I'll start here! I've endeavoured to set up a VSCode devcontainer for our Quarto projects based on the R community devcontainer template (which in turn uses the rocker/r-ver:4.2 Docker image). Quarto is included in the image, but attempts to render a test case lead to a Pandoc error:

$ quarto render index.qmd

Killed
Error in strsplit(info, "\n")[[1]] : subscript out of bounds
Calls: .main ... pandoc_available -> find_pandoc -> lapply -> FUN -> get_pandoc_version
In addition: Warning message:
In system(paste(shQuote(path), "--version"), intern = TRUE) :
  running command ''/usr/local/bin/pandoc' --version' had status 137
Execution halted

Even just running pandoc --version, using the version of Pandoc that Quarto appears to be calling, fails:

$ pandoc --version

Killed

I should say that the test case renders fine with my own local Quarto installation!

Checklist

• Please include a minimal, fully reproducible example in a single .qmd file? Please provide the whole file rather than the snippet you believe is causing the issue.
• Please format your issue so it is easier for us to read the bug report.
• Please document the RStudio IDE version you're running (if applicable), by providing the value displayed in the "About RStudio" main menu dialog?
• Please document the operating system you're running. If on Linux, please provide the specific distribution.

[jimjam-slam]

Rendering a doc in the same docker image alone doesn't seem to trigger this problem, so perhaps it's specific to the community dev container spec:

$ docker run rocker/r-ver:4.2 echo -e '---\ntitle: Hello world\n---\n\nOh hi there' > index.qmd; cat index.qmd; quarto render index.qmd
---
title: Hello world
---

Oh hi there
pandoc 
  to: html
  output-file: index.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  
metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  title: Hello world
  
Output created: index.html

Does Quarto typically use an internally bundled version of Pandoc on Linux, or one that's available to the system?

[jimjam-slam]

Okay, I'm very confused - after doing some testing with Docker, I can now re-open the devcontainer in VSCode and things work fine (including pandoc --version). Not sure what happened here! Might try to scrap and rebuild a couple of times to verify.

[jimjam-slam]

Trying to run the simpler test case works, and pandoc --version works, but if I then go back to trying to render the original test case, Quarto fails again (although pandoc --version keeps working). Very confusing 🤔

[jimjam-slam]

Oops! Looks like pandoc error 137 is an out of memory error. Increasing the RAM available to Docker from 8 GB to 12 GB did the trick.

I am a little surprised, though: there is quite a long delay before quarto render index.qmd starts producing output that I don't get locally, and monitoring memory usage with htop during the render, it climbs from a baseline of 1.8 GB to 8.8 GB before receding (and after it recedes, I start getting console output from Quarto). What could be going on to spike memory usage like that?

[cscheid]

We've observed pandoc uses a lot of memory when self-contained: true. I don't understand why it's so high (there's probably unnecessary laziness in the Haskell code while dealing with large strings), but that's what I suspect.

[jimjam-slam]

Interesting! This could be a separate issue: when I use self-contained: false (or omit it), the memory spikes to 9 GB but then falls again just before I see processing file: index.qmd. If I use self-contained: true, it spikes but then remains high until the render finishes.

[jimjam-slam]

Not sure if this helps, but this is what I see in htop when memory is spiking (sorted descending by memory use). Maybe I'm running the container on the wrong platform?

[jimjam-slam]

In fact, just running pandoc --version causes a similar memory spike:

With the expected output about 10 seconds later:

$ pandoc --version
pandoc 2.19.2
Compiled with pandoc-types 1.22.2.1, texmath 0.12.5.2, skylighting 0.13,
citeproc 0.8.0.1, ipynb 0.2, hslua 2.2.1
Scripting engine: Lua 5.4
User data directory: /home/rstudio/.local/share/pandoc
Copyright (C) 2006-2022 John MacFarlane. Web:  https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.

[jimjam-slam]

I should note that I'm on an M1 Mac, and this appears to be an x86_64 container, so this could be a weird Docker/Pandoc/platform edge case 🤷🏻‍♂️

$ uname -a
Linux 9137e99d112a 5.10.104-linuxkit #1 SMP PREEMPT Thu Mar 17 17:05:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[eitsupi]

This issue has nothing to do with the Rocker Project so I recommend changing the title.
I think it is related to Pandoc and Docker Desktop on arm mac's amd64 emulation.

[jimjam-slam]

@eitsupi It could be that the sensible thing is to close this issue altogether and open it with @pandoc, but I'll leave it for now (or at least until we confirm that it appears in Docker containers generally!). I don't intend it as a slight against Rocker :)