Script files not picked up by julia engine #10034

jkrumbiegel · 2024-06-17T14:11:21Z

Bug description

It appears that julia script files do not get sent to the julia engine even if specifying engine: julia in the frontmatter. Instead, they are run by the jupyter engine.

Steps to reproduce

I tried quarto render testfile.jl where testfile.jl had this content:

# %% [markdown]
# ---
# engine: julia
# ---

# %%
println("the result is $(1 + 1)")

Expected behavior

Julia engine would try to render this file

Actual behavior

Jupyter engine rendered the file.

Your environment

No response

Quarto check output

Quarto 1.5.40
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.2.0: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.41.0: OK
      Typst version 0.11.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.5.40
      Path: /Applications/quarto/bin

[✓] Checking tools....................OK
      TinyTeX: v2024.03
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: TinyTex
      Path: /Users/krumbiegel/Library/TinyTeX/bin/universal-darwin
      Version: 2023

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.12.2 (Conda)
      Path: /Users/krumbiegel/miniconda3/bin/python
      Jupyter: 5.5.0
      Kernels: julia-1.10, julia-1.8, julia-1.7, python3

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........OK
      Version: 4.3.2
      Path: /Library/Frameworks/R.framework/Resources
      LibPaths:
        - /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
      knitr: 1.45
      rmarkdown: 2.26

[✓] Checking Knitr engine render......OK

The text was updated successfully, but these errors were encountered:

cderv · 2024-06-17T19:22:57Z

For the context, currently those files are claimed by Jupyter engine

quarto-cli/src/execute/jupyter/jupyter.ts

Lines 148 to 151 in d5edb79

    
           claimsFile: (file: string, ext: string) => { 
        
             return kJupyterNotebookExtensions.includes(ext.toLowerCase()) || 
        
               isJupyterPercentScript(file); 
        
           },

and Julia claims no file

quarto-cli/src/execute/julia.ts

Line 66 in d5edb79

claimsFile: (file: string, ext: string) => false,

This is probably not clear in our doc, but as of now when using script as input to render, the engine option in those file is not used and this is too late - the file has already been claimed by an engine.

Does Julia engine support same Percent script syntax from Jupytext ? https://jupytext.readthedocs.io/en/latest/formats-scripts.html

Currently, we do detect .py and check if it has percent syntax. If so, Jupyter engine claims it.

If the same file could also be working for Julia engine, I think we should indeed parse the file header and detect the engine: julia specification t know which engine is claiming the file

jkrumbiegel · 2024-06-17T20:07:09Z

I did not add any claimsFile as the assumption was that the previous behavior with the jupyter backend should not be impacted. Otherwise I would've claimed normal julia codeblocks in a .qmd file as well.

Does Julia engine support same Percent script syntax from Jupytext

Yes, we do have support for that style

cderv · 2024-06-18T09:11:22Z

Thanks for the confirmation.

When we added Jupyter Percent Script support, we only associated it with the Jupyter engine, no matter is content.

So, we definitely need to adapt the claimFiles for Jupyter so that the Julia engine can also claim the .jl script.
I would say

Either we change default and Percent script .jl file should be claimed by Julia engine
Either we need to look into the Percent script to check which engine: julia is set in the field as we do for .qmd document.

For comparaison, engine: knitr does not support the same syntax of percent script. So any .R file with Percent script syntax will be claimed by Jupyter and not knitr engine.

Thanks for the report !

jkrumbiegel · 2024-06-18T09:25:11Z

one question, when you process such a script file, what does target.markdown contain in the end? A markdown transformation or the original script? Or would that not be set at all in that case, because the source is not markdown

cderv · 2024-06-18T10:20:27Z

Yes it will contain a markdown transformation. We process the script in Quarto to create a Markdown version

It happens in this function

quarto-cli/src/execute/jupyter/percent.ts

Lines 33 to 90 in 68cc8cb

    
           export function markdownFromJupyterPercentScript(file: string) { 
        
             // determine language/kernel 
        
             const ext = extname(file).toLowerCase(); 
        
             const language = ext === ".jl" ? "julia" : ext === ".r" ? "r" : "python"; 
        
             // break into cells 
        
             const cells: PercentCell[] = []; 
        
             const activeCell = () => cells[cells.length - 1]; 
        
             for (const line of lines(Deno.readTextFileSync(file).trim())) { 
        
               const header = percentCellHeader(line); 
        
               if (header) { 
        
                 cells.push({ header, lines: [] }); 
        
               } else { 
        
                 activeCell()?.lines.push(line); 
        
               } 
        
             } 
        
             // resolve markdown and raw cells 
        
             const isTripleQuote = (line: string) => !!line.match(/^"{3,}\s*$/); 
        
             const asCell = (lines: string[]) => lines.join("\n") + "\n\n"; 
        
             const stripPrefix = (line: string) => line.replace(/^#\s?/, ""); 
        
             const cellContent = (cellLines: string[]) => { 
        
               if ( 
        
                 cellLines.length > 2 && isTripleQuote(cellLines[0]) && 
        
                 isTripleQuote(cellLines[cellLines.length - 1]) 
        
               ) { 
        
                 return asCell(cellLines.slice(1, cellLines.length - 1)); 
        
               } else { 
        
                 // commented 
        
                 return asCell(cellLines.map(stripPrefix)); 
        
               } 
        
             }; 
        
             return cells.reduce((markdown, cell) => { 
        
               const cellLines = trimEmptyLines(cell.lines); 
        
               if (cell.header.type === "code") { 
        
                 if (cell.header.metadata) { 
        
                   const yamlText = asYamlText(cell.header.metadata); 
        
                   cellLines.unshift(...lines(yamlText).map((line) => `#| ${line}`)); 
        
                 } 
        
                 markdown += asCell(["```{" + language + "}", ...cellLines, "```"]); 
        
               } else if (cell.header.type === "markdown") { 
        
                 markdown += cellContent(cellLines); 
        
               } else if (cell.header.type == "raw") { 
        
                 let rawContent = cellContent(cellLines); 
        
                 const format = cell.header?.metadata?.["format"]; 
        
                 const mimeType = cell.header.metadata?.[kCellRawMimeType]; 
        
                 if (typeof (mimeType) === "string") { 
        
                   const rawBlock = mdRawOutput(mimeType, lines(rawContent)); 
        
                   rawContent = rawBlock || rawContent; 
        
                 } else if (typeof (format) === "string") { 
        
                   rawContent = mdFormatOutput(format, lines(rawContent)); 
        
                 } 
        
                 markdown += rawContent; 
        
               } 
        
               return markdown; 
        
             }, ""); 
        
           }

that we call in our jupyter engine functions producing markdown

markdownForFile function

quarto-cli/src/execute/jupyter/jupyter.ts

Lines 157 to 169 in 68cc8cb

    
           markdownForFile(file: string): Promise<MappedString> { 
        
             if (isJupyterNotebook(file)) { 
        
               const nbJSON = Deno.readTextFileSync(file); 
        
               const nb = JSON.parse(nbJSON) as JupyterNotebook; 
        
               return Promise.resolve(asMappedString(markdownFromNotebookJSON(nb))); 
        
             } else if (isJupyterPercentScript(file)) { 
        
               return Promise.resolve( 
        
                 asMappedString(markdownFromJupyterPercentScript(file)), 
        
               ); 
        
             } else { 
        
               return Promise.resolve(mappedStringFromFile(file)); 
        
             } 
        
           },

or another partitionedMarkdown

quarto-cli/src/execute/jupyter/jupyter.ts

Lines 225 to 233 in 68cc8cb

    
           partitionedMarkdown: async (file: string, format?: Format) => { 
        
             if (isJupyterNotebook(file)) { 
        
               return partitionMarkdown(await markdownFromNotebookFile(file, format)); 
        
             } else if (isJupyterPercentScript(file)) { 
        
               return partitionMarkdown(markdownFromJupyterPercentScript(file)); 
        
             } else { 
        
               return partitionMarkdown(Deno.readTextFileSync(file)); 
        
             } 
        
           },

albertomercurio · 2025-02-03T14:16:29Z

Hello, I have the same problem. But the file is not even executed. I only see a html page with all the code, without syntax highlighting and without any output.

Are there any progresses on that?

cderv · 2025-02-06T14:54:24Z

Are there any progresses on that?

If you look at the thread links just above and the sidebar on right, you should see that there is a PR in work. So progress there is.

Usually, if an issue is still open, it means it is still consider. Otherwise, it would be closed as non planned, or resolved. Regarding progress, milestones and other metadata on the issues can give you an hint on the activity.

Hope it helps clarify how issue tracking works. Thanks for your understanding.

jkrumbiegel added the bug Something isn't working label Jun 17, 2024

mcanouil added the julia label Jun 17, 2024

cscheid added this to the v1.6 milestone Jun 17, 2024

cscheid self-assigned this Jun 17, 2024

jkrumbiegel mentioned this issue Jun 17, 2024

Use quarto-provided markdown if present PumasAI/QuartoNotebookRunner.jl#154

Merged

cderv mentioned this issue Oct 1, 2024

Epic: single source of truth for document/project information #8700

Open

14 tasks

andreuvall mentioned this issue Oct 15, 2024

Information about Quarto upstream issues 10034 and 3157 PumasAI/QuartoNotebookRunner.jl#195

Closed

cscheid modified the milestones: v1.6, v1.7 Oct 30, 2024

jkrumbiegel linked a pull request Jan 7, 2025 that will close this issue

Allow to reorder engine prioritization with engines key in _quarto.yml #11807

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Script files not picked up by julia engine #10034

Script files not picked up by julia engine #10034

jkrumbiegel commented Jun 17, 2024

cderv commented Jun 17, 2024

jkrumbiegel commented Jun 17, 2024

cderv commented Jun 18, 2024

jkrumbiegel commented Jun 18, 2024 •

edited

Loading

cderv commented Jun 18, 2024

albertomercurio commented Feb 3, 2025

cderv commented Feb 6, 2025

Script files not picked up by julia engine #10034

Script files not picked up by julia engine #10034

Comments

jkrumbiegel commented Jun 17, 2024

Bug description

Steps to reproduce

Expected behavior

Actual behavior

Your environment

Quarto check output

cderv commented Jun 17, 2024

jkrumbiegel commented Jun 17, 2024

cderv commented Jun 18, 2024

jkrumbiegel commented Jun 18, 2024 • edited Loading

cderv commented Jun 18, 2024

albertomercurio commented Feb 3, 2025

cderv commented Feb 6, 2025

jkrumbiegel commented Jun 18, 2024 •

edited

Loading