Add compile_and_run convenience function

[tkittel]

As we discussed, it would be great to be essentially able to parse, compile, and run a sample .instr file in a few lines of code (i.e. in a notebook as part of a tutorial). The function compile_and_run that I was kindly pointed at in https://github.com/McStasMcXtrace/mccode-antlr/blob/main/tests/runtime/compiled.py#L85 looks like something that would be useful as a general utility function.

But possibly taking the directory as an argument, instead of the TemporaryDirectory thing (which the user could do in their calling code if needed)?

[g5t]

I guess you'd prefer a single function, but currently two functions can compile and run a parsed Instr object -- mccode_antlr.run.mccode_compile and mccode_antlr.run.mccode_run_compied.

A possible way to use the functions is in the updated version of the compile_and_run test utility you linked to before:
https://github.com/McStasMcXtrace/mccode-antlr/blob/99a5cd9b9a167badc80d92c6227cc90ed04e6324/tests/runtime/compiled.py#L85-L97

Would this work for you? If so, I can pull together a release for PyPI.

[tkittel]

For sure, that version of compile_and_run is much shorter, so for the workshop next week it should be good enough!

But thinking aloud... perhaps I would still love to be spoiled a bit more at some point in the future, i.e. have a "mcstas_run" function, which would automatically use the MCSTAS generator, and accept an "instrument" (i.e. a string, file, or an already-parsed instrument), and also accept a working directory argument (since in general the results should probably not be cleaned up before the function exits... i guess).

So Ideally I would love to have code like:

from mccode_antler.whatever import mcstas_run
myinstr="""
BLA SOME INSTRUMENT CODE HERE
"""
mcstas_run(myinstr,'./somedir')
#or:
with TemporaryDirectory() as d:
    mcstas_run(myinstr,d.joinpath('somedir'))
    #analyse output here

Does that make sense? However, there might be stuff I am not thinking about and so forth, so perhaps it is best that for the short term I simply try and use the few functions you already provide?

[g5t]

If you can come up with a sensible way to handle directories, and want to write the logic to take 3+ types of input as a positional argument, this could be a useful addition.

The directory scheme should probably match (or at least resemble) the McCode behavior, where the working directory is always the current directory, the binary ends up named after the instrument file, and the runtime output artifacts end up in a newly-created folder either specified as a command line option or named after the instrument file and current time.

[tkittel]

Sounds good to me, let us discuss it in person next time we have a chance!

In the meantime, for my workshop, I was trying to add a small utility function at the top of my google colab notebooks:

def mcstas_run( instr, rundir = None ):
    import pathlib
    import tempfile
    from mccode_antlr.translators.target import MCSTAS_GENERATOR as GEN 
    from mccode_antlr.run import mccode_compile, mccode_run_compiled 
    rundir = pathlib.Path( tempfile.mkdtemp('mcstas_rundir_')
                           if rundir is None else rundir )
    if not rundir.is_dir():
        raise FileNotFoundError("Directory not found: %s"%rundir_base)
    binary, target = mccode_compile(instr, rundir, generator=GEN)
    mccode_run_compiled(binary, target, rundir)
    return rundir

However, I didn't get to really test it yet since I get the error:

Downloading antlr4-4.13.2-complete.jar
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-4-372250295857> in <cell line: 1>()
----> 1 outdir = mcstas_run(my_instr)

<ipython-input-2-40ce4132f61d> in mcstas_run(instr, rundir)
      3     import tempfile
      4     from mccode_antlr.translators.target import MCSTAS_GENERATOR as GEN
----> 5     from mccode_antlr.run import mccode_compile, mccode_run_compiled
      6     rundir = pathlib.Path( tempfile.mkdtemp('mcstas_rundir_')
      7                            if rundir is None else rundir )

ModuleNotFoundError: No module named 'mccode_antlr.run'

Does that mean that I have to wait for you to make a new PyPI package first?

[tkittel]

Actually, I guess I can just pip install straight from your repo to try it out.

[tkittel]

Ok, this seemed to work:

%pip install git+https://github.com/McStasMcXtrace/mccode-antlr@issue-54-run

I will try it out now.

[tkittel]

Ok, so this went better. I had issues with the SHELL statement introduced in McStas 3.3, but without that and the updated function here I was able to proceed a bit further:

def mcstas_run( instr, rundir = None ):
    if hasattr(instr,'__fspath__') or not hasattr(instr,'name'):
        from mccode_antlr.loader import parse_mcstas_instr
        instr = parse_mcstas_instr(instr)
    import pathlib
    import tempfile
    from mccode_antlr.translators.target import MCSTAS_GENERATOR as GEN 
    from mccode_antlr.run import mccode_compile, mccode_run_compiled 
    rundir = pathlib.Path( tempfile.mkdtemp('mcstas_rundir_')
                           if rundir is None else rundir )
    if not rundir.is_dir():
        raise FileNotFoundError("Directory not found: %s"%rundir_base)
    binary, target = mccode_compile(instr, rundir, generator=GEN)
    mccode_run_compiled(binary, target, rundir)
    return rundir

Next problem is that it seems to hang after the following output. Do I need to somehow pass in a different number of sr neutrons? Actually, how DO I pass in parameters? Do you have an example?

...
Downloading file 'common/lib/share/metadata-r.c' from 'https://github.com/McStasMcXtrace/McCode/raw/main/common/lib/share/metadata-r.c' to '/root/.cache/libc'.
No initialization present?

-----------------------------------------------------------

Generating single GPU kernel or single CPU section layout:

-----------------------------------------------------------

Generating GPU/CPU -DFUNNEL layout:
-> GPU kernel from component myMat_ncrystal_proc
-> GPU kernel from component myMat
-> GPU kernel from component init
-> GPU kernel from component origin
-> GPU kernel from component source
-> GPU kernel from component powder_sample
Component master2 is NOACC, CPUONLY=False
->FUNNEL mode enabled, SPLIT within buffer.
-> GPU kernel from component master2
-> GPU kernel from component powder_pattern_detc
-> GPU kernel from component stop

-----------------------------------------------------------
Downloading file 'common/lib/share/mccode_main.c' from 'https://github.com/McStasMcXtrace/McCode/raw/main/common/lib/share/mccode_main.c' to '/root/.cache/libc'.
2024-08-28 08:34:39.551 | INFO     | mccode_antlr.compiler.c:compile_instrument:137 - Source written in example2.c

[tkittel]

For reference, this was my instrument file (actually, in a string):

DEFINE INSTRUMENT example2()
TRACE

/*
   The following code was auto generated by NCrystal v3.9.4 via Python:

     NCrystal.mcstasutils.cfgstr_2_union_instrument_code(
         cfgstr = 'phases<0.01*void.ncmat&0.99*Al_sg225.ncmat;temp=200K>',
         name = 'myMat' )

   Please rerun in case of major changes to input data or NCrystal.
*/

COMPONENT myMat_ncrystal_proc = NCrystal_process(
    cfg = "phases<0.01*void.ncmat&0.99*Al_sg225.ncmat;temp=200>" )
AT (0,0,0) ABSOLUTE

COMPONENT myMat = Union_make_material(
    process_string = "myMat_ncrystal_proc",
    my_absorption = 1.37745435679474 )
AT (0,0,0) ABSOLUTE

/* End of auto generated code from NCrystal v3.9.4. */

COMPONENT init = Union_init()
AT (0,0,0) ABSOLUTE
COMPONENT origin = Progress_bar()
  AT (0, 0, 0) RELATIVE ABSOLUTE
COMPONENT source =   Source_div(lambda0=1.539739, dlambda=0.01, xwidth=0.001, yheight=0.001, focus_aw=1, focus_ah=1)
  AT (0, 0, 0.3) RELATIVE origin
COMPONENT powder_sample = Union_cylinder(yheight=0.01, radius=0.01, priority=1, material_string="myMat")
AT (0, 0, 1) RELATIVE origin
COMPONENT master2 = Union_master()
AT (0, 0, 0) RELATIVE powder_sample
COMPONENT powder_pattern_detc = Monitor_nD(
    options = "banana, angle limits=[10 170], bins=500",
    radius = 0.05, yheight = 0.1)
  AT (0, 0, 0) RELATIVE powder_sample
COMPONENT stop = Union_stop()
AT (0,0,0) ABSOLUTE
END

[g5t]

It's good that installing from the git branch of the repository works 😆

---

Actually, how DO I pass in parameters? Do you have an example?

The binary runtime input parameters should be specified as a string, as in the test helper function

mccode_run_compiled(binary, target, Path(directory).joinpath('t'), parameters)

I'm a bit surprised that your call to mccode_run_compiled with only three positional inputs worked at all; I would have expected a TypeError for the missing parameter.

At the moment it's up to you to know what parameters your binary will accept, and the standard McCode compiled instrument parameters are of course allowed.

Note

You probably don't want to specify --dir {directory} in the parameters string, since that is added for you from the third positional input of the call to mccode_run_compiled

Warning

Your snippet will raise an error because you're passing the working directory rundir to mccode_run_compiled and McStas refuses to output simulation result artifacts into an existing directory.

[g5t]

@tkittel You should now be able to install mccode-antlr v0.7.0 directly via pip to get the new functionality.

[tkittel]

Great! I will most likely have to continue my attempts tomorrow.

But when you say that McStas refuses something, isn't that in your code now? Or is it in the c-files?

[tkittel]

So I still can't get this to run. It seems to hang during compilation (or just takes longer than my attention span). For reference, classic mcrun parses, compiles, and runs in <1sec.

Here is a standalone script showing my problem:

#!/usr/bin/env python3

#Needs "pip install mccode_antlr ncrystal"

_test_instr="""
DEFINE INSTRUMENT example2()
TRACE

COMPONENT origin = Progress_bar()
AT (0, 0, 0) RELATIVE ABSOLUTE

COMPONENT source =   Source_div(lambda0=1.539739, dlambda=0.01, xwidth=0.001, yheight=0.001, focus_aw=1, focus_ah=1)
AT (0, 0, 0.3) RELATIVE origin

COMPONENT mysample = NCrystal_sample(cfg="Al2O3_sg167_Corundum.ncmat",yheight=0.01,radius=0.01)
AT (0, 0, 0) RELATIVE PREVIOUS

COMPONENT powder_pattern_detc = Monitor_nD(
    options = "banana, angle limits=[10 170], bins=500",
    radius = 0.05, yheight = 0.1)
AT (0, 0, 0) RELATIVE PREVIOUS

END
"""

def mcstas_run( instr, rundir = None ):
    if hasattr(instr,'__fspath__') or not hasattr(instr,'name'):
        from mccode_antlr.loader import parse_mcstas_instr
        instr = parse_mcstas_instr(instr)
    import pathlib
    import tempfile
    from mccode_antlr.translators.target import MCSTAS_GENERATOR as GEN 
    from mccode_antlr.run import mccode_compile, mccode_run_compiled 
    rundir = pathlib.Path( tempfile.mkdtemp('mcstas_rundir_')
                           if rundir is None else rundir )
    if not rundir.is_dir():
        raise FileNotFoundError("Directory not found: %s"%rundir)
    for i in range(1,1000000):
        #Quick and dirty solution, to be changed:
        actual_rd = rundir / f'mcstasrun_{i}'
        if not actual_rd.exists():
            break
    binary, target = mccode_compile(instr, rundir, generator=GEN)
    mccode_run_compiled(binary, target, actual_rd, parameters='-n 1000')
    return actual_rd

def main():
    mcstas_run( _test_instr, './lala' )

if __name__=='__main__':
    main()

[tkittel]

Hmm... investigating a bit further, it seems to only hang when NCrystal_sample is present.

Could it be the FUNNEL (non-gpu) mode?

[g5t]

But when you say that McStas refuses something, isn't that in your code now? Or is it in the c-files?

The runtime refuses, see

https://github.com/McStasMcXtrace/McCode/blob/584eb8d56e493b29c856a4bfb3273de181f4cc77/common/lib/share/mccode-r.c#L2393-L2396

    if(mkdir(dirname, 0777)) {
#ifndef DANSE
      fprintf(stderr, "Error: unable to create directory '%s' (mcuse_dir)\n", dir);
      fprintf(stderr, "(Maybe the directory already exists?)\n");

[g5t]

I haven't used NCrystal_sample -- does it do any @include or %include-ing (or whatever the special McCode include keyword is)?