New reducing and merging tools #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

mcremone wants to merge 1,684 commits into LPC-DM:master from mcremone:master

.gitignore

-Original file line number
+Diff line change
@@ Expand Up / @@ -8,5 +8,10 @@ __pycache__ @@
     *.nfs0000003717774a2d000002c7
     docker_stderror
     *.futures
+    *.reduced
     *.merged
     *.scaled
+    *.model
+    *.processor
+    datacards*
+    results

README.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,5 +1,6 @@
  
    # Decaf

    <img src="https://user-images.githubusercontent.com/10731328/193421563-cf992d8b-8e5e-4530-9179-7dbd507d2e02.png" width="350"/>

    # **D**ark matter **E**xperience with the **C**offea **A**nalysis **F**ramework

    Following instructions are to run the generation of the histograms directly from NanoAOD. 

    ## Initial Setup

    @@ -94,7 +95,7 @@ python processors/darkhiggs.py --year 2018
  
    The module will generate a .coffea file, stored in the ```processors``` folder, that contains the processor instance to be used in the following steps. The ```--year``` option will allow for the generation of the processor corresponding to a specific year.

    ### Running with Phyton futures

    ### Running with Python futures

    Phyton futures allows for running on multiple processor on a single node. To run the local histogram generation:

    @@ -114,6 +115,7 @@ python submit_condor.py --year 2018 --processor darkhiggs2018 -t
  
    This way jobs to generate the full set of 2018 histograms will be submitted to condor. the ```-t``` will allow for tarring the working environment and the necessary dependences to run on condor nodes. The module has a ```--dataset``` option that works like described before for ```run.py```. Will allow you to run on a single batch, dataset, or batches/datasets that match the input string.

    <!-- COMMENT OUT

    ## Running with Spark

    To run with Spark, first copy your certificate to your home are on the fermicloud118 instance.

    @@ -200,3 +202,216 @@ The script will print a link inside start_jupyter.log. Copy-paste it on your bro
  
    ```

    sh stop_jupyter.sh

    ```

    -->

    ## From Coffea Histograms to Fits

    ### Generating the model

    Taking as example the dark Higgs analysis, run the following command to generate the background model, for example:

    ```

    python models/darkhiggs.py -y 2018 -f -m 40to120

    ```

    The ```models/darkhiggs.py``` module extracts coffea histgrams from the ```hists/darkhiggs201?.scaled``` files and utilize them to generate the different templates that will later be rendered into datacards/workspaces. It also defines transfer factors for the data-driven background models. It produces ```.model``` files that are saved into the ```data``` folder. More specifically, the ```models/darkhiggs.py``` module produces one ```.model``` file for each pass/fail and recoil bin.

    Running the following command will generate all model files, and the outputs will store in the data directory:

    ```

    ./make_model.sh

    ```

    ### Rendering the model

    In order for the rendering step to be completed successfully and for the resulting datacards and workspaces to be used in later steps, both the python version and the ROOT version should correspond to the ones that are going to be set when running ```combine```. It is therefore suggested to clone the ```decaf``` repository inside the ```src``` folder of ```CMSSSW``` release that will be used to perform the combine fit and run the rendering from there, without running the ```env_lcg.sh```. You should, instead, do the regular ```cmsenv``` command.

    At the time of writing, the recommended version is ```CMSSW_10_2_13``` (https://cms-analysis.github.io/HiggsAnalysis-CombinedLimit/#cc7-release-cmssw_10_2_x-recommended-version). This version uses python 2.7 and ROOT 6.12.

    In addition, you should install the following packages (to be done *after* ```cmsenv```):

    ```

    pip install --user flake8

    pip install --user cloudpickle

    pip install --user https://github.com/nsmith-/rhalphalib/archive/master.zip

    ```

    Finally, since we are using Python2.7, you should edit the file

    ```

    $HOME/.local/lib/python2.7/site-packages/rhalphalib/model.py

    ```

    and change line 94 from

    ```

    obsmap.insert(ROOT.std.pair('const string, RooDataHist*')(channel.name, obs))

    ```

    to

    ```

    obsmap.insert(ROOT.std.pair('const string, RooDataHist*')(channel.name.encode("ascii"), obs))

    ```

    and also make the following modification:

    ```

    import ROOT

    #add this line

    ROOT.v5.TFormula.SetMaxima(5000000)

    ```

    The models saved in the ```.model``` files at the previous step can be rendered into datacards by running the following command:

    ```

    python render.py -m model_name

    ```

    the ```-m``` or ```--model``` options provide in input the name of the model to render, that corresponds to the name of the ```.model``` file where the model is stored. The ```render.py``` module launches python futures jobs to process in parallel the different control/signal regions that belongs to a single model. Different models can also be rendered in parallel, by using condor. In order to run rendering condor job, the following command should be ran:

    ```

    python render_condor.py -m <model_name> -c <server_name> -t -x

    python render_condor.py -m darkhiggs -c kisti -t -x

    ```

    this time, all the models stored into ```.model``` files whose name contains the sting passed via the  ```-m``` options are going to be rendered. 

    The ```render.py``` module saves datacards and workspaces into folders that show the following naming:

    ```

    datacards/model_name

    ```

    The ```render_condor.py``` module returns ```.tgz``` tarballs that contain the different datacards/workspaces, and are stored into the ```datacards``` folder. To untar them, simply do:

    ```

    python macros/untar_cards.py -a darkhiggs 

    ```

    Where the ```-a``` or ```--analysis``` options correspond to the analysis name. The ```untar_cards.py``` script will untar all the tarballs that contain the string that is passed through the ```-a``` option.

    To merge the datacards, run the following script:

    ```

    python macros/combine_cards.py -a darkhiggs

    ```

    Where the ```-a``` or ```--analysis``` options correspond to the analysis name. The ```combine_datacards.py``` script will combine all the datacards whose name contains the string that is passed through the ```-a``` option. The script will create a folder inside ```datacards``` whose name corresponds to the string that is passed through the ```-a``` option, will move all the workspaces that correspond to the datacards it combined inside it, and will save in it the combined datacard, whose name will be set to the string that is passed through the ```-a``` option.

    You can also plot the inputs from a **single** workspace with the `makeWorkspacePlots.C` macro. It stacks all the inputs in a single histogram. From the folder you have just created at the previous step, do, for example:

    ```

    root -b -q ../../makeWorkspacePlots.C\('"zecr2018passrecoil4"',5000,2000\)

    ```

    where the first argument (`zecr2018passrecoil4`) is the name of the workspace; the ROOT file name must be the same, but with the `.root` extension; the second argument (`5000`) is the limit of the y-axis of the stack; the third argument (`2000`) is the y-position of an explanatory label; that label will simply be the same as the first argument again.

    ### Using Combine

    Make sure you edit your ```$CMSSW/src/HiggsAnalysis/CombinedLimit/scripts/text2workspace.py``` module as suggested below:

    ```

    import ROOT

    #add this line

    ROOT.v5.TFormula.SetMaxima(5000000)

    ```

    In the datacards directory, the directory is created based on the <model_name>. We are using the "MultiSignalModel" method to have one datacard includes all signal mass points. Please submint text2workspace job to the condor by using the following command: 

    ```

    python t2w_condor.py -a darkhiggs -c <server name> -t -x

    ```

    Currently, \<server name\> is either `lpc` or `kisti`. The output will be saved in the same directory with the same name as the datacard.

    After you get workspace, you will be able to run fit over condor by doing:

    ```

    python combine_condor.py -a darkhiggs -c <server name> -m <method> -t -x

    python combine_condor.py -a darkhiggs -c kisti -m cminfit -t -x

    ```

    <!-- COMMENT OUT

    make sure you edit your ```$CMSSW/src/HiggsAnalysis/CombinedLimit/scripts/text2workspace.py``` module as suggested below:

    ```

    import ROOT

    #add this line

    ROOT.v5.TFormula.SetMaxima(5000000)

    ```

    The result of this step will be a ```darkhiggs.root``` file to be used for the combine fit.

    where ```darkhiggs.txt``` is the name of the combined datacard generated at the previous step.

    #### Running special datacards

    Before running the combibe tools, you have to run ```ulimit -s unlimited``` command. Feel free to add it to your ```.bashrc```. Also, modify your ```$CMSSW/src/HiggsAnalysis/CombinedLimit/bin/combine.cpp```:

    ```

    using namespace boost;

    namespace po = boost::program_options;

    #add the line below

    ROOT::v5::TFormula::SetMaxima(5000000);

    ```

    Make sure you compile after making this modification. When you render the model, you can use the alternative datacards:

    ```

    analysis/models/darkhiggs_fourregions.py

    analysis/models/darkhiggs_fiveregions.py

    analysis/models/darkhiggs_eightregions.py

    ```

    that try the fit with "Signal, WMuon, TopMuon, DoubleMuon" (four);

    with all those plus "Gamma" (five);

    with all those plus "WEle, TopEle, DoubleEle" (eight).

    In all of those the minor MCs are not added at all to the fit.

    The usual `darkhiggs.py` has all eight regions + minor MCs.

    Alternative fits:

    Minuit:

    ```

    combine -M FitDiagnostics -d darkhiggs.root \

    --expectSignal 1 --forceRecreateNLL --cminDefaultMinimizerType Minuit \

    --ignoreCovWarning --saveShapes --saveWithUncertainties --saveWorkspace --plots \

    &> out_FitDiagnostics_Minuit.log

    ```

    Minuit (fast):   

    Drop the options: `--saveShapes`, `--saveWorkspace`, `--plots`   

    ```

    combine -M FitDiagnostics -d darkhiggs.root \

    --expectSignal 1 --forceRecreateNLL --cminDefaultMinimizerType Minuit \

    --ignoreCovWarning --saveWithUncertainties &> out_FitDiagnostics_Minuit.log

    ```

    Robust fit:

    ```

    combine -M FitDiagnostics -d darkhiggs.root \

    --expectSignal 1 --forceRecreateNLL  --robustFit=1 \

    --ignoreCovWarning --saveShapes --saveWithUncertainties --saveWorkspace --plots \

    &> out_FitDiagnostics_robustFit.log

    ```

    -->

    ## Macros

    ### Usage examples

    ```

    python macros/dump_templates.py -w datacards/darkhiggs/darkhiggs.root:w --observable fjmass -o plots/darkhiggs2018/model

    ```

    ```

    python macros/hessian.py -w datacards/darkhiggs/higgsCombineTest.FitDiagnostics.mH120.root:w -f datacards/darkhiggs/fitDiagnostics.root:fit_b

    ```

    #### Pulls plotting 

    Change the path to store outputs in `plotConfig.py`    

    ```

    python macros/diffNuisances.py -g pulls.root ../fitDiagnostics.root

    ```

    #### Postfit plotting - Method 1   

    If you use fast fit command, then the following command will make prefit and postfit histograms  

    ```

    PostFitShapesFromWorkspace -w path/to/darkhiggs.root -d path/to/darkhiggs.txt -f path/to/fitDiagnostics.root:fit_s --postfit --sampling --samples 300 --skip-proc-errs -o outputfile.root

    ```

    To make postfit stack plots  

    ```

    python macros/postFitShapesFromWorkSpace.py path/to/outputfile.root

    ```

    #### Postfit plotting - Method 2   

    This method uses the output from `dump_templates.py`.   

    ```

    python macros/combine_dump_outputs.py plots/darkhiggs2018/dump_postfit (or dump_prefit)

    ```

    Then you get combined root files in the `plots/darkhiggs2018/dump` directory.   

    ```

    python macros/dump_darkhiggs_postfit.py plots/darkhiggs2018/dump/outputs.root

    ```

    This will make postfit stack plots.

analysis/convert.py

-Original file line number
+Diff line change
@@ -0,0 +1,52 @@
+    #!/usr/bin/env python
+    import os
+    from optparse import OptionParser
+    if __name__ == '__main__':
+        parser = OptionParser()
+        parser.add_option('-d', '--datacard', help='datacard', dest='datacard')
+        parser.add_option('-o', '--outfile', help='outfile', dest='outfile')
+        parser.add_option('-m', '--maps', help='maps', dest='maps')
+        parser.add_option('-a', '--arguments', help='arguments', dest='arguments', default='--channel-masks')
+        (options, args) = parser.parse_args()
+        command = 'text2workspace.py '+options.datacard
+        datacard=open(options.datacard,'r')
+        process_lines=[]
+        for line in datacard.readlines():
+            if not line.startswith('process'): continue
+            process_lines.append(line.split())
+        signal_indices = [i for i in range(1, len(process_lines[1])) if int(process_lines[1][i]) <= 0]
+        signals = set([process_lines[0][i] for i in signal_indices if process_lines[0][i]])
+        def add_maps(command, options):
+            #command += ' -P HiggsAnalysis.CombinedLimit.PhysicsModel:multiSignalModel --PO verbose'
+            command += ' -P HiggsAnalysis.CombinedLimit.PhysicsModel:multiSignalModel'
+            maps= ''
+            for option in options.split('--PO '):
+                if not option: continue
+                if 'SIGNAL' in option:
+                    for signal in signals:
+                        if not all(piece in signal for piece in option.split(':')[0].split('/')[1].split('*')): continue
+                        #if option.split(':')[0].split('/')[1].replace('*','') not in signal: continue
+                        maps += ' --PO '+option.replace(option.split(':')[0].split('/')[1],signal).replace('SIGNAL',signal)
+                else:
+                    maps += ' --PO '+option
+            command += maps
+            return command
+        commands=[]
+        if options.maps:
+            if 'SIGNAL:' in options.maps:
+                for signal in signals:
+                    commands.append(add_maps(command, options.maps.replace('SIGNAL',signal))+' -o '+options.outfile.replace('SIGNAL',signal)+' '+options.arguments)
+            else:
+                commands.append(add_maps(command, options.maps)+' -o '+options.outfile+' '+options.arguments)
+        else:
+            commands.append(command+' -o '+options.outfile+' '+options.arguments)
+        for command in commands:
+            #print(command)
+            os.system(command)

analysis/convert.sh

-Original file line number
+Diff line change
@@ -0,0 +1,53 @@
+    #!/usr/bin/env bash
+    export USER=${5}
+    echo "User is: ${5}"
+    echo "Starting job on " `date` #Date/time of start of job
+    echo "Running on: `uname -a`" #Condor job is running on this node
+    echo "System software: `cat /etc/redhat-release`" #Operating System on that node
+    echo $(hostname)
+    source /cvmfs/cms.cern.ch/cmsset_default.sh
+    if [ "${4}" == "kisti" ]; then
+        voms-proxy-info -exists
+        if [ $? -eq 0 ]; then
+            echo "No need to copy"
+            ls -l /tmp/x509up_u$(id -u)
+            voms-proxy-info -all
+        else
+            cp ./x509up_u* /tmp
+            ls -l /tmp/x509up_u$(id -u)
+            voms-proxy-info -all
+        fi
+        xrdcp -s root://cms-xrdr.private.lo:2094//xrd/store/user/$USER/cmssw.tgz .
+        echo "cmssw correctly copied"
+    else
+        xrdcp -s root://cmseos.fnal.gov//store/user/$USER/cmssw.tgz .
+        echo "cmssw correctly copied"
+        xrdcp -s root://cmseos.fnal.gov//store/user/$USER/py2local.tgz .
+        echo "py2local correctly copied"
+        tar -zxvf py2local.tgz
+        rm py2local.tgz
+        export PYTHONPATH=${_CONDOR_SCRATCH_DIR}/site-packages:$PYTHONPATH
+        export PYTHONPATH=$(find ${_CONDOR_SCRATCH_DIR}/site-packages/ -name *.egg |tr '\n' ':')$PYTHONPATH
+        export PYTHONWARNINGS="ignore"
+        echo "Updated python path: " $PYTHONPATH
+    fi
+    echo "untar cmssw"
+    tar -zxvf cmssw.tgz
+    echo "cmssw untarred"
+    rm cmssw.tgz
+    export SCRAM_ARCH=slc7_amd64_gcc700
+    cd CMSSW_10_2_13/src
+    scramv1 b ProjectRename
+    eval `scramv1 runtime -sh` # cmsenv is an alias not on the workers
+    cd decaf/analysis
+    if [ "${3}" == "None"  ]; then
+        echo "python convert.py -d ${1} -o ${2}"
+        python convert.py -d ${1} -o ${2}
+    else
+        export maps=$( echo ${3} | tr '+' ' ' )
+        echo "python convert.py -d ${1} -o ${2} -m \"$maps\""
+        python convert.py -d ${1} -o ${2} -m "$maps"
+    fi
+    ls ${2}
+    cp ${2} ${_CONDOR_SCRATCH_DIR}/outfile.root

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New reducing and merging tools #62

Uh oh!

Diff view

Diff view

Uh oh!

There are no files selected for viewing

Uh oh!

New reducing and merging tools #62

Are you sure you want to change the base?

Uh oh!

New reducing and merging tools #62

Uh oh!

Diff view

Diff view

Uh oh!

There are no files selected for viewing

Uh oh!