[RelVal] Update README, add verbosity option

Author: Benedikt Volkel

RelVal/README.md

@@ -12,8 +12,8 @@ There are 2 ROOT macros which can in principle be used as-is. Their functionalit
 
 This macro allows to compare 2 ROOT files that contain `TH1` objects. Objects are considered to correspond to each other if they have the same name.
 At the moment, 3 different comparisons are implemented:
-1. `chi2`: Chi2 test of compared histograms,
-1. `bin_cont`: relative difference of normalised bin content of both histograms,
+1. `chi2`: Chi2 test of compared histograms (see also the [ROOT documentation](https://root.cern.ch/doc/master/classTH1.html#ab7d63c7c177ccbf879b5dc31f2311b27)),
+1. `kolmogorov`: shape comparison using Kolmogorov test (see also the [ROOT documentation](https://root.cern.ch/doc/master/classTH1.html#aeadcf087afe6ba203bcde124cfabbee4)),
 1. `num_entries`: relative difference in the number of entries.
 
 The first 2 tests are considered critical, hence if the threshold is exceeded, the comparison result is named `BAD`. Also the third test is considered critical in case efficiencies are compared coming from `TEfficiency` objects.
@@ -59,7 +59,7 @@ python o2dpg_release_validation.py rel-val -i <first-list-of-files> -j <second-l
 ```
 This performs all of the above mentioned tests. If only certain tests should be performed, this can be achieved with the flags `--with-test-<which-test>` where `<which-test>` is one of
 1. `chi2`,
-1. `bin-cont`,
+1. `kolmogorov`,
 1. `num-entries`.
 
 By default, all of them are switched on.
@@ -96,10 +96,10 @@ There are various plots created during the RelVal run. For each compared file th
 
 ## More details of `rel-val` command
 
-As mentioned above, the basic usage of the `rel-val` sub-command is straightforward. But there are quite a few more options available and some of them will be explained briefly below.
+As mentioned above, the basic usage of the `rel-val` sub-command is straightforward. But there are quite a few more options available and some of them will be explained briefly below. In fact, most of them also apply to the `inspect` sub-command.
 
 ### Setting new/custom thresholds from another RelVal run
-Each RelVal run produces a `Summary.json` file in the corresponding output directories. Among other things, it contains the computed values of all tests for each compared histogram pair. Such a `Summary.json` can now be used as a input file for a future RelVal to set all thresholds according to the values. In fact, multiple such files can be passed and for each historgam-test combination, the mean or max of the previously calculated values can be used to set the new thresholds.
+Each RelVal run produces a `Summary.json` file in the corresponding output directories. Among other things, it contains the computed values of all tests for each compared histogram pair. Such a `Summary.json` can now be used as a input file for a future RelVal to set all thresholds according to the values. In fact, multiple such files can be passed and for each histogram-test combination, the mean or max of the previously calculated values can be used to set the new thresholds.
 
 ```bash
 python ${O2DPG_ROOT}/RelVal/o2dpg_release_validation.py rel-val -i <first-list-of-files> -j <second-list-of-files> --use-values-as-thresholds <list-of-summaries> [--combine-thresholds {mean,max}] [--test-<name>-threshold-margin <value>]
@@ -114,23 +114,5 @@ There is an ongoing effort to unify the names of QC objects inside MC and data Q
 
 MC QC objects are usually distributed over multiple files while those from data are all contained in one single file. It is possible to directly compare them with
 ```bash
-python ${O2DPG_ROOT}/RelVal/o2dpg_release_validation.py rel-val -i ${MC_PRODUCTION}/QC/*.root -j ${DATA_PRODUCTION}/QC.root [--inlcude-dirs <include-directories]
+python ${O2DPG_ROOT}/RelVal/o2dpg_release_validation.py rel-val -i ${MC_PRODUCTION}/QC/*.root -j ${DATA_PRODUCTION}/QC.root [--include-dirs <include-directories>]
 ```
-
-### Apply to entire simulation outcome
-
-**This is still under development and does not yet work when e.g. comparing an MC directory to a data directory.**
-
-In addition to simply comparing 2 ROOT files, the script offers the possibility of comparing 2 corresponding directories that contain simulation artifacts (and potentially QC and analysis results). It is not foreseen to run over everything inside those directories but the files must be specifiec via a small config file. See [this example](config/rel_val_sim_dirs_default.json). It is passed via the option `--dirs-config`. In addition, top-level keys can be enabled(disabled) with `--dirs-config-enable <keys>`(`dirs-config-disable <keys>`) where disabling takes precedence.
-
-**NOTE** That each single one of the comparisons is only done if mutual files were found in the 2 corresponding directories. As an example, one could do
-```bash
-cd ${DIR1}
-python o2dpg_workflow_runner.py -f <workflow-json1>
-cd ${DIR2}
-# potentially something has changed in the software or the simulation/reconstruction parameters
-python o2dpg_workflow_runner.py -f <workflow-json2>
-python ${O2DPG_ROOT}/RelVal/o2dpg_release_validation.py rel-val -i ${DIR1} -j ${DIR2} --dirs-config ${O2DPG_ROOT}/RelVal/config/rel_val_sim_dirs_default.json --dirs-config-enable QC [-o <output/dir>] [<test-flags>]
-```
-This would run the RelVal von everything specified under the top key `QC`.
-

RelVal/o2dpg_release_validation.py

@@ -106,7 +106,7 @@
 ROOT_MACRO_EXTRACT=join(O2DPG_ROOT, "RelVal", "ExtractAndFlatten.C")
 ROOT_MACRO_RELVAL=join(O2DPG_ROOT, "RelVal", "ReleaseValidation.C")
 
-from ROOT import TFile, gDirectory, gROOT, TChain, TH1D
+from ROOT import gROOT
 
 DETECTORS_OF_INTEREST_HITS = ["ITS", "TOF", "EMC", "TRD", "PHS", "FT0", "HMP", "MFT", "FDD", "FV0", "MCH", "MID", "CPV", "ZDC", "TPC"]
 
@@ -300,7 +300,7 @@ def plot_pie_charts(summary, out_dir, title, include_patterns=None, exclude_patt
             continue
         # loop over tests done
         for test in tests:
-            test_name = test["test_name"];
+            test_name = test["test_name"]
             if test_name not in test_n_hist_map:
                 test_n_hist_map[test_name] = {}
             result = test["result"]
@@ -340,7 +340,7 @@ def extract_from_summary(summary, fields, include_patterns=None, exclude_pattern
             continue
         # loop over tests done
         for test in tests:
-            test_name = test["test_name"];
+            test_name = test["test_name"]
             if test_name not in test_histo_value_map:
                 test_histo_value_map[test_name] = {field: [] for field in fields}
                 test_histo_value_map[test_name]["histograms"] = []
@@ -728,7 +728,7 @@ def map_histos_to_severity(summary, include_patterns=None, exclude_patterns=None
     return test_n_hist_map
 
 
-def print_summary(summary, include_patterns=None, exclude_patterns=None):
+def print_summary(summary, include_patterns=None, exclude_patterns=None, long=False):
     """
     Check if any 2 histograms have a given severity level after RelVal
     """
@@ -739,6 +739,9 @@ def print_summary(summary, include_patterns=None, exclude_patterns=None):
     print(f"\n#####\nNumber of compared histograms: {n_all}\nBased on critical tests, severities are\n")
     for sev, histos in test_n_hist_map.items():
         print(f"  {sev}: {len(histos)}")
+        if long:
+            for i, h in enumerate(histos, start=1):
+                print(f"    {i}. {h}")
     print("#####\n")
 
 
@@ -854,7 +857,7 @@ def rel_val(args):
     func(args)
     global_summary, meta_info = make_global_summary(args.output)
     write_single_summary(global_summary, meta_info, join(args.output, "SummaryGlobal.json"))
-    print_summary(global_summary)
+    print_summary(global_summary, long=args.long)
     return 0
 
 
@@ -907,7 +910,7 @@ def inspect(args):
     current_summary, meta_info = read_single_summary(path)
     summary = make_single_summary(current_summary, args, output_dir, include_patterns, exclude_patterns, flags, flags_summary)
     write_single_summary(summary, meta_info, join(output_dir, "Summary.json"))
-    print_summary(summary, include_patterns)
+    print_summary(summary, include_patterns, long=args.long)
 
     if args.plot:
         plot_pie_charts(summary, output_dir, "", include_patterns, exclude_patterns)
@@ -1095,17 +1098,20 @@ def main():
     common_flags_parser.add_argument("--flags", nargs="*", help="extract all objects which have at least one test with this severity flag", choices=list(REL_VAL_SEVERITY_MAP.keys()))
     common_flags_parser.add_argument("--flags-summary", dest="flags_summary", nargs="*", help="extract all objects which have this severity flag as overall test result", choices=list(REL_VAL_SEVERITY_MAP.keys()))
 
+    common_verbosity_parser = argparse.ArgumentParser(add_help=False)
+    common_verbosity_parser.add_argument("--long", action="store_true", help="enhance verbosity")
+
     sub_parsers = parser.add_subparsers(dest="command")
-    rel_val_parser = sub_parsers.add_parser("rel-val", parents=[common_file_parser, common_threshold_parser])
+    rel_val_parser = sub_parsers.add_parser("rel-val", parents=[common_file_parser, common_threshold_parser, common_verbosity_parser])
     rel_val_parser.add_argument("--dir-config", dest="dir_config", help="What to take into account in a given directory")
     rel_val_parser.add_argument("--dir-config-enable", dest="dir_config_enable", nargs="*", help="only enable these top keys in your dir-config")
     rel_val_parser.add_argument("--dir-config-disable", dest="dir_config_disable", nargs="*", help="disable these top keys in your dir-config (precedence over dir-config-enable)")
-    rel_val_parser.add_argument("--include-dirs", dest="include_dirs", nargs="*", help="only inlcude directories; note that each pattern is assumed to start in the top-directory (at the moment no regex or *)")
+    rel_val_parser.add_argument("--include-dirs", dest="include_dirs", nargs="*", help="only include directories; note that each pattern is assumed to start in the top-directory (at the moment no regex or *)")
     rel_val_parser.add_argument("--add", action="store_true", help="If given and there is already a RelVal in the output directory, extracted objects will be added to the existing ones")
     rel_val_parser.add_argument("--output", "-o", help="output directory", default="rel_val")
     rel_val_parser.set_defaults(func=rel_val)
 
-    inspect_parser = sub_parsers.add_parser("inspect", parents=[common_threshold_parser, common_pattern_parser, common_flags_parser])
+    inspect_parser = sub_parsers.add_parser("inspect", parents=[common_threshold_parser, common_pattern_parser, common_flags_parser, common_verbosity_parser])
     inspect_parser.add_argument("path", help="either complete file path to a Summary.json or SummaryGlobal.json or directory where one of the former is expected to be")
     inspect_parser.add_argument("--plot", action="store_true", help="Plot the summary grid")
     inspect_parser.add_argument("--output", "-o", help="output directory, by default points to directory where the Summary.json was found")