Skip to content

aod-merger: add option to skip non-existing files#298

Merged
jgrosseo merged 3 commits into
AliceO2Group:masterfrom
mario-krueger:master
Nov 18, 2021
Merged

aod-merger: add option to skip non-existing files#298
jgrosseo merged 3 commits into
AliceO2Group:masterfrom
mario-krueger:master

Conversation

@mario-krueger

Copy link
Copy Markdown
Member

avoids crashes if the inputs.txt list for some reason contains files that dont exist or are not reachable

@jgrosseo jgrosseo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,
I am against this change. We can issue a fatal error in that case but we should not skip the file.
If the list cannot be processed an error has to be produced.

@mario-krueger

Copy link
Copy Markdown
Member Author

Hi,
currently it will break with a seg fault and TFile::Open will issue an error message which probably is enough to see what is the problem. I can close the PR again if skipping the file is not what we want.
However, I was wondering if in production we would want one single corrupt file to break the whole merging chain. Maybe one could log the file names that did not work and print / save this somehow as a feedback to the operators instead?

@jgrosseo

Copy link
Copy Markdown
Contributor

On the Grid, we do staged merging for analysis, so indeed you can have a failure on a certain level but this only involves a few files. The risk with skipping it that you never find those errors.

However, I see the use case in the merger for local use. So we could add that in case one passes a parameter --skip-not-existing that it goes to the next file as you suggested. What do you think?

@mario-krueger

Copy link
Copy Markdown
Member Author

Hi,
sounds like a good compromise.
For the case where we want to abort the merging, the program now ends gracefully without a seg fault and propagates that there was an error to the caller, so merging scripts can react accordingly.

@mario-krueger mario-krueger reopened this Nov 18, 2021
@mario-krueger mario-krueger changed the title aod-merger: skip reading if input file does not exist aod-merger: add option to skip non-existing files Nov 18, 2021

@jgrosseo jgrosseo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Here a few comments

Comment thread Common/Core/aodMerger.cxx Outdated
Comment thread Common/Core/aodMerger.cxx Outdated
Comment thread Common/Core/aodMerger.cxx
@jgrosseo jgrosseo enabled auto-merge (squash) November 18, 2021 12:39
@jgrosseo jgrosseo merged commit fb4a2e6 into AliceO2Group:master Nov 18, 2021
njacazio pushed a commit that referenced this pull request May 20, 2024
* PWGLF: findable <-> analysis connection

* Please consider the following formatting changes (#297)

* Latest version with conceptual fixes

* Extra work

* Findable latest dev

* Fix photon pair conversion process check

* Please consider the following formatting changes (#298)

* Megalinter fixes

* Megalinter happiness

---------

Co-authored-by: ALICE Builder <alibuild@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants