Extending DPL input handling and message caching related to [O2 1325] #7110
Merged
Conversation
ktf
reviewed
Sep 24, 2021
matthiasrichter
force-pushed
the
dev-dpl-message-model
branch
from
67a09fe to
3abec09
Compare
Collaborator
Author
|
I have changed from EDIT Nov 22: adding the benchmark result for the final version Compared to the original implementation The first draft of this PR with the message vector inside Also the |
matthiasrichter
force-pushed
the
dev-dpl-message-model
branch
3 times, most recently
from
3db9672 to
69d805b
Compare
Collaborator
Author
|
@ktf I need to find out whats wrong with the unit test in the macos build, lets see if this is a problem also on the other CIs, then need to find out why its working on my Ubuntu installation. But in general the code in |
matthiasrichter
force-pushed
the
dev-dpl-message-model
branch
3 times, most recently
from
1123ad1 to
db30112
Compare
matthiasrichter
force-pushed
the
dev-dpl-message-model
branch
4 times, most recently
from
8b3d73f to
be8db00
Compare
matthiasrichter
force-pushed
the
dev-dpl-message-model
branch
from
be8db00 to
e284c1b
Compare
Collaborator
|
Error while checking build/O2/o2-dataflow-cs8 for e284c1bcdfde4f60b9bfdb8a1e7d0866ada88363 at 2021-11-03 09:43: Full log here. |
Collaborator
|
Error while checking build/O2/o2-cs8 for e284c1bcdfde4f60b9bfdb8a1e7d0866ada88363 at 2021-11-03 04:50: Full log here. |
matthiasrichter
force-pushed
the
dev-dpl-message-model
branch
from
e284c1b to
ae5ae6b
Compare
matthiasrichter
force-pushed
the
dev-dpl-message-model
branch
2 times, most recently
from
3231b1c to
26a0daf
Compare
Collaborator
Author
This was referenced Nov 19, 2021
ktf
reviewed
Nov 19, 2021
matthiasrichter
force-pushed
the
dev-dpl-message-model
branch
from
26a0daf to
62dd757
Compare
matthiasrichter
force-pushed
the
dev-dpl-message-model
branch
from
62dd757 to
6b92040
Compare
This is supporting an upcoming extension of the O2 data model to allow not only header-payload pairs but also sequences of payloads preceded by a single header message. Such a sequence can be indicated by setting splitPayloadIndex == splitpayloadParts with splitpayloadParts > 1. A change in the version of DataHeader will be discussed, although there is no change in the members of the header. There is no change on the InputRecord API, only the payload size of the current payload is provided in the DataRef object. It needs to be discussed, which payload size is going to be provided in the data header in case of a sequence with multiple payloads. Changes alongside (most of them meanwhile moved to separate commits): - make MessageSet the owner of FairMQMessage objects in a linear vector, enhances performance also for split payload sequences in header-paiload format - generalizing DataRelayer to relay array of messages - storing the payload message size in the DataRef Extending and cleaning up DataRelayer test and benchmark - allocating header stack through transport allocator and avoiding copy - adding test for split parts in the header-payload scheme - adding test for split parts in payloads sequence scheme During the development a performance regression has been observed when using o2::pmr::getMessage, needs to be investigated.
Using e.g. RawDeviceService const& in the init callback leads to an exception which is difficult to understand and it comes from the adaptStateful converters (_ZNK2o29framework15ServiceRegistry3getEjmNS0_11ServiceKindEPKc+0x2b9)[0x56376fa3f8b1]
matthiasrichter
force-pushed
the
dev-dpl-message-model
branch
from
6b92040 to
97ea087
Compare
Member
|
The error is: could it be we have too many open files? It looks weird. Anyways, it's most likely unrelated, so merging this. |
Member
|
Regarding the issue with const / non const, the registry is separate between the two, so that we can avoid that people ask for const services as non-const, however we could indeed extend things so that the opposite is possible. |
ezradlesser
pushed a commit
to ezradlesser/AliceO2
that referenced
this pull request
Dec 2, 2021
…AliceO2Group#7110) This is supporting an upcoming extension of the O2 data model to allow not only header-payload pairs but also sequences of payloads preceded by a single header message. Such a sequence can be indicated by setting splitPayloadIndex == splitpayloadParts with splitpayloadParts > 1. A change in the version of DataHeader will be discussed, although there is no change in the members of the header. There is no change on the InputRecord API, only the payload size of the current payload is provided in the DataRef object. It needs to be discussed, which payload size is going to be provided in the data header in case of a sequence with multiple payloads. Changes alongside (most of them meanwhile moved to separate commits): - make MessageSet the owner of FairMQMessage objects in a linear vector, enhances performance also for split payload sequences in header-paiload format - generalizing DataRelayer to relay array of messages - storing the payload message size in the DataRef Extending and cleaning up DataRelayer test and benchmark - allocating header stack through transport allocator and avoiding copy - adding test for split parts in the header-payload scheme - adding test for split parts in payloads sequence scheme During the development a performance regression has been observed when using o2::pmr::getMessage, needs to be investigated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a prototype which also will allow with a few more changes to activate support for an updated O2 message model where split payloads can be transferred in a sequence with one preceding data header.
Has to be finally activated in
DataProcessingDevicetogether with a workflow test.For using this in the workflows together with Data Distribution, also the raw proxy needs to be extended, and for that a better encapsulation of the message model is in development based on the outcome of the prototype test.