DPL: default to transport=shmem on the same host

[ktf]

No description provided.

[ktf]

@davidrohr

[davidrohr]

@ktf : Please wait a bit merging this. I am seeing some issues in the TPC workflow, when I use this in multiple places. Is there perhaps a limit on the total size of shared memory buffers I can use?

[ktf]

I think the default by FairMQ is 2GB.

[ktf]

you can check in the device log, actually.

[davidrohr]

I assume you mean this: shm-segment-size = 2000000000 which indeed defaults to 2 GB.

I am certainly exceeding the 2GB in my test, but I would have expected some error message during allocation. Instead it crashes. I'll check why and where and then report back.

Another note: I tried to increase the size using the --shm-segment-size 4000000000 option. I assume this defines a global shared memory for the whole workflow, or is it per processor?
If I have a merged workflow, for which binary do I set the size?

[davidrohr]

@ktf @matthiasrichter @rbx @shahor02 : I tried with all the latest versions + this PR to test with shared memory, and I see 2 problems:

1. When I call DataAllocator::make<...> and there is insufficient shared memory available, it goes stuck in an endless loop and will never return. Backtrace below. I don't think we want such a bahavior. If there is insufficient memory available, it should throw an error. At least on the laptop for tests.

2. The setting of the shared memory size with --shm-segment-size 4000000000 is weird for merged workflows. It seems to work correctly when I add the option to each workflow binary. But adding it only to one of the workflow binaries, e.g. the last ones, it works sometimes, sometimes it doesn't. Perhaps it is then kind of a race condition what is the size of the shared segment?

I think these 2 issues should be addressed before this PR is merged.

Stack trace when stuck in allocation:

#0  0x00007f7c995d8569 in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f7c96bbbd79 in std::this_thread::sleep_for<long, std::ratio<1l, 1000l> > (__rtime=...) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/thread:378
#2  fair::mq::shmem::Message::InitializeChunk (this=0x55c29ed9ff80, size=2147483648) at /home/qon/alice/sw/SOURCES/FairMQ/v1.4.16/v1.4.16/fairmq/shmem/Message.cxx:114
#3  0x00007f7c96cb1679 in fair::mq::shmem::Message::Message (this=0x55c29ed9ff80, manager=..., size=<optimized out>, factory=<optimized out>) at /home/qon/alice/sw/SOURCES/FairMQ/v1.4.16/v1.4.16/fairmq/shmem/Message.cxx:53
#4  0x00007f7c96cc1677 in fair::mq::tools::make_unique<fair::mq::shmem::Message, fair::mq::shmem::Manager&, unsigned long const&, fair::mq::shmem::TransportFactory*> () at /home/qon/alice/sw/SOURCES/FairMQ/v1.4.16/v1.4.16/fairmq/tools/CppSTL.h:25
#5  fair::mq::shmem::TransportFactory::CreateMessage (this=0x55c29cb12b30, size=2147483648) at /home/qon/alice/sw/SOURCES/FairMQ/v1.4.16/v1.4.16/fairmq/shmem/TransportFactory.cxx:128
#6  0x00007f7c96cf9f08 in fair::mq::ChannelResource::do_allocate (this=0x55c29cb12b58, bytes=<optimized out>) at /home/qon/alice/sw/SOURCES/FairMQ/v1.4.16/v1.4.16/fairmq/MemoryResources.cxx:20
#7  0x00007f7c9c6a5751 in boost::container::pmr::memory_resource::allocate (alignment=1, bytes=2147483648, this=<optimized out>) at /usr/include/boost/container/pmr/memory_resource.hpp:44
#8  boost::container::pmr::polymorphic_allocator<char>::allocate (this=<optimized out>, n=2147483648) at /usr/include/boost/container/pmr/polymorphic_allocator.hpp:82
#9  std::allocator_traits<boost::container::pmr::polymorphic_allocator<char> >::allocate (__n=2147483648, __a=...) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/alloc_traits.h:306
#10 std::_Vector_base<char, boost::container::pmr::polymorphic_allocator<char> >::_M_allocate (__n=2147483648, this=<optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/stl_vector.h:343
#11 std::_Vector_base<char, boost::container::pmr::polymorphic_allocator<char> >::_M_create_storage (__n=2147483648, this=<optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/stl_vector.h:358
#12 std::_Vector_base<char, boost::container::pmr::polymorphic_allocator<char> >::_Vector_base (__a=..., __n=2147483648, this=<optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/stl_vector.h:302
#13 std::vector<char, boost::container::pmr::polymorphic_allocator<char> >::vector (__a=..., __n=2147483648, this=<optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/stl_vector.h:508
#14 o2::framework::MessageContext::ContainerRefObject<std::vector<char, boost::container::pmr::polymorphic_allocator<char> > >::ContainerRefObject<o2::framework::MessageContext, unsigned long> (index=0, bindingChannel="from_tpc-tracker_to_tpc-entropy-encoder", headerMsg=..., context=0x55c29c65d020, this=0x55c29ed9fe70) at /home/qon/alice/sw/SOURCES/O2/v1.2.0/0/Framework/Core/include/Framework/MessageContext.h:171
#15 o2::framework::MessageContext::VectorObject<char, o2::framework::MessageContext::ContainerRefObject<std::vector<char, boost::container::pmr::polymorphic_allocator<char> > > >::VectorObject<o2::framework::MessageContext*, std::unique_ptr<FairMQMessage, std::default_delete<FairMQMessage> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, unsigned long> (this=0x55c29ed9fe70) at /home/qon/alice/sw/SOURCES/O2/v1.2.0/0/Framework/Core/include/Framework/MessageContext.h:226
#16 std::make_unique<o2::framework::MessageContext::VectorObject<char, o2::framework::MessageContext::ContainerRefObject<std::vector<char, boost::container::pmr::polymorphic_allocator<char> > > >, o2::framework::MessageContext*, std::unique_ptr<FairMQMessage, std::default_delete<FairMQMessage> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, unsigned long> () at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/unique_ptr.h:857
#17 o2::framework::MessageContext::make<o2::framework::MessageContext::VectorObject<char, o2::framework::MessageContext::ContainerRefObject<std::vector<char, boost::container::pmr::polymorphic_allocator<char> > > >, std::unique_ptr<FairMQMessage, std::default_delete<FairMQMessage> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, unsigned long> (this=0x55c29c65d020) at /home/qon/alice/sw/SOURCES/O2/v1.2.0/0/Framework/Core/include/Framework/MessageContext.h:384
#18 o2::framework::MessageContext::add<o2::framework::MessageContext::VectorObject<char, o2::framework::MessageContext::ContainerRefObject<std::vector<char, boost::container::pmr::polymorphic_allocator<char> > > >, std::unique_ptr<FairMQMessage, std::default_delete<FairMQMessage> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, unsigned long> (this=0x55c29c65d020) at /home/qon/alice/sw/SOURCES/O2/v1.2.0/0/Framework/Core/include/Framework/MessageContext.h:371
#19 o2::framework::DataAllocator::make<std::vector<char, std::allocator<char> >, unsigned long> (this=this@entry=0x55c29c65d1b8, spec=...) at /home/qon/alice/sw/SOURCES/O2/v1.2.0/0/Framework/Core/include/Framework/DataAllocator.h:110
#20 0x00007f7c9c699780 in o2::tpc::<lambda(o2::framework::InitContext&)>::<lambda(o2::framework::ProcessingContext&)>::operator()(o2::framework::ProcessingContext &) const (__closure=<optimized out>, pc=...) at /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/g++-v9/bits/move.h:74

[davidrohr]

For reference, besides the above 2 issues, shared memory transport is fully working for the TPC workflow with TPC ZS Raw data, ClusterNative, and Compressed Clusters all moved via shared memory as zero-copy and the shared memory region is also registered correctly for GPU DMA transfer. (Needs this PR + #3528) Very nice! Let's get the above 2 issues solved and then LGTM!

[rbx]

1. When shmem transport tries to initialize a buffer and there is not enough memory, it assumes some messages are already "in flight" and will be freed eventually, so it just keeps trying until memory becomes free, or until device state changes. Admittedly this doesn't handle the case where you ask for more than the segment size. Or if you are creating multiple message before starting to send them out. I would be fine to switch the logic to throw a transport exception here, if you prefer. That way it's in your hands how to proceed if there is no memory.

2. This is an intentional race condition. We don't want to have a "master" device that is creating the shared memory segment. So the first device that requests shared memory will either find it or create it (this is properly locked). That's why you need to pass the segment size to every (shmem) device.

[davidrohr]

OK, that explains it.
For 1. : Could we on DPL level make sure to set the maximum shared memory size for all devices in case of a merged workflow?

For 2. : I think both approaches have there use cases, waiting or bailing out. So I would make it switchable. And then in my opinion the default inside DPL running on a laptop / desktop should be to throw an exception.

[ktf]

@davidrohr for 1. I suggest we track it in https://alice.its.cern.ch/jira/browse/O2-1434 and we merge this as is.

[ktf]

for 2. I agreed.

[davidrohr]

@ktf : This breaks the tpc workflow right now with default settings.

[rbx]

FairMQ @ v1.4.17 is now available with the fix for 1. The default setting is to throw fair::mq::MessageBadAlloc when out of memory. Old behavior can be set by passing --shm-throw-bad-alloc false to a device.

@Barthelemy the changes from @ktf with the signal handling are also in the tag.
@ironMann @sy-c the optimization with the bulk region calbacks is also in the tag.

The tag also contains factory->CreateMessage(size, fair::mq::Alignment{64}) (or NewMessage(size, fair::mq::Alignment{64}) if you are in a device). However it is currently only implemented for shmem sender scenario. Other cases will ignore the argument for the moment. Their implementation is coming in the near future.