diff --git a/_docs/concepts/client_library/execution_management/index.md b/_docs/concepts/client_library/execution_management/index.md
index d8eb3560..5a1d742a 100644
--- a/_docs/concepts/client_library/execution_management/index.md
+++ b/_docs/concepts/client_library/execution_management/index.md
@@ -15,11 +15,27 @@ redirect_from:
* [Architecture](#architecture)
* [Scheduling Semantics](#scheduling-semantics)
-* [RCLC-Executor](#rclc-executor)
- * [Requirement Analysis](#requirement-analysis)
+* [Analysis of processing patterns](#analysis-of-processing-patterns)
+ * [Sense-plan-act pipeline in robotics](#sense-plan-act-pipeline-in-robotics)
+ * [Synchronization of multiple rates](#synchronization-of-multiple-rates)
+ * [High-priority processing path](#high-priority-processing-path)
+ * [Real-time embedded applications](#real-time-embedded-applications)
+* [rclc Executor](#rclc-executor)
* [Features](#features)
+ * [Trigger condition](#trigger-condition)
+ * [Sequential execution](#sequential-execution)
+ * [LET-Semantics](#let-semantics)
+ * [Multi-threading and scheduling configuration](#multi-threading-and-scheduling-configuration)
* [Executor API](#executor-api)
+ * [Configuration phase](#configuration-phase)
+ * [Running phase](#running-phase)
* [Examples](#examples)
+ * [Sense-plan-act pipeline in robotics example](#sense-plan-act-pipeline-in-robotics-example)
+ * [Synchronization of multiple rates example](#synchronization-of-multiple-rates-example)
+ * [High-priority processing path example](#high-priority-processing-path-example)
+ * [Real-time embedded applications example](#real-time-embedded-applications-example)
+ * [ROS 2 Executor Workshop Reference System](#ros-2-executor-workshop-reference-system)
+ * [Future work](#future-work)
* [Download](#download)
* [Callback-group-level Executor](#callback-group-level-executor)
@@ -27,38 +43,40 @@ redirect_from:
* [Test Bench](#test-bench)
* [Related Work](#related-work)
+ * [Fawkes Framework](#fawkes-framework)
* [References](#references)
* [Acknowledgments](#acknowledgments)
## Introduction
-Predictable execution under given real-time constraints is a crucial requirement for many robotic applications. While the service-based paradigm of ROS allows a fast integration of many different functionalities, it does not provide sufficient control over the execution management. For example, there are no mechanisms to enforce a certain execution order of callbacks within a node. Also the execution order of multiple nodes is essential for control applications in mobile robotics. Cause-effect-chains comprising of sensor acquisition, evaluation of data and actuation control should be mapped to ROS nodes executed in this order, however there are no explicit mechanisms to enforce it. Furthermore, when input data is collected in field tests, saved with ROS-bags and re-played, often results are different due to non-determinism of process scheduling.
+Predictable execution under given real-time constraints is a crucial requirement for many robotic applications. While the service-based paradigm of ROS allows a fast integration of many different functionalities, it does not provide sufficient control over the execution management. For example, there are no mechanisms to enforce a certain execution order of callbacks within a node. Also the execution order of multiple nodes is essential for control applications in mobile robotics. Cause-effect-chains comprising of sensor acquisition, evaluation of data and actuation control should be mapped to ROS nodes executed in this order, however there are no explicit mechanisms to enforce it. Furthermore, when data recordings collected in field tests as ROS-bags are re-played, then the results are often surprisingly different due to non-determinism of process scheduling.
-Manually setting up a particular execution order of subscribing and publishing topics in the callbacks or by tweaking the priorities of the corresponding Linux processes is always possible. However, this approach is error-prone, difficult to extend and requires an in-depth knowledge of the deplyed ROS 2 packages in the system.
+Manually setting up a particular execution order of subscriptions and publishing topics as well as defining use-case specific priorities of the corresponding Linux processes is always possible. However, this approach is error-prone, difficult to extend and requires an in-depth knowledge of the deployed ROS 2 packages in the system.
-Therefore the goal of the Real-Time Executor is to support roboticists with practical and easy-to-use real-time mechanisms which provide solutions for:
+Therefore the goal of the Executor in micro-ROS is to support roboticists with practical and easy-to-use real-time mechanisms which provide solutions for:
- Deterministic execution
- Real-time guarantees
- Integration of real-time and non real-time functionalities on one platform
- Specific support for RTOS and microcontrollers
-In ROS 1 a network thread is responsible for receiving all messages and putting them into a FIFO queue (in roscpp). That is, all callbacks were called in a FIFO manner, without any execution management. With the introduction of DDS (data distribution service) in ROS 2, the messages are buffered in DDS. In ROS 2, an Executor concept was introduced to support execution management, like priorization. At the rcl-layer, a _wait-set_ is configured with handles to be received and in a second step, the handles are taken from the DDS-queue. A handle is a term defined in rcl-layer and summarizes timers, subscriptions, clients and services etc..
+In ROS 1 a network thread is responsible for receiving all messages and putting them into a FIFO queue (in roscpp). That is, all callbacks were called in a FIFO manner, without any execution management. With the introduction of DDS (data distribution service) in ROS 2, the messages are buffered in DDS. In ROS 2, an Executor concept was introduced to support execution management. At the rcl-layer, a _wait-set_ is configured with handles to be received and in a second step, the handles are taken from the DDS-queue. A handle is a generic term defined in rcl-layer for timers, subscriptions, services, clients and guard conditions.
-The standard implementation of the ROS 2 Executor for the C++ API (rclcpp) has, however, certain unusual features, like precedence of timers over all other DDS handles, non-preemptive round-robin scheduling for non-timer handles and considering only one input data for each handle (even if multiple could be available). These features have the consequence, that in certain situations the standard rclcpp Executor is not deterministic and it makes proving real-time guarantees hard. We have not looked at the ROS 2 Executor implementation for Python frontend (rclpy) because we consider a micro-controllers platform, on which typically C or C++ appliations will run.
+The standard implementation of the ROS 2 Executor for the C++ API (rclcpp) has, however, certain unusual features, like precedence of timers over all other DDS handles, non-preemptive round-robin scheduling for non-timer handles and considering only one input data for each handle (even if multiple could be available). These features have the consequence, that in certain situations the standard rclcpp Executor is not deterministic and it makes guaranteeing real-time requirements very hard [[CB2019](#CB2019)]. We have not looked at the ROS 2 Executor implementation for Python Frontend (rclpy) because we consider a micro-controllers platform, on which typically C or C++ appliations will run.
Given the goals for a Real-Time Executor and the limitations of the ROS 2 standard rclcpp Executor, the challenges are:
- to develop an adequate and well-defined scheduling mechanisms for the ROS 2 framework and the real-time operating system (RTOS)
-- to define an easy-to-use interface for ROS-developers
+- to define an easy-to-use interface for ROS developers
- to model requirements (like latencies, determinism in subsystems)
-- mapping of ROS framework and OS scheduler (semi-automated and optimized mapping is desired as well as generic, well-understood framework mechanisms)
+- mapping of ROS 2 framework and operating system schedulers (semi-automated and optimized mapping is desired as well as generic, well-understood framework mechanisms)
-Our approach is to provide Real-Time Executors on two layers as described in section [Introduction to Client Library](../). One based on the rcl-layer written in C programming language and one based on rclcpp written in C++.
+Our approach is to provide a real-time-capable Executor for the rcl+rclc layer (as described in section [Introduction to Client Library](../).) in the C programming language .
-As the first step, we propose the LET-Executor for the rcl-layer in C, which implements static order scheduling policy with logic execution time semantics. In this scheduling policy, all callbacks are executed in a pre-defined order. Logical execution time refers to the concept, that first input data is read before tasks are executed. Secondly, we developed a Callback-group-level Executor, which allows to prioritize a group of callbacks. These approaches are based on the concept of Executors, which have been introduced in ROS 2.
+As the first step, we propose the rclc Executor for the rcl-layer in C programming language with several new features to support real-time and deterministic execution: It supports 1.) user-defined static sequential execution, 2) conditional execution semantics, 3) multi-threaded execution with scheduling configuration, and 4) logical execution semantics (LET). Sequential execution refers to the runtime behavior, that all callbacks are executed in a pre-defined order independent of the arrival time of messages. Conditional execution is available with a trigger condition which enables typical processing patterns in robotics (which are analyzed in detail in section [Analysis of processing patterns](#analysis-of-processing-patterns). Configuration of scheduling parameters for multi-threaded application accomplishes prioritized execution. The logical execution time concept (LET) provides data synchronization for fixed periodic task scheduling of embedded applications.
+
+Beyond the advanced execution management mechanisms for micro-ROS, we also contributed to improving and extending the Executor concept in rclcpp for standard ROS 2: the callback group-level Executor. It is not a new Executor but rather a refinement of the ROS 2 Executor API allowing to prioritize a group of callbacks which is not possible with the ROS 2 default Executor in its current Galactic release.
-In the future, we plan to provide other Real-Time Executors for the rcl- and rclcpp-layer.
## Analysis of rclcpp standard Executor
@@ -72,19 +90,23 @@ The dispatching mechanism resembles the ROS 1 spin thread behavior: the Executor
The following diagram depicts the relevant classes of the standard ROS 2 Executor implementation:
-
+
+
+
Note that an Executor instance maintains weak pointers to the NodeBaseInterfaces of the nodes only. Therefore, nodes can be destroyed safely, without notifying the Executor.
Also, the Executor does not maintain an explicit callback queue, but relies on the queue mechanism of the underlying DDS implementation as illustrated in the following sequence diagram:
-
+
+
+
The Executor concept, however, does not provide means for prioritization or categorization of the incoming callback calls. Moreover, it does not leverage the real-time characteristics of the underlying operating-system scheduler to have finer control on the order of executions. The overall implication of this behavior is that time-critical callbacks could suffer possible deadline misses and a degraded performance since they are serviced later than non-critical callbacks. Additionally, due to the FIFO mechanism, it is difficult to determine usable bounds on the worst-case latency that each callback execution may incur.
### Scheduling Semantics
-In a recent paper [CB2019](#CB2019), the rclcpp Executor has been analyzed in detail and a response time analysis of cause-effect chains has been proposed under reservation-based scheduling. The Executor distinguishes four categories of callbacks: _timers_, which are triggered by system-level timers, _subscribers_, which are triggered by new messages on a subscribed topic, _services_, which are triggered by service requests, and _clients_, which are triggered by responses to service requests. The Executor is responsible for taking messages from the input queues of the DDS layer and executing the corresponding callback. Since it executes callbacks to completion, it is a non-preemptive scheduler, However it does not consider all ready tasks for execution, but only a snapshot, called readySet. This readySet is updated when the Executor is idle and in this step it interacts with the DDS layer updating the set of ready tasks. Then for every type of task, there are dedicated queues (timers, subscriptions, services, clients) which are processed sequentially. The following undesired properties were pointed out:
+In a recent paper [[CB2019](#CB2019)], the rclcpp Executor has been analyzed in detail and a response time analysis of cause-effect chains has been proposed under reservation-based scheduling. The Executor distinguishes four categories of callbacks: _timers_, which are triggered by system-level timers, _subscribers_, which are triggered by new messages on a subscribed topic, _services_, which are triggered by service requests, and _clients_, which are triggered by responses to service requests. The Executor is responsible for taking messages from the input queues of the DDS layer and executing the corresponding callback. Since it executes callbacks to completion, it is a non-preemptive scheduler, However it does not consider all ready tasks for execution, but only a snapshot, called readySet. This readySet is updated when the Executor is idle and in this step it interacts with the DDS layer updating the set of ready tasks. Then for every type of task, there are dedicated queues (timers, subscriptions, services, clients) which are processed sequentially. The following undesired properties were pointed out:
* Timers have the highest priority. The Executor processes _timers_ always first. This can lead to the intrinsic effect, that in overload situations messages from the DDS queue are not processed.
* Non-preemptive round-robin scheduling of non-timer handles. Messages arriving during the processing of the readySet are not considered until the next update, which depends on the execution time of all remaining callbacks. This leads to priority inversion, as lower-priority callbacks may implicitly block higher-priority callbacks by prolonging the current processing of the readySet.
@@ -92,135 +114,160 @@ In a recent paper [CB2019](#CB2019), the rclcpp Executor has been analyzed in de
Due to these findings, the authors present an alternative approach to provide determinism and to apply well-known schedulability analyses to a ROS 2 systems. A response time analysis is described under reservation-based scheduling.
-## RCLC-Executor
-Here we introduce the rclc Executor, which is a ROS 2 Executor implemented based on and for the rcl API, for applications written in the C language. Often embedded applications require real-time to guarantee end-to-end latencies and need deterministic runtime behavior to correctly replay test data. However, this is difficult with the default ROS 2 Executor because of its complex semantics, as discussed in the previous section.
-
-First, we will analyse the requirements for such applications and, secondly, derive simple features for an Executor to enable deterministic and real-time behavior. Then we will present the API of the RCLC-Executor and provide example usages of the RCLC-Executor to address these requirements.
-
-### Requirement Analysis
-First we discuss a use-case in the embedded domain, in which the time-triggered paradigm is often used to guarantee deterministic and real-time behavior. Then we analyse software design patterns in mobile robotics which enable deterministic behavior.
-
-#### Real-time embedded application used-case
-In embedded systems, real-time behavior is approached by using the time-triggered paradigm, which means that the processes are periodically activated. Processes can be assigned priorities to allow pre-emptions. Figure 1 shows an example, in which three processes with fixed periods are shown. The middle and lower process are preempted multiple times depicted with empty dashed boxes.
-
-
-
-Figure 1: Fixed periodic preemptive scheduling
-
-To each process one or multiple tasks can be assigned, as shown in Figure 2. These tasks are executed sequentially, which is often called cooperative scheduling.
-
-
-
-Figure 2: Processes with sequentially executed tasks.
-
-While there are different ways to assign priorities to a given number of processes,
-the rate-monotonic scheduling assignment, in which processes with a shorter period have a higher priority, has been shown optimal if the processor utilization is less than 69% [LL1973](#LL1973).
-
- In the last decades many different scheduling approaches have been presented, however fixed-periodic preemptive scheduling is still widely used in embedded real-time systems [KZH2015](#KZH2015). This becomes also obvious, when looking at the features of current operating systems. Like Linux, real-time operating systems, such as NuttX, Zephyr, FreeRTOS, QNX etc., support fixed-periodic preemptive scheduling and the assignment of priorities, which makes the time-triggered paradigm the dominant design principle in this domain.
-
-However, data consistency is often an issue when preemptive scheduling is used and if data is being shared across multiple processes via global variables. Due to scheduling effects and varying execution times of processes, writing and reading these variables could occur sometimes sooner or later. This results in an latency jitter of update times (the timepoint at which a variable change becomes visible to other processes). Race conditions can occur when multiple processes access a variable at the same time. So solve this problem, the concept of logical-execution time (LET) was introduced in [HHK2001](#HHK2001), in which communication of data occurs only at pre-defined periodic time instances: Reading data only at the beginning of the period and writing data only at the end of the period. The cost of an additional latency delay is traded for data consistency and reduced jitter. This concept has also recently been applied to automotive applications [NSP2018](#NSP2018).
-
-
-
-Figure 3: Data communication without and with Logical Execution Time paradigm.
-
-An Example of the LET concept is shown in Figure 3. Assume that two processes are communicating data via one global variable. The timepoint when this data is written is at the end of the processing time. In the default case (left side), the process p3 and p4 receive the update. At the right side of the figure, the same scenario is shown with LET semantics. Here, the data is communicated only at period boundaries. In this case, the lower process communicates at the end of the period, so that always process p3 and p5 receive the new data.
-
-The described embedded use case relies on the following concepts:
-- periodic execution of processes
-- assignment of fixed priorities to processes
-- preemptive scheduling of processes
-- co-operative scheduling of tasks within a process (sequential execution)
-- data synchronization with LET-semantics
-
-While periodic activation is possible in ROS2 by using timers, preemptive scheduling is supported by the operating system and assigning priorities on the granularity of threads/processes that correspond to the ROS nodes; it is not possible to sequentially execute callbacks, which have no data-dependency. Furthermore data is read from the DDS queue just before the callback is executed and data is written sometime during the time the application is executed. While the `spin_period` function of the rclcpp-Executor allows to check for data at a fixed period and executing those callbacks for which data is available, however, with this spin-function does not execute all callbacks irrespective wheter data is available or not. So `spin_period` is not helpful to periodically execute a number of callbacks (aka tasks within a process). So we need a mechanism that triggers the execution of multiple callbacks (aka tasks) based on a timer. Data transmission is achieved via DDS which does not allow to implement a LET-semantics. To summarize, we derive the following requirements:
-
-Derived Requirements:
-- trigger the execution of multiple callbacks
-- sequential processing of callbacks
-- data synchronization with LET semantics
+## Analysis of processing patterns
+The development of an execution management mechanism for micro-ROS is based on an analysis of processing patterns commonly used in robotics and embedded systems. First, typical processing patterns in mobile robotics are presented which are used to implement deterministic behavior. Then, the processing patterns in the real-time embedded systems are analyzed, in which, as a main difference, the time-triggered paradigm is applied to accomplish real-time behavior.
-#### Sense-plan-act pipeline in robotics
+### Sense-plan-act pipeline in robotics
Now we describe common software design patterns which are used in mobile robotics to achieve deterministic behavior. For each design pattern we describe the concept and the derived requirements for a deterministic Executor.
-Concept:
-A common design paradigm in mobile robotics is a control loop, consisting of several phases: A sensing phase to aquire sensor data, a plan phase for localization and path planning and an actuation-phase to steer the mobile robot. Of course, more phases are possible, here these three phases shall serve as an example. Such a processing pipeline is shown in Figure 4.
+**Concept:**
-
+A common design paradigm in mobile robotics is a control loop, consisting of several phases: A sensing phase to aquire sensor data, a plan phase for localization and path planning and an actuation-phase to steer the mobile robot. Of course, more phases are possible, here these three phases shall serve as an example. Such a processing pipeline is shown in Figure 4.
+
+
+
+
Figure 4: Multiple sensors driving a Sense-Plan-Act pipeline.
+
Typically multiple sensors are used to perceive the environment. For example an IMU and a laser scanner. The quality of localization algorithms highly depend on how old such sensor data is when it is processed. Ideally the latest data of all sensors should be processed. One way to achive this is to execute first all sensor drivers in the sense-phase and then process all algorithms in the plan-phase.
-Currently, such a processing order cannot be defined with the default ROS2-Executor. One could in principle design a data-driven pipeline, however if e.g. the Laser scan is needed by some other callback in the sense-phase as well as in the plan-phase, the processing order of these subscribers is arbitrary.
+Currently, such a processing order cannot be defined with the default Executor of rclcpp. One could in principle design a data-driven pipeline, however if e.g. the Laser scan is needed by some other callback in the sense-phase as well as in the plan-phase, the processing order of these subscribers is arbitrary.
For this sense-plan-act pattern, we could define one executor for each phase. The plan-phase would be triggered only when all callbacks in the sense-phase have finished.
-Derived Requirements:
-- triggered execution of callbacks
+**Derived requirements:**
+- triggered execution
-#### Synchronization of multiple rates
+### Synchronization of multiple rates
-Concept:
+**Concept:**
Often multiple sensors are being used to sense the environment for mobile robotics. While an IMU sensor provides data samples at a very high rate (e.g., 500 Hz), laser scans are availabe at a much slower frequency (e.g. 10Hz) determined by the revolution time. Then the challenge is, how to deterministically fuse sensor data with different frequencies. This problem is depicted in Figure 5.
-
-
+
+
+
+
Figure 5: How to deterministically process multi-frequent sensor data.
+
Due to scheduling effects, the callback for evaluating the laser scan might be called just before or just after an IMU data is received. One way to tackle this is to write additional synchronization code inside the application. Obviously, this is a cumbersome and not-portable solution.
An Alternative would be to evalute the IMU sample and the laser scan by synchronizing their frequency. For example by processing always 50 IMU samples with one laser scan. This approach is shown in Figure 6. A pre-processing callback aggregates the IMU samples and sends an aggregated message with 50 samples at 10Hz rate. Now both messages have the same frequency. With a trigger condition, which fires when both messages are available, the sensor fusion algorithm can expect always synchronized input data.
-
-
+
+
+
+
Figure 6: Synchronization of multiple input data with a trigger.
+
-In ROS 2, this is currently not possible to be modeled because of the lack of a trigger concept in the ROS2 Executor. Message filters could be used to synchronize input data based on the timestamp in the header, but this is only available in rclcpp (and not in rcl). Further more, it would be more efficient to have such a trigger concept directly in the Executor.
-
+In ROS 2, this is currently not possible to be modeled because of the lack of a trigger concept in the Executors of rclcpp and rclpy. Message filters could be used to synchronize input data based on the timestamp in the header, but this is only available in rclcpp (and not in rcl). Further more, it would be more efficient to have such a trigger concept directly in the Executor.
Another idea would be to activly request for IMU data only when a laser scan is received. This concept is shown in Figure 7. Upon arrival of a laser scan mesage, first, a message with aggregated IMU samples is requested. Then, the laser scan is processed and later the sensor fusion algorithm. An Executor, which would support sequential execution of callbacks, could realize this idea.
-
-
+
+
+
+
Figure 7: Synchronization with sequential processing.
+
-
-Derived Requirements from both concepts:
+**Derived requirements:**
- triggered execution
- sequential procesing of callbacks
-#### High-priority processing path
-Motivation:
-
+### High-priority processing path
+**Concept**
Often a robot has to fullfill several activities at the same time. For example following a path and avoiding obstacles. While path following is a permanent activity, obstacle avoidance is trigged by the environment and should be immediately reacted upon. Therefore one would like to specify priorities to activities. This is depicted in Figure 8:
-
-
+
+
+
+
Figure 8: Managing high priority path with sequential order.
+
Assuming a simplified control loop with the activities sense-plan-act, the obstacle avoidance, which might temporarily stop the robot, should be processed before the planning phase. In this example we assume that these activites are processed in one thread.
-Derived requirements:
+**Derived requirements:**
- sequential processing of callbacks
-### Features
-Based on the real-time embedded use-case as well as the software architecture patterns in mobile robotics, we propose an Executor with the following main features:
-- user-defined sequential execution of callbacks
-- trigger condition to activate processing
-- data synchronization: LET-semantics or rclcpp Executor semantics
+### Real-time embedded applications
+In embedded systems, real-time behavior is approached by using the time-triggered paradigm, which means that the processes are periodically activated. Processes can be assigned priorities to allow pre-emptions. Figure 1 shows an example, in which three processes with fixed periods are shown. The middle and lower process are preempted multiple times depicted with empty dashed boxes.
+
+
+
+To each process one or multiple tasks can be assigned, as shown in Figure 2. These tasks are executed sequentially, which is often called cooperative scheduling.
+
+
+
+
+
+Figure 2: Processes with sequentially executed tasks.
+
+
+While there are different ways to assign priorities to a given number of processes,
+the rate-monotonic scheduling assignment, in which processes with a shorter period have a higher priority, has been shown optimal if the processor utilization is less than 69% [[LL1973](#LL1973)].
+
+ In the last decades many different scheduling approaches have been presented, however fixed-periodic preemptive scheduling is still widely used in embedded real-time systems [[KZH2015](#KZH2015)]. This becomes also obvious, when looking at the features of current operating systems. Like Linux, real-time operating systems, such as NuttX, Zephyr, FreeRTOS, QNX etc., support fixed-periodic preemptive scheduling and the assignment of priorities, which makes the time-triggered paradigm the dominant design principle in this domain.
+
+However, data consistency is often an issue when preemptive scheduling is used and if data is being shared across multiple processes via global variables. Due to scheduling effects and varying execution times of processes, writing and reading these variables could occur sometimes sooner or later. This results in an latency jitter of update times (the timepoint at which a variable change becomes visible to other processes). Race conditions can occur when multiple processes access a variable at the same time. So solve this problem, the concept of logical-execution time (LET) was introduced in [[HHK2001](#HHK2001)], in which communication of data occurs only at pre-defined periodic time instances: Reading data only at the beginning of the period and writing data only at the end of the period. The cost of an additional latency delay is traded for data consistency and reduced jitter. This concept has also recently been applied to automotive applications[[NSP2018](#NSP2018)].
+
+
+
+
+
+Figure 3: Data communication without and with Logical Execution Time paradigm.
+
+
+An Example of the LET concept is shown in Figure 3. Assume that two processes are communicating data via one global variable. The timepoint when this data is written is at the end of the processing time. In the default case (left side), the process p3 and p4 receive the update. At the right side of the figure, the same scenario is shown with LET semantics. Here, the data is communicated only at period boundaries. In this case, the lower process communicates at the end of the period, so that always process p3 and p5 receive the new data.
+
+**Concept:**
+- periodic execution of processes
+- assignment of fixed priorities to processes
+- preemptive scheduling of processes
+- co-operative scheduling of tasks within a process (sequential execution)
+- data synchronization with LET-semantics
+
-As stated before, this Executor is based on the RCL library and is written in C to nativly support microcontroller applications written in C. These features are now described in more detail.
+While periodic activation is possible in ROS 2 by using timers, preemptive scheduling is supported by the operating system and assigning priorities on the granularity of threads/processes that correspond to the ROS nodes; it is not possible to sequentially execute callbacks, which have no data-dependency. Furthermore data is read from the DDS queue just before the callback is executed and data is written sometime during the time the application is executed. While the `spin_period` function of the rclcpp Executor allows to check for data at a fixed period and executing those callbacks for which data is available, however, with this spin-function does not execute all callbacks irrespective wheter data is available or not. So `spin_period` is not helpful to periodically execute a number of callbacks (aka tasks within a process). So we need a mechanism that triggers the execution of multiple callbacks (aka tasks) based on a timer. Data transmission is achieved via DDS which does not allow to implement a LET-semantics. To summarize, we derive the following requirements.
-The rclc-Executor supports all event types like the ROS 2 rclc executor, which are:
-- subscription
-- timer
-- service
-- client
-- guard condition
+**Derived requirements:**
+- trigger the execution
+- sequential processing of callbacks
+- data synchronization with LET semantics
+## rclc Executor
+
+The rclc Executor is a ROS 2 Executor based on the rcl-layer in C programming language. As discussed above, the default rclcpp Executor is not suitable to implement real-time applications because of three main reasons: timers are preferred over all other handles, no priorization of callback execution and the round-robin to completion execution of callbacks. On the other hand, several processing patterns have been developed as best practices to achieve non-functional requirements, such as bounded end-to-end latencies, low jitter of response times of cause-effect chains, deterministic processing and short response times even in overload situations. These processing patterns are difficult to implement with the concepts availabe in the default ROS 2 Executor, therefore we have developed a flexible Executor: the rclc Executor.
+
+### Features
+
+The rclc Executor is feature-complete, i.e. it supports all event types as the default ROS 2 Executor, which are:
+- subscriptions
+- timers
+- services
+- clients
+- guard conditions
+- actions
+- lifecycle
+
+The flexible rclc Executor provides on top the following new features:
+- triggered execution
+- user-defined sequential execution
+- multi-threading and scheduling configuration (WIP)
+- LET-semantics for data synchronization of periodic process scheduling
+
+First, a *trigger condition* allows to define when the processing of a callback shall start. This is useful to implement sense-plan-act control loops or more complex processing structures with directed acyclic graphs. Second, a user can specify the *processing order* in which these callbacks will be executed. With this feature, the pattern of sensor fusion with multiple rates, in which data is requested from a sensor based on the arrival of some other sensor, can be easily implemented. Third, the assignment of scheduling parameters (e.g., priorities) of the underlying operating system. With this feature, prioritized processing can be implemented. Finally, for periodic applications, the *LET Semantics* has been implemented to support data consistency for periodic process scheduling. These features are now described in more detail.
#### Sequential execution
@@ -230,15 +277,62 @@ The rclc-Executor supports all event types like the ROS 2 rclc executor, which a
- if the configuration of handle is ON_NEW_DATA, then the corresponding callback is only called if new data is available
- if the configuration of the handle is ALWAYS, then the corresponding callback is always. If no data is available, then the callback is called with no data (e.g. NULL pointer).
+Figure 9 shows three callbacks, A, B and C. Assume, they shall be executed in the order *B,A,C*. Then the user adds the callbacks to the rclc Executor in this order. Whenever new messages have arrived then the callbacks for which a new message is availabe will be always executed in the user-defined processing order.
+
+
+
+
+Figure 9: Sequential execution semantics.
+
+
#### Trigger condition
-- Given a set of handles, a trigger condition based on the input data of these handles shall decide when the processing is started.
+- Given a set of handles, a trigger condition, which is based on the availability of input data of these handles, decides when the processing of all callbacks starts. This is shown in Figure 10.
+
+
+
+
+
+Figure 10: Executor with trigger condition
+
- Available options:
- ALL operation: fires when input data is available for all handles
- - ANY operation: fires when input data is available for at least one handle
+ - ANY operation: fires when input data is available for at least one handle (OR semantics)
- ONE: fires when input data for a user-specified handle is available
- - User-defined function: user can implement more sophisticated logic
+ - User-defined function: user can implement custom logic
+
+Figure 11 shows an example of the ALL semantics. Only if all messages *msg_A, msg_B, msg_C* were received, then trigger condition is fullfilled and the callbacks are processed in a user-defined order.
+
+
+
+
+Figure 11: Trigger condition ALL
+
+
+Figure 12 shows an example of the ANY semantics. Thas is, if any messages *msg_A, msg_B, msg_C* was received, then trigger condition is fullfilled and the callbacks are processed in a user-defined order. This is equivalent to OR semantics.
+
+
+
+
+Figure 12: Trigger condition ANY (OR)
+
+
+Figure 13 shows an example of the ONE semantics. Thas is, only if message *msg_B* was received, the trigger condition is fullfilled and (potentially all) callbacks are processed in a user-defined order.
+
+
+
+
+Figure 13: Trigger condition ONE
+
+
+Figure 14 describes the custom semantics. A custom trigger condition with could be a more complex logic of multiple messages, can be passed to the executor. This might also include hardware triggers, like interrupts.
+
+
+
+
+Figure 14: Trigger condition user-defined
+
#### LET-Semantics
- Assumption: time-triggered system, the executor is activated periodically
@@ -254,8 +348,19 @@ Additionally we have implemented the current rclcpp Executor semantics ("RCLCPP"
The selection of the Executor semantics is optional. The default semantics is "RCLCPP".
+#### Multi-threading and scheduling configuration
+
+The rclc Executor has been extended for multi-threading. It supports the assignment of scheduling policies, like priorities or more advanced scheduling algorithms as reservation-based scheduling, to subscription callbacks. [[Pull Request](https://github.com/ros2/rclc/pull/87), Pre-print [SLD2021](#SLD2021)]. The overall architecture is shown in Figure 15. One Executor thread is responsible for checking for new data from the DDS queue. For every callback, a thread is spawned with the dedicted scheduling policy provided by the operating system. The Executor then dispatches new data of a subscription to it's corresponding callback function, which is then executed in its own thread by operating system.
+
+
+
+
+
+Figure 15: multi-threaded rclc-Executor
+
+
### Executor API
-The API of the RCLC-Executor can be divided in two phases: Configuration and Running.
+The API of the rclc Executor can be divided in two phases: Configuration and Running.
#### Configuration phase
During the configuration phase, the user shall define:
- the total number of callbacks
@@ -263,7 +368,7 @@ During the configuration phase, the user shall define:
- trigger condition (optional, default: ANY)
- data communcation semantics (optional, default ROS2)
-As the Executor is intended for embedded controllers, dynamic memory management is crucial. Therefore at initialization of the RCLC-Executor, the user defines the total number of callbacks. The necessary dynamic memory will be allocated only in this phase and no more memory in the running phase. This makes this Executor static in the sense, that during runtime no additional callbacks can be added.
+As the Executor is intended for embedded controllers, dynamic memory management is crucial. Therefore at initialization of the rclc Executor, the user defines the total number of callbacks. The necessary dynamic memory will be allocated only in this phase and no more memory in the running phase. This makes this Executor static in the sense, that during runtime no additional callbacks can be added.
Then, the user adds handles and the corresponding callbacks (e.g. for subscriptions and timers) to the Executor. The order in which this takes place, defines later the sequential processing order during runtime.
@@ -295,70 +400,9 @@ Available spin functions are
- `spin` - spin indefinitly
### Examples
-We provide the relevant code snippets how to setup the RCLC-Executor for the embedded use case and for the software design patterns in mobile robotics applications as described above.
-#### Embedded use-case
-
-With seqential execution the co-operative scheduling of tasks within a process can be modeled. The trigger condition is used to periodically activate the process which will then execute all callbacks in a pre-defined order. Data will be communicated using the LET-semantics. Every Executor is executed in its own tread, to which an appropriate priority can be assigned.
-
-In the following example, the Executor is setup with 4 handles. We assume a process has three subscriptions `sub1`, `sub2`, `sub3`. The sequential processing order is given by the order as they are added to the Executor. A timer `timer` defines the period. The `trigger_one` with the paramter `timer` is used, so that whenever the timer is ready, all callbacks are processed. Finally the data communication semantics LET is defined.
-```C
-#include "rcl_executor/let_executor.h"
-
-// define subscription callback
-void my_sub_cb1(const void * msgin)
-{
- // ...
-}
-// define subscription callback
-void my_sub_cb2(const void * msgin)
-{
- // ...
-}
-// define subscription callback
-void my_sub_cb3(const void * msgin)
-{
- // ...
-}
-
-// define timer callback
-void my_timer_cb(rcl_timer_t * timer, int64_t last_call_time)
-{
- // ...
-}
-
-// necessary ROS 2 objects
-rcl_context_t context;
-rcl_node_t node;
-rcl_subscription_t sub1, sub2, sub3;
-rcl_timer_t timer;
-rcle_let_executor_t exe;
-
-// define ROS context
-context = rcl_get_zero_initialized_context();
-// initialize ROS node
-rcl_node_init(&node, &context,...);
-// create subscriptions
-rcl_subscription_init(&sub1, &node, ...);
-rcl_subscription_init(&sub2, &node, ...);
-rcl_subscription_init(&sub3, &node, ...);
-// create a timer
-rcl_timer_init(&timer, &my_timer_cb, ... );
-// initialize executor with four handles
-rclc_executor_init(&exe, &context, 4, ...);
-// define static execution order of handles
-rclc_executor_add_subscription(&exe, &sub1, &my_sub_cb1, ALWAYS);
-rclc_executor_add_subscription(&exe, &sub2, &my_sub_cb2, ALWAYS);
-rclc_executor_add_subscription(&exe, &sub3, &my_sub_cb3, ALWAYS);
-rclc_executor_add_timer(&exe, &timer);
-// trigger when handle 'timer' is ready
-rclc_executor_set_trigger(&exe, rclc_executor_trigger_one, &timer);
-// select LET-semantics
-rclc_executor_data_comm_semantics(&exe, LET);
-// spin forever
-rclc_executor_spin(&exe);
-```
+We provide the relevant code snippets how to setup the rclc Executor for the processing patterns as described above.
-#### Sense-plan-act pipeline
+#### Sense-plan-act pipeline in robotics example
In this example we want to realise a sense-plan-act pipeline in a single thread. The trigger condition is demonstrated by activating
the sense-phase when both data for the Laser and IMU are available. Three executors are necessary `exe_sense`, `exe_plan` and `exe_act`. The two sensor acquisition callbacks `sense_Laser` and `sense_IMU` are registered in the Executor `exe_sense`.
@@ -390,7 +434,8 @@ while (true) {
rclc_executor_spin_some(&exe_act);
}
```
-#### Sensor fusion
+
+#### Synchronization of multiple rates example
The sensor fusion synchronizing the multiple rates with a trigger is shown below.
@@ -432,7 +477,7 @@ rclc_executor_set_trigger(&exe_sense, rclc_executor_trigger_one, &sense_Laser);
// spin
rclc_executor_spin(&exe_sense);
```
-#### High priorty path
+#### High priority processing path example
This example shows the sequential processing order to execute the obstacle avoidance `obst_avoid`
after the callbacks of the sense-phase and before the callback of the planning phase `plan`.
@@ -458,9 +503,74 @@ rclc_executor_set_trigger(&exe, rclc_executor_trigger_one, &sense_Laser);
rclc_executor_spin(&exe);
```
-### Summary
+#### Real-time embedded applications example
+
+With seqential execution the co-operative scheduling of tasks within a process can be modeled. The trigger condition is used to periodically activate the process which will then execute all callbacks in a pre-defined order. Data will be communicated using the LET-semantics. Every Executor is executed in its own tread, to which an appropriate priority can be assigned.
+
+In the following example, the Executor is setup with 4 handles. We assume a process has three subscriptions `sub1`, `sub2`, `sub3`. The sequential processing order is given by the order as they are added to the Executor. A timer `timer` defines the period. The `trigger_one` with the paramter `timer` is used, so that whenever the timer is ready, all callbacks are processed. Finally the data communication semantics LET is defined.
+```C
+#include "rcl_executor/let_executor.h"
+
+// define subscription callback
+void my_sub_cb1(const void * msgin)
+{
+ // ...
+}
+// define subscription callback
+void my_sub_cb2(const void * msgin)
+{
+ // ...
+}
+// define subscription callback
+void my_sub_cb3(const void * msgin)
+{
+ // ...
+}
+
+// define timer callback
+void my_timer_cb(rcl_timer_t * timer, int64_t last_call_time)
+{
+ // ...
+}
+
+// necessary ROS 2 objects
+rcl_context_t context;
+rcl_node_t node;
+rcl_subscription_t sub1, sub2, sub3;
+rcl_timer_t timer;
+rcle_let_executor_t exe;
+
+// define ROS context
+context = rcl_get_zero_initialized_context();
+// initialize ROS node
+rcl_node_init(&node, &context,...);
+// create subscriptions
+rcl_subscription_init(&sub1, &node, ...);
+rcl_subscription_init(&sub2, &node, ...);
+rcl_subscription_init(&sub3, &node, ...);
+// create a timer
+rcl_timer_init(&timer, &my_timer_cb, ... );
+// initialize executor with four handles
+rclc_executor_init(&exe, &context, 4, ...);
+// define static execution order of handles
+rclc_executor_add_subscription(&exe, &sub1, &my_sub_cb1, ALWAYS);
+rclc_executor_add_subscription(&exe, &sub2, &my_sub_cb2, ALWAYS);
+rclc_executor_add_subscription(&exe, &sub3, &my_sub_cb3, ALWAYS);
+rclc_executor_add_timer(&exe, &timer);
+// trigger when handle 'timer' is ready
+rclc_executor_set_trigger(&exe, rclc_executor_trigger_one, &timer);
+// select LET-semantics
+rclc_executor_data_comm_semantics(&exe, LET);
+// spin forever
+rclc_executor_spin(&exe);
+```
+
+#### ROS 2 Executor Workshop Reference System
+The rclc Executor has been presented at the workshop 'ROS 2 Executor: How to make it efficient, real-time and deterministic?' at [ROS World 2021](https://roscon.ros.org/world/2021/) (i.e. the online version of ROSCon)[[S2021](#S2021)]. A [Reference System](https://github.com/ros-realtime/reference-system) for testing and benchmarking ROS Executors has been developed for this workshop. The application of the rclc Executor on the reference system with the trigger condition can be found in the [rclc-executor branch of the Reference System](https://github.com/ros-realtime/reference-system/tree/rclc_executor).
+
+
-The RCLC Executor is an Executor for C applications and can be used with default rclcpp Executor semantics. If additional deterministic behavior is necessary, the user can rely on pre-defined sequential execution, trigged execution and LET-Semantics for data synchronization. The concept of the rclc-Executor has been published in [SLL2020](#SLL2020).
+The slides can be downloaded [here](https://ec2a4d36-bac8-4759-b25e-bb1f794177f4.filesusr.com/ugd/984e93_749e27b917a54b45b9ccb5be930841b8.pdf). All information and the videos and slides of the other talks of the workshop can be found at [www.apex.ai/roscon-21](https://www.apex.ai/roscon-21).
### Future work
@@ -468,11 +578,10 @@ The RCLC Executor is an Executor for C applications and can be used with default
- one publisher that periodically publishes
- if Executors are running in multiple threads,
publishing needs to be atomic
+- Multi-threaded executor with assignment of scheduling policies of unerlying operating system. [[Pull Request](https://github.com/ros2/rclc/pull/87), pre-print [SLD2021](#SLD2021)].
### Download
-The RCLC-Executor can be downloaded from the [micro-ROS rclc repository](https://github.com/ros2/rclc). In this repository, the [rclc package](https://github.com/ros2/rclc/tree/master/rclc) provides the RCLC-Executor and the [rclc_examples package](https://github.com/ros2/rclc/tree/master/rclc_examples) provides several demos. Further more, the rclc Executor is also available from the [ros2/rclc repository](https://github.com/ros2/rclc).
-
-
+The rclc Executor can be downloaded from the [ros2/rclc repository](https://github.com/ros2/rclc). It is available for the ROS 2 versions Foxy, Galactic and Rolling. The repository provides several packages including the [rclc Executor](https://github.com/ros2/rclc/tree/master/rclc) and an [rclc_examples package](https://github.com/ros2/rclc/tree/master/rclc_examples) with several application examples.
## Callback-group-level Executor
@@ -484,27 +593,29 @@ Thus, an Executor instance can be dedicated to specific callback group(s) and th
The following figure illustrates this approach with two nodes served by three Callback-group-level Executors in one process:
-
+
+
+
The different callbacks of the Drive-Base node are distributed to different Executors (visualized by the color red, yellow and green). For example the onCmdVel and publishWheelTicks callback are scheduled by the same Executor (yellow). Callbacks from different nodes can be serviced by the same Executor.
### API Changes
In this section, we describe the necessary changes to the Executor API:
-* [include/rclcpp/callback\_group.hpp](https://github.com/boschresearch/ros2_rclcpp/tree/cbg-executor-0.5.1/rclcpp/include/rclcpp/callback_group.hpp):
+* [include/rclcpp/callback\_group.hpp](https://github.com/ros2/rclcpp/blob/master/rclcpp/include/rclcpp/callback_group.hpp):
* Introduced an enum to distinguish up to three real-time classes (requirements) per node (RealTimeCritical, SoftRealTime, BestEffort)
* Changed association with Executor instance from nodes to callback groups.
-* [include/rclcpp/executor.hpp](https://github.com/boschresearch/ros2_rclcpp/tree/cbg-executor-0.5.1/rclcpp/include/rclcpp/executor.hpp)
+* [include/rclcpp/executor.hpp](https://github.com/ros2/rclcpp/blob/master/rclcpp/include/rclcpp/executor.hpp)
* Added functions to add and remove individual callback groups in addition to whole nodes.
* Replaced private vector of nodes with a map from callback groups to nodes.
-* [include/rclcpp/memory\_strategy.hpp](https://github.com/boschresearch/ros2_rclcpp/tree/cbg-executor-0.5.1/rclcpp/include/rclcpp/memory_strategy.hpp)
+* [include/rclcpp/memory\_strategy.hpp](https://github.com/ros2/rclcpp/blob/master/rclcpp/include/rclcpp/memory_strategy.hpp)
* Changed all functions that expect a vector of nodes to the just mentioned map.
-* [include/rclcpp/node.hpp](https://github.com/boschresearch/ros2_rclcpp/tree/cbg-executor-0.5.1/rclcpp/include/rclcpp/node.hpp) and [include/rclcpp/node_interfaces/node_base.hpp](https://github.com/boschresearch/ros2_rclcpp/tree/cbg-executor-0.5.1/rclcpp/include/rclcpp/node_interfaces/node_base.hpp)
+* [include/rclcpp/node.hpp](https://github.com/ros2/rclcpp/blob/master/rclcpp/include/rclcpp/node.hpp) and [include/rclcpp/node_interfaces/node_base.hpp](https://github.com/ros2/rclcpp/blob/master/rclcpp/include/rclcpp/node_interfaces/node_base.hpp)
* Extended arguments of create\_callback\_group function for the real-time class.
* Removed the get\_associated\_with\_executor\_atomic function.
@@ -515,13 +626,19 @@ The callback-group-level executor has been merged into ROS 2 rclcpp in [pull req
As a proof of concept, we implemented a small test bench in the present package cbg-executor_ping-pong_cpp. The test bench comprises a Ping node and a Pong node which exchange real-time and best-effort messages simultaneously with each other. Each class of messages is handled with a dedicated Executor, as illustrated in the following figure.
-
+
+
+
+
With the test bench, we validated the functioning of the approach.
-
+
+
+
+
In this example, the callback for the high priority task (red line) consumes 10ms and the low priority task (blue line) 40ms in the Pong Node. With a ping rate of 20 Hz, the CPU saturates (10ms\*20+40ms\*20=1000ms). With higher frequencies the high priorty task can continue to send its pong message, while the low priority pong task degrades. With a frequency of 100Hz the high priority task requires 100% CPU utilization. With higher ping rates it keeps sending pong messages with 100Hz, while the low priority task does not get any CPU ressources any more and cannot send any messages.
-The test bench is provided in the [cbg_executor_demo](https://github.com/boschresearch/ros2_demos/tree/cbg_executor_demo/cbg_executor_demo).
+The test bench is provided in the [cbg_executor_demo](https://github.com/ros2/examples/tree/master/rclcpp/executors/cbg_executor).
## Related Work
@@ -570,7 +687,9 @@ A module (thread) can be configured independent of these sense-plan-act synchron
The high level overview of the Fawkes framework is shown in the next figure. At compile-time the configuration of the sense-plan act wakeup hook is done (upper part), while at run-time the scheduler iterates through this list of wakeup-hooks (lower part):
-
+
+
+
Hence, at run-time, the hooks are executed as a fixed static schedule without preemption. Multiple threads registered in the same hook are executed in parallel.
@@ -608,27 +727,31 @@ URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8376277&isnumber=83
## References
+* [S2021] J. Staschulat, "Micro-ROS: The rclc Executor", in Workshop ROS 2 Executor: How to make it efficient, real-time and deterministic? at ROS World, Oct. 2021, [[slides](https://ec2a4d36-bac8-4759-b25e-bb1f794177f4.filesusr.com/ugd/984e93_749e27b917a54b45b9ccb5be930841b8.pdf)] [[Video](https://www.youtube.com/embed/IazrPF3RN1U)]
+
+* [SLD2021] J. Staschulat, R. Lange and D. N. Dasari, "Budget-based real-time Executor for Micro-ROS", arXiv Pre-Print, May 2021. [[paper](https://arxiv.org/abs/2105.05590)]
* [L2020] Ralph Lange: Advanced Execution Management with ROS 2, ROS-Industrial Conference, Dec 2020 [[Slides]](https://micro-ros.github.io/download/2020-12-16_Advanced_Execution_Management_with_ROS_2.pdf)
* [SLL2020] J. Staschulat, I. Lütkebohle and R. Lange, "The rclc Executor: Domain-specific deterministic scheduling mechanisms for ROS applications on microcontrollers: work-in-progress," 2020 International Conference on Embedded Software (EMSOFT), Singapore, Singapore, 2020, pp. 18-19. [[Paper]](https://ieeexplore.ieee.org/document/9244014) [[Video]](https://whova.com/embedded/session/eswe_202009/1145800/)
-* [L2018] Ralph Lange: Callback-group-level Executor for ROS 2. Lightning talk at ROSCon 2018. Madrid, Spain. Sep 2018. [[Slides]](https://roscon.ros.org/2018/presentations/ROSCon2018_Lightning1_4.pdf) [[Video]](https://vimeo.com/292707644)
-
* [CB2019] D. Casini, T. Blaß, I. Lütkebohle, B. Brandenburg: Response-Time Analysis of ROS 2 Processing Chains under Reservation-Based Scheduling, in Euromicro-Conference on Real-Time Systems 2019. [[Paper]](http://drops.dagstuhl.de/opus/volltexte/2019/10743/) [[slides]](https://t-blass.de/talks/ECRTS2019.pdf)
+* [L2018] Ralph Lange: Callback-group-level Executor for ROS 2. Lightning talk at ROSCon 2018. Madrid, Spain. Sep 2018. [[Slides]](https://roscon.ros.org/2018/presentations/ROSCon2018_Lightning1_4.pdf) [[Video]](https://vimeo.com/292707644)
+
* [EK2018] R. Ernst, S. Kuntz, S. Quinton, M. Simons: The Logical Execution Time Paradigm: New Perspectives for Multicore Systems, February 25-28 2018 (Dagstuhl Seminar 18092). [[Paper]](http://drops.dagstuhl.de/opus/volltexte/2018/9293/pdf/dagrep_v008_i002_p122_18092.pdf)
+* [NSP2018] A. Naderlinger, S. Resmerita, and W. Pree: LET for Legacy and Model-based Applications,
+Proceedings of The Logical Execution Time Paradigm: New Perspectives for Multicore Systems (Dagstuhl Seminar 18092), Wadern, Germany, February 2018.
+
* [BP2017] A. Biondi, P. Pazzaglia, A. Balsini, M. D. Natale: Logical Execution Time Implementation and Memory Optimization Issues in AUTOSAR Applications for Multicores, International Worshop on Analysis Tools and Methodologies for Embedded and Real-Time Systems (WATERS2017), Dubrovnik, Croatia.[[Paper]](https://pdfs.semanticscholar.org/4a9e/b9a616c25fd0b4a4f7810924e73eee0e7515.pdf)
-* [LL1973] Liu, C. L.; Layland, J.:Scheduling algorithms for multiprogramming in a hard real-time environment, Journal of the ACM, 20 (1): 46–61, 1973.
+* [KZH2015] S. Kramer, D. Ziegenbein, and A. Hamann: Real World Automotive Benchmarks For Free, International Workshop on Analysis Tools and Methodologies for Embedded adn Real-Time Sysems (WATERS), 2015.
* [HHK2001] Henzinger T.A., Horowitz B., Kirsch C.M. (2001) Giotto: A Time-Triggered Language for Embedded Programming. In: Henzinger T.A., Kirsch C.M. (eds) Embedded Software. EMSOFT 2001. Lecture Notes in Computer Science, vol 2211. Springer, Berlin, Heidelberg
-* [NSP2018] A. Naderlinger, S. Resmerita, and W. Pree: LET for Legacy and Model-based Applications,
-Proceedings of The Logical Execution Time Paradigm: New Perspectives for Multicore Systems (Dagstuhl Seminar 18092), Wadern, Germany, February 2018.
+* [LL1973] Liu, C. L.; Layland, J.:Scheduling algorithms for multiprogramming in a hard real-time environment, Journal of the ACM, 20 (1): 46–61, 1973.
-* [KZH2015] S. Kramer, D. Ziegenbein, and A. Hamann: Real World Automotive Benchmarks For Free, International Workshop on Analysis Tools and Methodologies for Embedded adn Real-Time Sysems (WATERS), 2015.[[Paper]](https://www.ecrts.org/forum/download/file.php?id=9&sid=efda71c95b6afdd240d72cc1e491bb8b)
## Acknowledgments
This activity has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement n° 780785).
diff --git a/_docs/concepts/client_library/execution_management/cbg-executor_sample_system.png b/_docs/concepts/client_library/execution_management/png/cbg-executor_sample_system.png
similarity index 100%
rename from _docs/concepts/client_library/execution_management/cbg-executor_sample_system.png
rename to _docs/concepts/client_library/execution_management/png/cbg-executor_sample_system.png
diff --git a/_docs/concepts/client_library/execution_management/cbg-executor_test-bench_architecture.png b/_docs/concepts/client_library/execution_management/png/cbg-executor_test-bench_architecture.png
similarity index 100%
rename from _docs/concepts/client_library/execution_management/cbg-executor_test-bench_architecture.png
rename to _docs/concepts/client_library/execution_management/png/cbg-executor_test-bench_architecture.png
diff --git a/_docs/concepts/client_library/execution_management/cbg-executor_test-bench_results.png b/_docs/concepts/client_library/execution_management/png/cbg-executor_test-bench_results.png
similarity index 100%
rename from _docs/concepts/client_library/execution_management/cbg-executor_test-bench_results.png
rename to _docs/concepts/client_library/execution_management/png/cbg-executor_test-bench_results.png
diff --git a/_docs/concepts/client_library/execution_management/cbg_executor_demo_plot.png b/_docs/concepts/client_library/execution_management/png/cbg_executor_demo_plot.png
similarity index 100%
rename from _docs/concepts/client_library/execution_management/cbg_executor_demo_plot.png
rename to _docs/concepts/client_library/execution_management/png/cbg_executor_demo_plot.png
diff --git a/_docs/concepts/client_library/execution_management/executor_class_diagram.png b/_docs/concepts/client_library/execution_management/png/executor_class_diagram.png
similarity index 100%
rename from _docs/concepts/client_library/execution_management/executor_class_diagram.png
rename to _docs/concepts/client_library/execution_management/png/executor_class_diagram.png
diff --git a/_docs/concepts/client_library/execution_management/executor_to_dds_sequence_diagram.png b/_docs/concepts/client_library/execution_management/png/executor_to_dds_sequence_diagram.png
similarity index 100%
rename from _docs/concepts/client_library/execution_management/executor_to_dds_sequence_diagram.png
rename to _docs/concepts/client_library/execution_management/png/executor_to_dds_sequence_diagram.png
diff --git a/_docs/concepts/client_library/execution_management/fawkes_executor_diagram.png b/_docs/concepts/client_library/execution_management/png/fawkes_executor_diagram.png
similarity index 100%
rename from _docs/concepts/client_library/execution_management/fawkes_executor_diagram.png
rename to _docs/concepts/client_library/execution_management/png/fawkes_executor_diagram.png
diff --git a/_docs/concepts/client_library/execution_management/meta-executor_concept.png b/_docs/concepts/client_library/execution_management/png/meta-executor_concept.png
similarity index 100%
rename from _docs/concepts/client_library/execution_management/meta-executor_concept.png
rename to _docs/concepts/client_library/execution_management/png/meta-executor_concept.png
diff --git a/_docs/concepts/client_library/execution_management/meta-executor_concept.pptx b/_docs/concepts/client_library/execution_management/png/meta-executor_concept.pptx
similarity index 100%
rename from _docs/concepts/client_library/execution_management/meta-executor_concept.pptx
rename to _docs/concepts/client_library/execution_management/png/meta-executor_concept.pptx
diff --git a/_docs/concepts/client_library/execution_management/ping_pong_diagram.png b/_docs/concepts/client_library/execution_management/png/ping_pong_diagram.png
similarity index 100%
rename from _docs/concepts/client_library/execution_management/ping_pong_diagram.png
rename to _docs/concepts/client_library/execution_management/png/ping_pong_diagram.png
diff --git a/_docs/concepts/client_library/execution_management/png/rclc_executor_multi_threaded.png b/_docs/concepts/client_library/execution_management/png/rclc_executor_multi_threaded.png
new file mode 100644
index 00000000..f28cbbef
Binary files /dev/null and b/_docs/concepts/client_library/execution_management/png/rclc_executor_multi_threaded.png differ
diff --git a/_docs/concepts/client_library/execution_management/png/rclc_executor_sequential_execution.png b/_docs/concepts/client_library/execution_management/png/rclc_executor_sequential_execution.png
new file mode 100644
index 00000000..90e9f37b
Binary files /dev/null and b/_docs/concepts/client_library/execution_management/png/rclc_executor_sequential_execution.png differ
diff --git a/_docs/concepts/client_library/execution_management/png/trigger_01.png b/_docs/concepts/client_library/execution_management/png/trigger_01.png
new file mode 100644
index 00000000..207e7e49
Binary files /dev/null and b/_docs/concepts/client_library/execution_management/png/trigger_01.png differ
diff --git a/_docs/concepts/client_library/execution_management/png/trigger_ALL.png b/_docs/concepts/client_library/execution_management/png/trigger_ALL.png
new file mode 100644
index 00000000..493d948a
Binary files /dev/null and b/_docs/concepts/client_library/execution_management/png/trigger_ALL.png differ
diff --git a/_docs/concepts/client_library/execution_management/png/trigger_ONE.png b/_docs/concepts/client_library/execution_management/png/trigger_ONE.png
new file mode 100644
index 00000000..ced1c7de
Binary files /dev/null and b/_docs/concepts/client_library/execution_management/png/trigger_ONE.png differ
diff --git a/_docs/concepts/client_library/execution_management/png/trigger_OR.png b/_docs/concepts/client_library/execution_management/png/trigger_OR.png
new file mode 100644
index 00000000..e1b4fa05
Binary files /dev/null and b/_docs/concepts/client_library/execution_management/png/trigger_OR.png differ
diff --git a/_docs/concepts/client_library/execution_management/png/trigger_user_defined.png b/_docs/concepts/client_library/execution_management/png/trigger_user_defined.png
new file mode 100644
index 00000000..3d020c89
Binary files /dev/null and b/_docs/concepts/client_library/execution_management/png/trigger_user_defined.png differ
+
+
-
-Figure 1: Fixed periodic preemptive scheduling
-
-To each process one or multiple tasks can be assigned, as shown in Figure 2. These tasks are executed sequentially, which is often called cooperative scheduling.
-
-
-
-Figure 2: Processes with sequentially executed tasks.
-
-While there are different ways to assign priorities to a given number of processes,
-the rate-monotonic scheduling assignment, in which processes with a shorter period have a higher priority, has been shown optimal if the processor utilization is less than 69% [LL1973](#LL1973).
-
- In the last decades many different scheduling approaches have been presented, however fixed-periodic preemptive scheduling is still widely used in embedded real-time systems [KZH2015](#KZH2015). This becomes also obvious, when looking at the features of current operating systems. Like Linux, real-time operating systems, such as NuttX, Zephyr, FreeRTOS, QNX etc., support fixed-periodic preemptive scheduling and the assignment of priorities, which makes the time-triggered paradigm the dominant design principle in this domain.
-
-However, data consistency is often an issue when preemptive scheduling is used and if data is being shared across multiple processes via global variables. Due to scheduling effects and varying execution times of processes, writing and reading these variables could occur sometimes sooner or later. This results in an latency jitter of update times (the timepoint at which a variable change becomes visible to other processes). Race conditions can occur when multiple processes access a variable at the same time. So solve this problem, the concept of logical-execution time (LET) was introduced in [HHK2001](#HHK2001), in which communication of data occurs only at pre-defined periodic time instances: Reading data only at the beginning of the period and writing data only at the end of the period. The cost of an additional latency delay is traded for data consistency and reduced jitter. This concept has also recently been applied to automotive applications [NSP2018](#NSP2018).
-
-
-
-Figure 3: Data communication without and with Logical Execution Time paradigm.
-
-An Example of the LET concept is shown in Figure 3. Assume that two processes are communicating data via one global variable. The timepoint when this data is written is at the end of the processing time. In the default case (left side), the process p3 and p4 receive the update. At the right side of the figure, the same scenario is shown with LET semantics. Here, the data is communicated only at period boundaries. In this case, the lower process communicates at the end of the period, so that always process p3 and p5 receive the new data.
-
-The described embedded use case relies on the following concepts:
-- periodic execution of processes
-- assignment of fixed priorities to processes
-- preemptive scheduling of processes
-- co-operative scheduling of tasks within a process (sequential execution)
-- data synchronization with LET-semantics
-
-While periodic activation is possible in ROS2 by using timers, preemptive scheduling is supported by the operating system and assigning priorities on the granularity of threads/processes that correspond to the ROS nodes; it is not possible to sequentially execute callbacks, which have no data-dependency. Furthermore data is read from the DDS queue just before the callback is executed and data is written sometime during the time the application is executed. While the `spin_period` function of the rclcpp-Executor allows to check for data at a fixed period and executing those callbacks for which data is available, however, with this spin-function does not execute all callbacks irrespective wheter data is available or not. So `spin_period` is not helpful to periodically execute a number of callbacks (aka tasks within a process). So we need a mechanism that triggers the execution of multiple callbacks (aka tasks) based on a timer. Data transmission is achieved via DDS which does not allow to implement a LET-semantics. To summarize, we derive the following requirements:
-
-Derived Requirements:
-- trigger the execution of multiple callbacks
-- sequential processing of callbacks
-- data synchronization with LET semantics
+## Analysis of processing patterns
+The development of an execution management mechanism for micro-ROS is based on an analysis of processing patterns commonly used in robotics and embedded systems. First, typical processing patterns in mobile robotics are presented which are used to implement deterministic behavior. Then, the processing patterns in the real-time embedded systems are analyzed, in which, as a main difference, the time-triggered paradigm is applied to accomplish real-time behavior.
-#### Sense-plan-act pipeline in robotics
+### Sense-plan-act pipeline in robotics
Now we describe common software design patterns which are used in mobile robotics to achieve deterministic behavior. For each design pattern we describe the concept and the derived requirements for a deterministic Executor.
-Concept:
-A common design paradigm in mobile robotics is a control loop, consisting of several phases: A sensing phase to aquire sensor data, a plan phase for localization and path planning and an actuation-phase to steer the mobile robot. Of course, more phases are possible, here these three phases shall serve as an example. Such a processing pipeline is shown in Figure 4.
+**Concept:**
-
+A common design paradigm in mobile robotics is a control loop, consisting of several phases: A sensing phase to aquire sensor data, a plan phase for localization and path planning and an actuation-phase to steer the mobile robot. Of course, more phases are possible, here these three phases shall serve as an example. Such a processing pipeline is shown in Figure 4.
+
-
+
-
+
-
+
-
+
+
+
+
+
+
+
+
+
+
+
+