Skip to content

Commit f18c86d

Browse files
FranFinFranFin
committed
Grouping memory profiling and benchmarking under the same section.
1 parent eaa8cdc commit f18c86d

10 files changed

Lines changed: 174 additions & 8 deletions

File tree

‎_data/docs.yml‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,7 @@
3737
docs:
3838
- concepts/benchmarking/concept
3939
- concepts/benchmarking/results
40-
41-
- title: Memory profiling
42-
docs:
43-
- concepts/memo_prof
40+
- concepts/benchmarking/memo_prof
4441

4542
- title: Core Tutorials
4643
docs:

‎_docs/concepts/benchmarking/benchmarking/index.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
---
2+
title: Benchmarking
3+
redirect_from: /benchmarking/
4+
permalink: /docs/concepts/benchmarking/benchmarking/
5+
---
6+
7+
## Table of contents
8+
9+
* [Introduction to Benchmarking](#introduction-to-benchmarking)
10+
* [Our benchmarking tool framework](#our-benchmarking-tool-framework)
11+
* [Trace Framework Abstraction](#trace-framework-abstraction)
12+
* [Shadow Builder](#shadow-builder)
13+
* [Binary generation for instrumented code](#binary-generation-for-instrumented-code)
14+
* [Receiving inputs](#receiving-inputs)
15+
* [Parse and Check](#parse-and-check)
16+
* [TFA Execution](#tfa-execution)
17+
* [Compilation](#compilation)
18+
* [Step to start benchmarking](#step-to-start-benchmarking)
19+
20+
21+
22+
## Introduction to Benchmarking
23+
24+
Developing working and stable application from the scribbles to the final
25+
executing binary is long and hard tasks. During this process developers may come
26+
across stabilities issues, perfomances issues. In addition to these issues, some
27+
specified QoS might be difficult to quantify. Solving those problems without the
28+
proper tools might be frustrating, tedious tasks leading to reduce developers
29+
efficiency. An adapted benchmarking tool could overcome all those development
30+
obstacles and increase development time. There are different KPI (Keep
31+
Performance Indicators) that one might be interested into. In the framework of
32+
this micro-ROS, the KPI can be freely chosen by the developer. In this way, the
33+
benchmarking tool will remain flexible and allow the community to constantly add
34+
some support for a lot of different KPI.
35+
36+
The problems we want to tackle are:
37+
38+
* Out there, many benchmarking tools exist. Each of targeting different KPIs.
39+
* Different platforms (Linux/Nuttx/Baremetal et.c.).
40+
* Too few time/resources to code benchmarking tool for each.
41+
* Avoid code overhead: Keep code clarity.
42+
* Avoid execution overhead: Do not want to make execution slower when benchmarking.
43+
44+
## Our Benchmarking tool framework
45+
46+
The benchmarking tool under development is providing a framework to allow
47+
developers to create their own benchmarking tool. Each part a developer wants to
48+
benchmark can be added as a plugin using the provided framework. In this way
49+
plugins can be shared and this improves re-usability as much as possible.
50+
51+
52+
## Trace Framework Abstraction
53+
54+
The Shadow builder alone only parse comments from the application and pass it
55+
along to the Trace Framework Abstraction (TFA) Core. The TFA core is aware of
56+
the plugins that are available, all the plugins’ capabilities and platform
57+
target. The process goes as explained below:
58+
59+
* The line containing the functionality Benchmarking::XX::YY will be checked
60+
against all the available plugins.
61+
* Plugins that are capable of handling functionality will respond with a piece of
62+
code that will be replaced with a piece of code.
63+
* Then the output file will be added in a folder corresponding to the platform
64+
type and benchmarking type.
65+
66+
Being generic is the key for this benchmarking tool. The plugins will in
67+
contrary bring the specific implementation needed to benchmark a specific
68+
platform. Every plugin will provide information as requested by the parser:
69+
70+
* Provide a list of supported platforms.
71+
* Provide a list of functions that are handled.
72+
* Provide snippets codes that will be added for benchmarking.
73+
* Provide a list of patches and/or patch code
74+
* Optional provide an end script to run and execute the benchmarks
75+
76+
77+
## Shadow Builder
78+
79+
This section will introduce some concept related to the shadow builder (SB).
80+
81+
The Shadow builder is a tool that will transparently instrument the code to
82+
benchmark. The tools will be able to output an “instrumented code” that will be
83+
later be compiled as a normal code. The following steps describe what the shadow
84+
builder process flow:
85+
86+
* Get configuration file from the user (Benchmarking Configuration File).
87+
* Get appropriate sources.
88+
* Execute Trace Framework Abstraction Configuration file.
89+
* Parse the sources file needed Injecting code.
90+
* Compile the targeted binary for different platform.
91+
* If needed, depending what type benchmark is undertaken, compile another
92+
target binary benchmarking.
93+
94+
The SB (Shadow Builder) is meant to be as transparent as possible for the user.
95+
And if the benchmarking is not activated, it should be bypassed. Get
96+
appropriate sources
97+
98+
The SB is in charge of getting the path/git repository to the source code that
99+
needs to be benchmarking. The benchmarking. The sources are specified by the
100+
user in the benchmarking configuration file. Injecting code
101+
102+
In order to inject code, there are some tools that allow this. CLang AST tool
103+
will allow to inject some code.
104+
105+
106+
## Binary generation for instrumented code
107+
108+
The binary generation is the process of compiling the source code. In order to
109+
benchmark, previously to compile the source code, it is necessary to instrument
110+
the code. The code will be instrumented in a transparent way for the
111+
programmer/user. Therefore, a configuration file provided by the programmer will
112+
be parsed and code injected as described in a configuration file.
113+
114+
### Receiving inputs
115+
116+
The binary generation's pipeline receives two inputs to work with:
117+
* Configuration Benchmarking file.
118+
* Source code to benchmark.
119+
120+
In short, the configuration describes:
121+
122+
* What is benchmarked (sources).
123+
* Where to benchmark.
124+
* What type of benchmark.
125+
* Optionally against what base line to compare (base line source)
126+
127+
### Parse and Check
128+
129+
Once the input received the **Shadow Builder** parses the configuration
130+
file. From the configuration file, the Shadow builder gets:
131+
132+
* The different benchmarking to be achieved.
133+
* The targeted platforms.
134+
135+
In addition to parsing, the Shadow Builder is in charge of checking
136+
capabilities and consistency within the configuration file and the different
137+
TFA's plugins registered in the TFA module.
138+
139+
### TFA Execution
140+
141+
Once parsed and checked against the TFA module capabilities, the Shadow
142+
Builder will be in charge of translating configuration into source code. The
143+
translated sources will also be achieved in cooperation with the TFA module. The
144+
detailed steps of the TFA can be found here. At the end of this step, the TFA
145+
will generate the new forged source code ready for compilation. In addition to
146+
patched source code, the TFA will generate scripts that will the benchmarks.
147+
148+
### Compilation
149+
150+
The compilation will happen for every kind of benchmarks and
151+
platforms targeted. Depending on the kind of benchmark that is being executed,
152+
there will be one or more binaries per benchmarks session. The number of binary
153+
generated also depends on what plugins are provided by the user to the shadow
154+
builder. The shadow builder will retrieve capabilities of the plugins and
155+
request from the developer, match them and generated software according to the
156+
matches.
157+
158+
159+
## Step to start benchmarking
160+
161+
The shadow Builder will be executed as follow:
162+
163+
* Software sources are passed to the Shadow Builder.
164+
* The source are passed and upon comments containing /*Benchmarking::XX::YY*/
165+
(a tag) the code line is passed to the Trace Framework Abstraction module.
166+
Using comments is preferable → No includes needed.
167+
* All plugins that registered to the TFA the Benchmarking::XX::YY functionality
168+
will return a piece of code that will be added to the source.
169+
* Once all parsed, the shadow builder will compile for all the different
170+
platforms requested either by plugins or by user configuration.
171+

_docs/concepts/memo_prof/3mems.png renamed to _docs/concepts/benchmarking/memo_prof/3mems.png

File renamed without changes.

_docs/concepts/memo_prof/index.md renamed to _docs/concepts/benchmarking/memo_prof/index.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
---
22
title: Memory profiling
3-
permalink: /docs/concepts/memo_prof/
3+
redirect_from: /memo_prof/
4+
permalink: /docs/concepts/benchmarking/memo_prof/
45
---
56

67
## Abstract

_docs/concepts/memo_prof/overall.png renamed to _docs/concepts/benchmarking/memo_prof/overall.png

File renamed without changes.

_docs/concepts/memo_prof/pubsub_diagram.png renamed to _docs/concepts/benchmarking/memo_prof/pubsub_diagram.png

File renamed without changes.

_docs/concepts/memo_prof/rmw_history.png renamed to _docs/concepts/benchmarking/memo_prof/rmw_history.png

File renamed without changes.

_docs/concepts/memo_prof/servcli.png renamed to _docs/concepts/benchmarking/memo_prof/servcli.png

File renamed without changes.

_docs/concepts/memo_prof/services_diagram.png renamed to _docs/concepts/benchmarking/memo_prof/services_diagram.png

File renamed without changes.

‎_includes/docs_nav.html‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,6 @@
1616
{% if page.path contains "_docs/concepts" and section.title == "Benchmarking" %}
1717
{% assign should_show_this_menu = true %}
1818
{% endif %}
19-
{% if page.path contains "_docs/concepts" and section.title == "Memory profiling" %}
20-
{% assign should_show_this_menu = true %}
21-
{% endif %}
2219
{% if page.path contains "_docs/tutorials" and section.title contains "Tutorials" %}
2320
{% assign should_show_this_menu = true %}
2421
{% endif %}

0 commit comments

Comments
 (0)