RT-Bench An Extensible Benchmark Framework for the

Pragmatic EEMBC [6] Popular Benchmarks Suites Full-Scale PapaBench [10] Synthetic RTEval [2] Pragmatic TACLeBench [7] MiBench [9] RT-Tests [1] SD-VBS [13] PARSEC [4] IsolBench [12] Mälardalen [8] Rodinia [5] SPLASH [3,11] 4<br>

Pragmatic EEMBC [6] Benchmarks Suites RT Features Full-Scale PapaBench [10] Synthetic RTEval [2] Pragmatic TACLeBench [7] MiBench [9] RT-Tests [1] SD-VBS [13] PARSEC [4] IsolBench [12] Mälardalen [8] Rodinia [5] SPLASH [3,11] 5<br>

RT-Bench: Motivations and Principles RT-Bench Implements RT features in a generic way.
Multiple benchmarks use the same implementation. All benchmarks have the same features. Adapt existing suites Time could be better spent
Bugs can be introduced.
Could be done multiple times.
Different implementations of the same features. Missing features Many suites have missing features for RT-Systems. Real-Time Abstracti-on Payloads are executed periodically.
Statistics are collected for each period. Common Interface All benchmarks report the same statistics.
Original output can be preserved. Extensibi-lity Adding benchmarks is easy.
Documentation is available.
Linux & POSIX.4 compliant. Compati-bility Improve existing benchmarks without altering their logic.
Only benchmark payload considered for stats. 6<br>

Application layer RT-Bench: Structure and Portability Hardware Layer Operating System Utils (optional) RT-Bench generator Overhead Test
Profiled WSS,WCET,…
Plots generation x86
ARM64 POSIX.4 compliant
Linux scheduler syscalls
Glibc Adds RT features 7<br>

RT-Bench generator … 8<br>

New period? Under the hood: RT-Bench generator Profiling/Monitoring Thread Initiali-
zation Exit No Iteration == n ? Yes Execution Yes Error Error Tear-down Measures
& log Captures
timestamp Period timer No Enable? Sample &
Wait Bmark. running? No Yes Yes No 9<br>

Extending RT-Bench Benchmark collection Split main logic into:
Initialization
Execution
Tear-down RT-Bench compatible benchmarks ~300 SLOCs for one SD-VBS benchmark 10<br>

Bare-Bones Use Case 11<br>

Bare-Bones Use Case: Demo 12<br>

Advanced Use Case 13<br>

Don’t be a stranger! RT-Bench is available at: https://gitlab.com/bastoni/rt-bench 14<br>

Thanks! My contacts:
mnico@bu.edu
https://cs-people.bu.edu/mnico<br>

References 1. RT-Test. https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/rt-tests
2. RTEval. https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/rteval
3. Splash2x benchmark suite. https://parsec.cs.princeton.edu/parsec3-doc.htm#splash2x
4. Bienia et al. 2008. The PARSEC benchmark suite: Characterization and architectural implications. In Proceedings of the 17th international conference on Parallel architectures and compilation techniques. 72–81.
5. Che et al. 2009. Rodinia: A benchmark suite for heterogeneous computing. In 2009 IEEE international symposium on workload characterization (IISWC). IEEE, 44–54.
6. Embedded Microprocessor Benchmark Consortium. EEMBC Benchmarks. https://www.eembc.org/products
7. Falk et al. 2016. TACLeBench: A Benchmark Collection to Support Worst-Case Execution Time Research. In 16th International Workshop on Worst-Case Execution Time Analysis (WCET 2016) (OpenAccess Series in Informatics (OASIcs), Vol. 55), Martin Schoeberl (Ed.). Schloss Dagstuhl–Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 2:1–2:10.
8. Gustafsson et al. 2010. The Mälardalen WCET benchmarks: Past, present and future. In 10th International Workshop on Worst-Case Execution Time Analysis (WCET 2010). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
9. Guthaus et al. 2001. MiBench: A free, commercially representative embedded benchmark suite. In Proceedings of the fourth annual IEEE international workshop on workload characterization. WWC-4 (Cat. No. 01EX538). IEEE, 3–14.
10. Nemeret al. 2006. Papabench: a free real-time benchmark. In 6th International Workshop on Worst-Case Execution Time Analysis (WCET’06). Schloss Dagstuhl-Leibniz-Zentrum für Informatik.
11. Sakalis et al. 2016. Splash-3: A properly synchronized benchmark suite for contemporary research. In 2016 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 101–111.
12. Valsan et al. 2016. Taming Non-Blocking Caches to Improve Isolation in Multicore Real-Time Systems. In 2016 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS). 1–12. https://doi.org/10.1109/RTAS.2016.7461361
13. Venkata et al. 2009. SD-VBS: The San Diego Vision Benchmark Suite. In 2009 IEEE International Symposium on Workload Characterization (IISWC). 55–64. https://doi.org/10.1109/IISWC.2009.5306794 16<br>