Benchmarking is important. It’s one of the most important things a developer should do when evaluating different platforms, either from different vendors or from one vendor. However, for useful results to be realized, the proper questions must be asked. And that starts with which benchmarks are being run, as many are available.
I’m not suggesting that a systems integrator choose one benchmark over another. Developers must proceed with caution and use the appropriate tools to measure the specific functions that will be utilized in their application and not rely on any single tool.
Benchmarks serve a significant purpose in the development/system integration process but, unfortunately, are often misapplied or misused. In other words, they are not producing the same results that the system might exhibit when deployed in the field. A key factor for this “mis-guidance” is that the benchmark rarely takes the performance of the entire system into account. This is especially the case in embedded systems where I/O and timing requirements can vary greatly.
I’ll share an example. Take the Linux BogoMIPS as a benchmark tool. According to Wikipedia, the name comes from combining the terms bogus and MIPS, and it represents “a crude measurement of CPU speed made by the Linux kernel when it boots to calibrate an internal busy-loop. An often-quoted definition of the term is ‘the number of million times per second a processor can do absolutely nothing.’” Yikes!
Is it relevant? I guess there could be instances where it’s accurate, but I don’t think I would be betting my company’s reputation on that. Yet, I do see people using it.
The best developers employ multiple benchmarks to make comparisons of embedded systems. The more complicated the function that you require, the more careful you have to be when analyzing the results of the benchmarks.
Here is a sample of some of the tools we use at WINSYSTEMS to compare systems:
• Geekbench
• PTS
• Cyclictest
• Crystal Disk Mark
• IPERF
• Bonnie++
• Memtester
Keep in mind that, for the most part, these benchmarks are looking at overall board/system performance. If you’re looking to compare Arm-based systems versus those based on X86 CPUs, you’ve introduced another whole level of complexity—it’s just not an apples-to-apples comparison.
One single-board computer (SBC) I welcome you to put through the paces is the WINSYSTEMS SBC35-427. This 3.5-in. SBC is based on Intel’s Apollo Lake-I E3900 series processor. With its extensive expansion and configurability, including the ability to run three displays simultaneously, it suits just about any industrial IoT application. We’ve taken the board through all the benchmarks listed here but focused on very specific I/O timing differences between systems running a standard Linux Kernel and applying the Real-Time Linux PREEMPT_RT patches.
When it’s all said and done, benchmarks, in general, serve a very important purpose for comparing not only hardware, but software and operating-system differences, and microprocessors within a family or series. For example, it makes little difference whether you deploy a quad-core versus dual-core processor if the OS and application aren’t written in a framework to make use of the additional cores.
When conducting your own tests, you need to have a clear understanding of what performance characteristics are most important for your application. It could be raw throughput, deterministic (real-time) behavior, GPU performance, or something entirely different. Only you know what your specific needs are.