Post Quantum Signature Benchmark

From Embedded Lab Vienna for IoT & Security
Jump to navigation Jump to search

Report about PQ-signature benchmark on constrained devices

Targets

Hardware Targets

This benchmark target-devices are a total of four, consisting of two Raspberry Pis running Ubuntu, a consumer smartphone running a terminal emulator called Termux, and a microcontroller programmed with RiotOS.

Termux is a terminal emulator and Linux environment for Android, it provides a minimal base system with a package manager and provides no root access.

RiotOS is an open source operating system for microcontrollers. It provides features such as cross compiling and flashing tools. It has real time capabilities and easier access to random number generation, while remaining energy and memory efficient.

Hardware Targets
Device Name RAM Chip Chip-Architecture OS
Raspberry Pi5 16GB BCM2712 ARM-64 Ubuntu 25.10
Samsung S10e 6GB Exynos 9820 ARM-64 Ubuntu 25.10
Raspberry Pi3 B+ 1GB BCM2837B0 ARM-64 Ubuntu 25.10
ESP-32 NodeMCU 520KB ESP32-WROOM 32 32-bit LX6 RIOT 2026.07


Algorithm Targets

This benchmark ran a list of post-quantum signature algorithms, as well as one pre-quantum signature algorithm for comparison.

Algorithm Targets
Name Variants
ML-DSA -44, -65, -87
Falcon -512, -1024
SLH-DSA-SHA2 s -128s, -196s, -256s
SLH-DSA-SHA2 f -128f, -196f, -256f


ML-DSA is currently considered NIST's primary standard for quantum safe signature algorithms. It is lattice-based and weighs speed and signature sizes in a balanced manner.

Falcon is similarly lattice-based, but also has compact keys and signatures.

SLH-DSA has many variants. The variants used in this benchmark are based on SHA2. The small signature variant (SLH-DSA-SHA2 s) is optimised for smaller signatures, which comes at the price of very slow signing times. The fast signing variant (SLH-DSA-SHA2 f) is instead optimised for faster signing times, at the cost of larger signatures.

Results

ML-DSA

The different ML-DSA variants do not appear to show a structure in terms of how it scaled according to the algorithm size. While it almost always does scale in a way, where a larger version of the algorithm takes longer to finish an operation, there is no clear multiplicative factor that is shared across all devices used in this benchmark.

Overall, ML-DSA resulted in the most balanced results. This can be seen especially well on the RPi3B, where the key generation follows: 0.628ms (ML-DSA-44) --> 0.845 (ML-DSA-65) --> 1.018 (ML-DSA-87) In general, both the signing and verification follow the same pattern of a steady climb.

There were two unexpected results found that concerned ML-DSA. One of which is ML-DSA-87 having a shorter average signing time (1.814 ms) than ML-DSA-65 (2.328 ms). One of the reasons for this could be the difference in standard deviation between the two. ML-DSA-65 has a much higher standard deviation (1.457 ms) when compared to that of ML-DSA-87 (0.973 ms).

The other unexpected and considerably more grave result, is that ML-DSA-65 and ML-DSA-87 both consistently faced an Rmutex error after roughly 350 iterations when run on the ESP-32. This error specifically is a Riot-kernel panic, which reports an Rmutex error before fully crashing. It is important to note here that the benchmark code does not use recursive mutex.

While not confirmed whether this error was caused by RiotOS or a fault of liboqs, it is assumed that the main thread of RiotOS is running out of stack memory, causing temporary values to overwrite neighbouring memory, which could be causing the mutex-counter issues.

Falcon

The two variants of Falcon stay consistent with each other. During key-pair generation Falcon-1024 takes roughly three times as long, while during both signing and verification Falcon-512 is close to twice as fast. This is best seen in the Raspberry Pi5’s signing time, with it being 0.304 ms for Falcon-512, and 0.608ms for Falcon-1024.

The two versions of Falcon, Falcon-512 and Falcon-1024 had by far the worst Key-pair generation time of all algorithms, with 37.7 ms and 103.4 ms on our Raspberry Pi 3 B+ respectively. They, however, boasted the best verification time, even consistently beating ECDSA. It also performed well in singing time and has by far the smallest signatures size out of all of the benchmarked quantum safe algorithms. A similar pattern is visible in Falcon’s standard deviation being very consistent in verification and signing, and very inconsistent in its key generation time. For these reasons, it can be concluded that Falcon would make for a fitting option for consumer applications.

There are some anomalies in the data that need to be discussed. Falcon performed noticeably worse on the Samsung S10e, especially when compared to any other device. While the verification performed mostly as expected, both key-pair time and especially signing time underperformed. The most likely cause of this is either the chip architecture of the Exynos 9820 or Termux. further testing would be required to say for certain.

SLH-DSA-SHA2

SLH-DSA-SHA2 s

SLH-DSA-small performed by far the worst out of all the benchmarked algorithms. While it beat Falcon in key-pair generation time, it was still several times slower than all other algorithms. It managed to beat its fast-signing counterpart in verification times but still falls well short of ML-DSA or Falcon. By far the worst outcome is the signing time, where SLH-DSA-192s was consistently more than 20 times slower than its fast counterpart who was itself up to over 100 times slower than ML-DSA and Falcon.

The supposed benefit it has over its fast counterpart of smaller signature sizes, put SLH-DSA-128s still well in last place with 7856 Bytes. This more than anything puts SLH-DSA out of question for any consumer application. At seven times the signature size relative to its message, SLH-DSA is not just unpractical, but also uneconomical.

Once again there is some nuance to the data. While SLH-DSA-192s was noticeably slower than the 128 variant, the 256 variant consistently outperformed 192 in signing and even outperformed 128 in key generation. This result is consistent across all our devices, which ruled out many possible causes. Additionally, using different message sizes was tried to no change in the outcome. Since SLH-DSA was recently implemented and has many variants, it is assumed this problem most likely originated within the implementation by liboqs.


SLH-DSA-SHA2 f

SLH-DSA-fast while performing much better than its small counterpart in most regards, still underperformed greatly. It fails to compete with Falcon and ML-DSA in verification and signing. Together with its signature size, which is double the size of SLH-DSA-small, it brings little to the table that would make it an attractive option.

The signing process is by far the most consistent and follows a rough pattern. A 1.6x jump from 128 to 192 and a doubling from 192 to 256. This is also the area where it outperformed the small version the most, saving an average of 631ms per iteration on the Raspberry Pi5.

There are a lot of nuances in the data that lead to more questions than answers. For one, the same pattern that was missing in the small version’s verification results, where 256 outperforms 192, is once again present in the fast variants verification results, providing more evidence to the hypothesis that there are some issues in liboqs’ implementation.

Furthermore, in key-pair generation 192 performs much better than expected, almost reaching parity with 128 on the Samsung S10e. In all likelihood SLH-DSA-fast suffers from the same code problems as its small counterpart, it would be best to run this experiment again if and when SLH-DSA is more standardized and further developed.

Results Table

The following is the table of results from this benchmark, shortened down for this Wiki-entry. The full version can be found on the GitLab.

Set-Up Guide

Linux Systems

da guide here

Microcontrollers

da guide also here

References