Post Quantum Signature Benchmark

From Embedded Lab Vienna for IoT & Security
Jump to navigation Jump to search

Post-quantum cryptography (PQC) develops cryptographic algorithms intended to resist attacks from both classical and quantum computers. This benchmark evaluates NIST-standardised and selected post-quantum digital signature algorithms on constrained devices, using liboqs, the Open Quantum Safe project's open-source C library.

Targets

Hardware Targets

This benchmark target-devices are a total of four, consisting of two Raspberry Pis running Ubuntu, a consumer smartphone running a terminal emulator called Termux, and a microcontroller programmed with RiotOS.

Termux is a terminal emulator and Linux environment for Android, it provides a minimal base system with a package manager and provides no root access.

RiotOS is an open source operating system for microcontrollers. It provides features such as cross compiling and flashing tools. It has real time capabilities and easier access to random number generation, while remaining energy and memory efficient.

Hardware Targets
Device Name RAM Chip Chip-Architecture OS
Raspberry Pi5 16 GB BCM2712 ARM-64 Ubuntu 25.10
Samsung S10e 6 GB Exynos 9820 ARM-64 Ubuntu 25.10 (via Termux)
Raspberry Pi3 B+ 1 GB BCM2837B0 ARM-64 Ubuntu 25.10
ESP-32 NodeMCU 520 KB ESP32-WROOM 32 32-bit LX6 RIOT 2026.07

Algorithm Targets

This benchmark ran a list of post-quantum signature algorithms, as well as one pre-quantum signature algorithm for comparison.

Algorithm Targets
Name Variants
ML-DSA -44, -65, -87
Falcon -512, -1024
SLH-DSA-SHA2 s -128s, -192s, -256s
SLH-DSA-SHA2 f -128f, -192f, -256f


ML-DSA is currently considered NIST's general purpose standard for quantum-safe signature algorithms. It is lattice-based and weighs speed and signature sizes in a balanced manner.

Falcon is similarly lattice-based, but also has compact keys and signatures.

SLH-DSA has many variants. The variants used in this benchmark are based on SHA2. The small signature variant (SLH-DSA-SHA2 s) is optimised for smaller signatures, which comes at the price of very slow signing times. The fast signing variant (SLH-DSA-SHA2 f) is instead optimised for faster signing times, at the cost of larger signatures.

NIST standardised ML-DSA and SLH-DSA in 2024. It also selected Falcon for future standardisation.

Results

All of the algorithms were run for a total of 1000 iterations with a message length of 1024 on each device. Each of these runs was preceded by 50 unmeasured warmup iterations.

ML-DSA

The different ML-DSA variants do not appear to show a structure in terms of how it scaled according to the algorithm size. While it almost always does scale in a way, where a larger version of the algorithm takes longer to finish an operation, there is no clear multiplicative factor that is shared across all devices used in this benchmark.

Overall, ML-DSA resulted in the most balanced results. This can be seen especially well on the RPi3B, where the verification time follows: 0.628ms (ML-DSA-44) --> 0.845 (ML-DSA-65) --> 1.018 (ML-DSA-87) In general, verification increases steadily with each variant. Signing also follows the same pattern, except on the RPi3B, where ML-DSA-87 was surprisingly faster than ML-DSA-65.

There were two unexpected results found that concerned ML-DSA. One of which is ML-DSA-87 having a shorter average signing time (2.161 ms) than ML-DSA-65 (2.328 ms) on the RPi3B. One of the reasons for this could be the difference in standard deviation between the two. ML-DSA-65 has a much higher standard deviation (1.457 ms) when compared to that of ML-DSA-87 (0.973 ms).

The other unexpected and considerably more grave result, is that ML-DSA-65 and ML-DSA-87 both consistently faced an Rmutex error after roughly 350 iterations when run on the ESP-32. This error specifically is a RIOT kkernel panic, which reports an Rmutex error before fully crashing. It is important to note here that the benchmark code does not use recursive mutex.

The cause has not been confirmed. One possible explanation is that the main thread of Riot OS exhausted its remaining stack, resulting in memory corrupting that later affected the mutex stat, which could be causing the mutex-counter issues. This requires further stack-usage measurements and debugging in order to fully distinguish this from a RIOT or liboqs error.

ML-DSA Results
ML-DSA-44 ML-DSA-65 ML-DSA-87
Public Key (B) 1312 1952 2592
Secret Key (B) 2560 4032 4896
Signature Avg. (B) 2420 3309 4627
RPi5 (ms) 0.070 / 0.199 / 0.071 0.120 / 0.307 / 0.116 0.202 / 0.409 / 0.197
S10e (ms) 0.070 / 0.266 / 0.075 0.131 / 0.404 / 0.115 0.185 / 0.510 / 0.188
RPi3B (ms) 0.640 / 1.814 / 0.628 0.884 / 2.328 / 0.845 1.054 / 2.161 / 1.018
ESP32 (ms) 321.289 / 669.470 / 325.121 566.117 / 1073.873 / 546.950 981.477 / 1527.113 / 945.991

Table Information: Key and signature sizes are in bytes. Each device has thre time measurements in milliseconds. The measurements are in the following order: average key-generation time, average signing time, average verification time

Falcon

The two variants of Falcon stay consistent with each other. During key-pair generation Falcon-1024 takes roughly three times as long, while during both signing and verification Falcon-512 is close to twice as fast. This is best seen in the Raspberry Pi5’s signing time, with it being 0.304 ms for Falcon-512, and 0.608ms for Falcon-1024.

The two versions of Falcon, Falcon-512 and Falcon-1024 had by far the worse key-pair generation time when compared to ML-DSA and ECDSA, with 37.7 ms and 103.4 ms on our Raspberry Pi 3 B+ respectively. They, however, boasted the best verification time, with Falcon-512 consistently beating ECDSA. Falcon-1024 also consistently beat ECDSA on all devices except for the Samsung S10e. It also performed well in signing time and has by far the smallest signature size out of all of the benchmarked quantum-safe algorithms. A similar pattern is visible in Falcon’s standard deviation being very consistent in verification and signing, and very inconsistent in its key generation time. For these reasons, it can be concluded that Falcon would make for a fitting option for consumer applications.

There are some anomalies in the data that need to be discussed. Falcon performed noticeably worse on the Samsung S10e, especially when compared to any other device. While the verification performed mostly as expected, both key-pair time and especially signing time underperformed. A potential, though unverified cause of this is either the chip architecture of the Exynos 9820 or Termux. Further testing would be required to say for certain.

Falcon Results
Falcon-512 Falcon-1024
Public Key (B) 897 1793
Secret Key (B) 1281 2305
Signature Avg. (B) 654 1270
RPi5 (ms) 9.605 / 0.304 / 0.058 27.805 / 0.608 / 0.109
S10e (ms) 16.307 / 4.562 / 0.067 50.040 / 10.364 / 0.135
RPi3B (ms) 37.712 / 1.189 / 0.225 103.425 / 2.429 / 0.420
ESP32 (ms) 11447.507 / 1097.989 / 70.613 23597.083 / 2353.382 / 112.524

Table Information: Key and signature sizes are in bytes. Each device has thre time measurements in milliseconds. The measurements are in the following order: average key-generation time, average signing time, average verification time

SLH-DSA-SHA2

SLH-DSA-SHA2 s

SLH-DSA-small performed by far the worst out of all the benchmarked algorithms. SLH-DSA-small was slower than both Falcon and ML-DSA in key-pair generation on all three Linux-based devices. It managed to beat its fast-signing counterpart in verification times but still falls well short of ML-DSA or Falcon. By far the worst outcome is the signing time, where SLH-DSA-192s was consistently more than 20 times slower than its fast counterpart who was itself up to over 100 times slower than ML-DSA and Falcon.

The supposed benefit it has over its fast counterpart of smaller signature sizes, put SLH-DSA-128s still well in last place with 7856 Bytes. This more than anything puts SLH-DSA out of question for any consumer application. At seven times the signature size relative to its message, SLH-DSA is not just impractical, but also uneconomical.

Once again there is some nuance to the data. While SLH-DSA-192s was noticeably slower than the 128 variant, the 256 variant consistently outperformed 192 in signing and even outperformed 128 in key generation. This result is consistent across all our devices, which ruled out many possible causes. Additionally, testing different message sizes did not change the outcome. Since SLH-DSA was recently implemented and has many variants, it is assumed this problem most likely originated within the implementation by liboqs.

SLH DSA s Results
SLH-DSA-SHA2-128s SLH-DSA-SHA2-192s SLH-DSA-SHA2-256s
Public Key (B) 32 48 64
Secret Key (B) 64 96 128
Signature Avg. (B) 7856 16224 29792
RPi5 (ms) 86.667 / 662.235 / 0.666 126.079 / 1178.281 / 0.980 83.879 / 1053.059 / 1.445
S10e (ms) 110.708 / 835.178 / 0.830 179.500 / 1664.802 / 1.374 89.906 / 1128.312 / 1.461
RPi3B (ms) 266.370 / 2044.750 / 2.073 381.977 / 3611.358 / 3.013 240.072 / 3079.630 / 4.177
ESP32 (ms) 19655.639 / 150246.329 / 150.383 29350.728 / 310347.801 / 274.589 19552.853 / 292324.862 / 397.303

Table Information: Key and signature sizes are in bytes. Each device has thre time measurements in milliseconds. The measurements are in the following order: average key-generation time, average signing time, average verification time

SLH-DSA-SHA2 f

SLH-DSA-fast while performing much better than its small counterpart in most regards, still underperformed greatly. It fails to compete with Falcon and ML-DSA in verification and signing. Together with its signature size, which is roughly double the size of SLH-DSA-small, it brings little to the table that would make it an attractive option.

The signing process is by far the most consistent and follows a rough pattern. A 1.6x jump from 128 to 192 and a doubling from 192 to 256. This is also the area where it outperformed the small version the most, saving an average of 630ms per iteration on the Raspberry Pi5 when running SLH-DSA-128f.

There are a lot of nuances in the data that lead to more questions than answers. For one, the same pattern that was missing in the small version’s verification results, where 256 outperforms 192, is once again present in the fast variants verification results, providing more evidence to the hypothesis that there are some issues in liboqs’ implementation.

Furthermore, in key-pair generation 192 performs much better than expected, almost reaching parity with 128 on the Samsung S10e. In all likelihood SLH-DSA-fast suffers from the same code problems as its small counterpart, it would be best to run this experiment again further developed.

SLH-DSA f Results
SLH-DSA-SHA2-128f SLH-DSA-SHA2-192f SLH-DSA-SHA2-256f
Public Key (B) 32 48 64
Secret Key (B) 64 96 128
Signature Avg. (B) 17088 35664 49856
RPi5 (ms) 1.358 / 31.790 / 1.921 1.975 / 52.177 / 2.804 5.213 / 107.236 / 2.858
S10e (ms) 2.210 / 39.074 / 2.826 2.292 / 60.408 / 3.248 5.326 / 109.321 / 2.924
RPi3B (ms) 4.278 / 100.426 / 6.057 6.245 / 165.952/ 8.864 14.861 / 308.838 / 8.213
ESP32 (ms) 307.186 / 7210.101 / 434.568 458.901 / 13246.498 / 710.424 N/A

Table Information: Key and signature sizes are in bytes. Each device has thre time measurements in milliseconds. The measurements are in the following order: average key-generation time, average signing time, average verification time

Set-Up Guide

Project Structure

The benchmark is split into two parts. The first part is the benchmarking code LinuxAlgs/PQAlgvar.c which outputs CSV files. To run the code on Linux-based systems, a "--alg" flag is required to specify which algorithm to test. The results will be written in CSV format into stdout. 

#Example

./LinuxAlgs/PQAlgvar --alg ML-DSA-44 > Benches/ML-DSA-44.csv

Running the benchmark code on microcontrollers is more involved and will be further described within the microcontroller set-up guide.

The second part of the benchmark is the code that runs the mathematical operations, namely Math/math_bench.c, or for microcontrollers Math/math_micro.c. The code takes CSV files as input and produces a human-readable report in .txt format. The code expects a CSV file as its first input. It also has an optional "--output" argument for its output file, which defaults to "benchmark_report.txt" if not specified. 

#Example

./Math/math_bench Benches/ML-DSA-44.csv --output Results/ML-DSA-44.txt

For the sake of simplicity, a small bash script called run_benches.sh was created, which will automatically make a complete run of all algorithms as well as writing the reports afterwards.

Linux Systems

For the initial set-up, the GitLab repository of this project contains a bash script named initiate.sh, which will complete the following steps.

The first step is the installation of dependencies and cloning the repository. 

sudo apt install -y git build-essential cmake ninja-build libssl-dev

git clone https://git.hcw.ac.at/c2410475078/pq-benchmarking.git

Afterwards, one must clean up liboqs, install the liboqs directory, and then build it as a static library. 

cd LinuxAlgs

rm -rf liboqs 

git clone https://github.com/open-quantum-safe/liboqs.git liboqs

cd liboqs 

mkdir build && cd build 
cmake -GNinja .. \ 

-DBUILD_SHARED_LIBS=OFF \

-DOQS_BUILD_ONLY_LIB=ON \

-DOQS_USE_OPENSSL=ON \

-DCMAKE_BUILD_TYPE=Release \

-DCMAKE_INSTALL_PREFIX=/usr/local 
ninja

sudo ninja install

sudo ldconfig

Finally, the binaries of the C programs need to be built: 

cd ~/pq-benchmarking

gcc ./LinuxAlgs/PQAlgvar.c -o ./LinuxAlgs/PQAlgvar \

-I/usr/local/include \

-L/usr/local/lib \

-loqs -lssl -lcrypto -lm

gcc ./Math/math_bench.c -o ./Math/math_bench -lm

Microcontrollers

The set-up for the microcontroller benchmark relies heavily on Riot OS. This benchmark uses an ESP-32, which is compatible with Riot OS. When using different microcontrollers, it is recommended that one checks for compatibility first. Additionally, some changes may need to be made at certain points in the set-up.

The first step is to install dependencies, clone all relevant repositories, as well as adding the Path to ~/.bashrc 

sudo apt install -y git build-essential cmake ninja-build libssl-dev

sudo apt install -y python3 python3-pip python3-venv python3-psutil

git clone https://git.hcw.ac.at/c2410475078/pq-benchmarking.git

git clone https://github.com/RIOT-OS/RIOT.git

git clone https://github.com/open-quantum-safe/liboqs.git pq-benchmarking/RiotOS/liboqs

. ./export.sh esp32

cd ~/pq-benchmarking/RiotOS

There are multiple instances within run.sh and the Makefile where absolute paths are in use and variables like the cross compiler and board ID are defined. If a different microcontroller is being used, these will require heavy modification. For example, if the user is not the administrator user, the following lines will need to be changed within run.sh: 

-DCMAKE_CXX_COMPILER=/home/admin/.espressif/tools/xtensa-esp-elf/esp-14.2.0_20241119/xtensa-esp-elf/bin/xtensa-esp32-elf-g++ \

-DCMAKE_AR=/home/admin/.espressif/tools/xtensa-esp-elf/esp-14.2.0_20241119/xtensa-esp-elf/bin/xtensa-esp32-elf-ar \

-DCMAKE_RANLIB=/home/admin/.espressif/tools/xtensa-esp-elf/esp-14.2.0_20241119/xtensa-esp-elf/bin/xtensa-esp32-elf-ranlib\

Running the Microcontroller Benchmark

The code for microcontrollers is highly optimised to save on memory. For this reason, every run of this benchmark requires a new minimal build of liboqs to be compiled. To simplify this process, run.sh was created. At the top of the file, the variable ALG_NAME dictates what algorithm is built and the name of the output file. It is important that the exact names listed in run.sh are used, because the minimal build of liboqs is case sensitive. Finally, the following lines need to be manually adjusted in the RiotPQALG.c file, depending on the chosen algorithm. 

#define BENCH_ALG_NAME OQS_SIG_alg_ml_dsa_44

#define BENCH_PUBLIC_KEY_MAX OQS_SIG_ml_dsa_44_length_public_key

#define BENCH_SECRET_KEY_MAX OQS_SIG_ml_dsa_44_length_secret_key

#define BENCH_SIGNATURE_MAX OQS_SIG_ml_dsa_44_length_signature

Further adjustments, such as number of iterations, number of warmup iterations, and message length can all be adjusted there as well.

Once both files have been properly configured, run.sh can be executed. A successful operation can be seen in the contents of the first CSV file. The output of these files can be found in "~/pq-benchmarking/RiotOS/Benches". Since the output is not clean CSV, the math function cannot use it. Before executing the math file, one must first clean it by running Benches/clean.sh. 

WARNING: clean.sh will break if used while the code is still running or if the following content is missing from the output:
# liboqs_version=0.15.0

Finally, the separate math function for microcontrollers needs to be built before it can output the report. 

cd ~/pq-benchmarking
gcc ./Math/math_micro.c -o ./Math/math_micro -lm
#Example:
./Math/math_micro RiotOS/Benches/SIG_ml_dsa_44.csv --output RiotOS/Results/ml_dsa_44.txt

References

Retrieved from "https://wiki.elvis.science/index.php?title=Post_Quantum_Signature_Benchmark&oldid=18276"