CI: Extend benchmark build-only tests #809

hanno-becker · 2025-02-25T19:13:44Z

No description provided.

oqs-bot

Intel Xeon 4th gen (c7i)

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`9626` cycles	`9634` cycles	`1.00`
`ML-KEM-512 encaps`	`11237` cycles	`11231` cycles	`1.00`
`ML-KEM-512 decaps`	`15375` cycles	`15304` cycles	`1.00`
`ML-KEM-768 keypair`	`16295` cycles	`16437` cycles	`0.99`
`ML-KEM-768 encaps`	`17867` cycles	`17987` cycles	`0.99`
`ML-KEM-768 decaps`	`23512` cycles	`23771` cycles	`0.99`
`ML-KEM-1024 keypair`	`22352` cycles	`22253` cycles	`1.00`
`ML-KEM-1024 encaps`	`24141` cycles	`24067` cycles	`1.00`
`ML-KEM-1024 decaps`	`31971` cycles	`31923` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Arm Cortex-A76 (Raspberry Pi 5) benchmarks

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`29564` cycles	`29537` cycles	`1.00`
`ML-KEM-512 encaps`	`35024` cycles	`35129` cycles	`1.00`
`ML-KEM-512 decaps`	`45692` cycles	`45749` cycles	`1.00`
`ML-KEM-768 keypair`	`50310` cycles	`50468` cycles	`1.00`
`ML-KEM-768 encaps`	`55992` cycles	`55831` cycles	`1.00`
`ML-KEM-768 decaps`	`70887` cycles	`70803` cycles	`1.00`
`ML-KEM-1024 keypair`	`73448` cycles	`73367` cycles	`1.00`
`ML-KEM-1024 encaps`	`82360` cycles	`82310` cycles	`1.00`
`ML-KEM-1024 decaps`	`102420` cycles	`102577` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 4th gen (c7i) (no-opt)

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`29401` cycles	`29311` cycles	`1.00`
`ML-KEM-512 encaps`	`34181` cycles	`34251` cycles	`1.00`
`ML-KEM-512 decaps`	`44368` cycles	`44426` cycles	`1.00`
`ML-KEM-768 keypair`	`47956` cycles	`47944` cycles	`1.00`
`ML-KEM-768 encaps`	`56307` cycles	`56239` cycles	`1.00`
`ML-KEM-768 decaps`	`68030` cycles	`67992` cycles	`1.00`
`ML-KEM-1024 keypair`	`72110` cycles	`72091` cycles	`1.00`
`ML-KEM-1024 encaps`	`84847` cycles	`84763` cycles	`1.00`
`ML-KEM-1024 decaps`	`101605` cycles	`101696` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 4th gen (c7a)

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`11649` cycles	`11651` cycles	`1.00`
`ML-KEM-512 encaps`	`13324` cycles	`13310` cycles	`1.00`
`ML-KEM-512 decaps`	`18230` cycles	`18205` cycles	`1.00`
`ML-KEM-768 keypair`	`20544` cycles	`20143` cycles	`1.02`
`ML-KEM-768 encaps`	`21192` cycles	`21215` cycles	`1.00`
`ML-KEM-768 decaps`	`28502` cycles	`28436` cycles	`1.00`
`ML-KEM-1024 keypair`	`26977` cycles	`26963` cycles	`1.00`
`ML-KEM-1024 encaps`	`29031` cycles	`29015` cycles	`1.00`
`ML-KEM-1024 decaps`	`38531` cycles	`38516` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 3rd gen (c6a)

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`17285` cycles	`17265` cycles	`1.00`
`ML-KEM-512 encaps`	`19057` cycles	`19192` cycles	`0.99`
`ML-KEM-512 decaps`	`24644` cycles	`24569` cycles	`1.00`
`ML-KEM-768 keypair`	`29414` cycles	`29666` cycles	`0.99`
`ML-KEM-768 encaps`	`30821` cycles	`30939` cycles	`1.00`
`ML-KEM-768 decaps`	`38609` cycles	`39133` cycles	`0.99`
`ML-KEM-1024 keypair`	`43619` cycles	`43715` cycles	`1.00`
`ML-KEM-1024 encaps`	`45132` cycles	`45100` cycles	`1.00`
`ML-KEM-1024 decaps`	`55590` cycles	`55595` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 3rd gen (c6i)

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`16164` cycles	`16166` cycles	`1.00`
`ML-KEM-512 encaps`	`18809` cycles	`18398` cycles	`1.02`
`ML-KEM-512 decaps`	`24996` cycles	`24950` cycles	`1.00`
`ML-KEM-768 keypair`	`27840` cycles	`27913` cycles	`1.00`
`ML-KEM-768 encaps`	`30555` cycles	`29608` cycles	`1.03`
`ML-KEM-768 decaps`	`38935` cycles	`38960` cycles	`1.00`
`ML-KEM-1024 keypair`	`37737` cycles	`38513` cycles	`0.98`
`ML-KEM-1024 encaps`	`40736` cycles	`40768` cycles	`1.00`
`ML-KEM-1024 decaps`	`53289` cycles	`53259` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Intel Xeon 3rd gen (c6i)'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.03.

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-768 encaps`	`30555` cycles	`29608` cycles	`1.03`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 4th gen (c7a) (no-opt)

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`36456` cycles	`36390` cycles	`1.00`
`ML-KEM-512 encaps`	`42974` cycles	`42933` cycles	`1.00`
`ML-KEM-512 decaps`	`55891` cycles	`55856` cycles	`1.00`
`ML-KEM-768 keypair`	`59126` cycles	`59064` cycles	`1.00`
`ML-KEM-768 encaps`	`68087` cycles	`67576` cycles	`1.01`
`ML-KEM-768 decaps`	`84518` cycles	`84510` cycles	`1.00`
`ML-KEM-1024 keypair`	`87401` cycles	`87392` cycles	`1.00`
`ML-KEM-1024 encaps`	`98304` cycles	`98321` cycles	`1.00`
`ML-KEM-1024 decaps`	`119703` cycles	`119620` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 3rd gen (c6a) (no-opt)

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`39708` cycles	`39819` cycles	`1.00`
`ML-KEM-512 encaps`	`48041` cycles	`48167` cycles	`1.00`
`ML-KEM-512 decaps`	`62142` cycles	`62398` cycles	`1.00`
`ML-KEM-768 keypair`	`64505` cycles	`64756` cycles	`1.00`
`ML-KEM-768 encaps`	`76225` cycles	`75800` cycles	`1.01`
`ML-KEM-768 decaps`	`94441` cycles	`94482` cycles	`1.00`
`ML-KEM-1024 keypair`	`96375` cycles	`95934` cycles	`1.00`
`ML-KEM-1024 encaps`	`109366` cycles	`109397` cycles	`1.00`
`ML-KEM-1024 decaps`	`132860` cycles	`132969` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 3rd gen (c6i) (no-opt)

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`46830` cycles	`46850` cycles	`1.00`
`ML-KEM-512 encaps`	`55463` cycles	`55434` cycles	`1.00`
`ML-KEM-512 decaps`	`71281` cycles	`71309` cycles	`1.00`
`ML-KEM-768 keypair`	`76202` cycles	`76202` cycles	`1`
`ML-KEM-768 encaps`	`87519` cycles	`87519` cycles	`1`
`ML-KEM-768 decaps`	`108400` cycles	`108142` cycles	`1.00`
`ML-KEM-1024 keypair`	`112424` cycles	`112419` cycles	`1.00`
`ML-KEM-1024 encaps`	`126605` cycles	`126474` cycles	`1.00`
`ML-KEM-1024 decaps`	`152864` cycles	`152708` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton2

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`29565` cycles	`29569` cycles	`1.00`
`ML-KEM-512 encaps`	`35018` cycles	`35065` cycles	`1.00`
`ML-KEM-512 decaps`	`45692` cycles	`45784` cycles	`1.00`
`ML-KEM-768 keypair`	`50321` cycles	`50417` cycles	`1.00`
`ML-KEM-768 encaps`	`56005` cycles	`55932` cycles	`1.00`
`ML-KEM-768 decaps`	`70913` cycles	`70859` cycles	`1.00`
`ML-KEM-1024 keypair`	`73473` cycles	`73382` cycles	`1.00`
`ML-KEM-1024 encaps`	`82396` cycles	`82321` cycles	`1.00`
`ML-KEM-1024 decaps`	`102456` cycles	`102596` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton4

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`18054` cycles	`18050` cycles	`1.00`
`ML-KEM-512 encaps`	`21437` cycles	`21418` cycles	`1.00`
`ML-KEM-512 decaps`	`28128` cycles	`28124` cycles	`1.00`
`ML-KEM-768 keypair`	`31015` cycles	`31074` cycles	`1.00`
`ML-KEM-768 encaps`	`34146` cycles	`34152` cycles	`1.00`
`ML-KEM-768 decaps`	`43742` cycles	`43783` cycles	`1.00`
`ML-KEM-1024 keypair`	`44831` cycles	`44910` cycles	`1.00`
`ML-KEM-1024 encaps`	`50374` cycles	`50329` cycles	`1.00`
`ML-KEM-1024 decaps`	`63369` cycles	`63267` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton3

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`19185` cycles	`19156` cycles	`1.00`
`ML-KEM-512 encaps`	`22897` cycles	`22937` cycles	`1.00`
`ML-KEM-512 decaps`	`30215` cycles	`30239` cycles	`1.00`
`ML-KEM-768 keypair`	`32785` cycles	`32832` cycles	`1.00`
`ML-KEM-768 encaps`	`36516` cycles	`36514` cycles	`1.00`
`ML-KEM-768 decaps`	`46899` cycles	`46950` cycles	`1.00`
`ML-KEM-1024 keypair`	`47458` cycles	`47385` cycles	`1.00`
`ML-KEM-1024 encaps`	`53418` cycles	`53362` cycles	`1.00`
`ML-KEM-1024 decaps`	`67345` cycles	`67315` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton4 (no-opt)

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`35773` cycles	`35739` cycles	`1.00`
`ML-KEM-512 encaps`	`41553` cycles	`40725` cycles	`1.02`
`ML-KEM-512 decaps`	`52077` cycles	`52087` cycles	`1.00`
`ML-KEM-768 keypair`	`59665` cycles	`60568` cycles	`0.99`
`ML-KEM-768 encaps`	`66877` cycles	`67490` cycles	`0.99`
`ML-KEM-768 decaps`	`81479` cycles	`81160` cycles	`1.00`
`ML-KEM-1024 keypair`	`88782` cycles	`88817` cycles	`1.00`
`ML-KEM-1024 encaps`	`98758` cycles	`98809` cycles	`1.00`
`ML-KEM-1024 decaps`	`117417` cycles	`117445` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton2 (no-opt)

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`59614` cycles	`59718` cycles	`1.00`
`ML-KEM-512 encaps`	`68291` cycles	`68346` cycles	`1.00`
`ML-KEM-512 decaps`	`87007` cycles	`87157` cycles	`1.00`
`ML-KEM-768 keypair`	`99392` cycles	`99068` cycles	`1.00`
`ML-KEM-768 encaps`	`111318` cycles	`110818` cycles	`1.00`
`ML-KEM-768 decaps`	`134839` cycles	`135230` cycles	`1.00`
`ML-KEM-1024 keypair`	`149037` cycles	`149117` cycles	`1.00`
`ML-KEM-1024 encaps`	`164325` cycles	`164613` cycles	`1.00`
`ML-KEM-1024 decaps`	`195791` cycles	`195986` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton3 (no-opt)

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`39097` cycles	`39112` cycles	`1.00`
`ML-KEM-512 encaps`	`44886` cycles	`44880` cycles	`1.00`
`ML-KEM-512 decaps`	`56728` cycles	`56751` cycles	`1.00`
`ML-KEM-768 keypair`	`64400` cycles	`64458` cycles	`1.00`
`ML-KEM-768 encaps`	`71923` cycles	`72632` cycles	`0.99`
`ML-KEM-768 decaps`	`87960` cycles	`87895` cycles	`1.00`
`ML-KEM-1024 keypair`	`96184` cycles	`96097` cycles	`1.00`
`ML-KEM-1024 encaps`	`106248` cycles	`106159` cycles	`1.00`
`ML-KEM-1024 decaps`	`126780` cycles	`127059` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

SpacemiT K1 8 (Banana Pi F3) benchmarks

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`225742` cycles	`225723` cycles	`1.00`
`ML-KEM-512 encaps`	`272681` cycles	`272700` cycles	`1.00`
`ML-KEM-512 decaps`	`347839` cycles	`347779` cycles	`1.00`
`ML-KEM-768 keypair`	`373193` cycles	`373204` cycles	`1.00`
`ML-KEM-768 encaps`	`435210` cycles	`435174` cycles	`1.00`
`ML-KEM-768 decaps`	`533585` cycles	`533524` cycles	`1.00`
`ML-KEM-1024 keypair`	`555151` cycles	`555056` cycles	`1.00`
`ML-KEM-1024 encaps`	`636060` cycles	`635975` cycles	`1.00`
`ML-KEM-1024 decaps`	`758205` cycles	`758176` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Arm Cortex-A55 (Snapdragon 888) benchmarks

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`59493` cycles	`59493` cycles	`1`
`ML-KEM-512 encaps`	`67135` cycles	`67135` cycles	`1`
`ML-KEM-512 decaps`	`86422` cycles	`86472` cycles	`1.00`
`ML-KEM-768 keypair`	`101371` cycles	`101269` cycles	`1.00`
`ML-KEM-768 encaps`	`112360` cycles	`112419` cycles	`1.00`
`ML-KEM-768 decaps`	`139419` cycles	`139318` cycles	`1.00`
`ML-KEM-1024 keypair`	`153306` cycles	`153571` cycles	`1.00`
`ML-KEM-1024 encaps`	`170669` cycles	`170749` cycles	`1.00`
`ML-KEM-1024 decaps`	`207001` cycles	`206829` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Arm Cortex-A72 (Raspberry Pi 4) benchmarks

Benchmark suite	Current: `cf375e2`	Previous: `50beac5`	Ratio
`ML-KEM-512 keypair`	`53631` cycles	`53117` cycles	`1.01`
`ML-KEM-512 encaps`	`62050` cycles	`61858` cycles	`1.00`
`ML-KEM-512 decaps`	`79036` cycles	`78640` cycles	`1.01`
`ML-KEM-768 keypair`	`90181` cycles	`91091` cycles	`0.99`
`ML-KEM-768 encaps`	`98601` cycles	`99064` cycles	`1.00`
`ML-KEM-768 decaps`	`122756` cycles	`122706` cycles	`1.00`
`ML-KEM-1024 keypair`	`136648` cycles	`135563` cycles	`1.01`
`ML-KEM-1024 encaps`	`149494` cycles	`149163` cycles	`1.00`
`ML-KEM-1024 decaps`	`182807` cycles	`181334` cycles	`1.01`

This comment was automatically generated by workflow using github-action-benchmark.

This commit splits the `quickcheck` job in `ci.yml` into `quickcheck` and `quickcheck_bench`, where `quickcheck` does a simple build + test, and `quickcheck_bench` ensures that the benchmarking binaries at least build. On MacOS, we only test -c M1, while on Ubuntu, we test PERF and PMU. Signed-off-by: Hanno Becker <beckphan@amazon.co.uk>

hanno-becker added the benchmark this PR should be benchmarked in CI label Feb 25, 2025