SHA3 and SHAKE - New API Design

[manastasova]

Issues:

Resolves #CryptoAlg-2810

Description of changes:

AWS-LC supports SHA3 and SHAKE algorithms though low level SHA3_Init, SHA3_Update, SHA3_Final and SHAKE_init, SHAKE_Final APIs. Currently, there are two issues with the implementation and usage of SHA3 and SHAKE:

• There is no support for SHAKE_Update function. SHAKE is implemented by calling SHAKE_Init, SHA3_Update and SHAKE_Final.
• SHAKE_Final allows multiple consecutive calls to enable incremental XOF output generation.

This PR addresses both of them as follows:

• Introduce new API layers - FIPS202, SHA3 and SHAKE.

  • Keccak1600 layer (Keccak1600_Squeeze/Absorb Layer (rename) #2097) implements KeccakF1600 Absorb and Squeeze functions; Keccak1600 layer does not manage internal input/output buffers.
  • FIPS202 layer implements Reset, Init, Update, and Finalize functionalities; FIPS202 layer manages the internal input/output buffers, allowing incremental requests (not necessarily multiple of block size) to Update (Absorb) and Squeeze for input/output processing. (Other functionalities, such as zero-ing of bitstate, block size checks, etc. are also handled by FIPS202 API layer).
    • FIPS202 layer implements all common behavior between SHA3 and SHAKE algorithms.
    • FIPS202 layer checks/updates the |ctx->state| flag when handling a common behavior between SHA3 and SHAKE algorithms. |ctx->state| is updated in the higher level SHA3_ SHAKE_ API layer when the behavior of both algorithms diverges (SHAKE can allow incremental squeezes).
  • SHA3 layer implements Init, Update, and Final functionalities; SHA3 layer only implements SHA3 algorithm, thus, offers a single-call SHA3_Final function. SHA3_Final will update internal |ctx->state| flag to prevent any sequential calls.
  • SHAKE layer implements XOF SHAKE algorithm, therefore, offers Init, Absorb, Squeeze, and Final functionalities;
    • SHAKE layer implements Init, and Absorb, Squeeze with incremental call support for absorb (byte-wise) and squeeze (block-wise).
    • SHAKE layer implements a single-call SHAKE_Final function that generates an arbitrary length output and finalizes SHAKE. Incremental XOF output generation is handled by |SHAKE_Squeeze|. |SHAKE_Squeeze| can be called multiple times. SHAKE_Final should be called only once.

• KECCAK600_CTX struct updates:

  • Remove |padded| field
  • Introduce |state| field
    • |state| can be |KECCAK1600_STATE_ABSORB|, |KECCAK1600_STATE_SQUEEZE|, |KECCAK1600_STATE_FINAL|
    • |KECCAK1600_STATE_ABSORB| - allows incremental absorbs until the state is changed
    • |KECCAK1600_STATE_SQUEEZE| - allows incremental squeezes for |SHAKE_Squeeze|
    • |KECCAK1600_STATE_Final| - prevents from incremental squeezes via |SHAKE_Final| and prevents from consecutive calls to |SHA3_Final| (Final functions are single-shot functions).

SHA3 vs SHAKE algorithms (APIs usage):

• SHA3 digest generation: SHA3_Init; SHA3_Update; SHA3_Final;
• SHAKE (single-shot-output) output generation: SHAKE_Init; SHAKE_Absorb; SHAKE_Final;
• SHAKE (incremental) output generation: SHAKE_Init; SHAKE_Absorb; SHAKE_Squeeze⁺;

Call-outs:

Service indicator is updated:

• Inside SHA3 and SHAKE single shot APIs (as previously in AWS-LC);
• Inside SHA3_Final (as previously in AWS-LC);
• Inside SHAKE_Final (Single-Shot XOF Final output generation as previously in AWS-LC);
• Inside SHAKE_Squeeze (Streaming XOF Squeezes output generation updates the service indicator after each extendable output update);

All other algorithms that use SHA3/SHAKE APIs are updated:

• ML-KEM (SHA3/SHAKE calls will be inlined later)
• ML-DSA (SHAKE_Squeeze (incremental XOF output functionality) inside ML-DSA is never invoked with the KAT test vectors and gtests)

Testing:

./crypto/crypto_test --gtest_filter="KeccakInternalTest.*"
./crypto/crypto_test --gtest_filter="SHA3Test.*"
./crypto/crypto_test --gtest_filter="SHAKETest.*"

./crypto/crypto_test --gtest_filter="All/PerKEMTest.*"
./crypto/crypto_test --gtest_filter="All/PQDSAParameterTest.*"

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 83.87097% with 15 lines in your changes missing coverage. Please review.

Project coverage is 78.76%. Comparing base (3cea179) to head (077ef78).

Files with missing lines	Patch %	Lines
crypto/fipsmodule/sha/sha3.c	81.92%	15 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2098      +/-   ##
==========================================
+ Coverage   78.75%   78.76%   +0.01%     
==========================================
  Files         598      598              
  Lines      103657   103700      +43     
  Branches    14720    14732      +12     
==========================================
+ Hits        81637    81682      +45     
+ Misses      21368    21366       -2     
  Partials      652      652              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[darylmartin100]

You're removing a parameter from the public API. Will there be any impact for downstream users?

[justsmth]

I don't think this function is currently public. To verify, I checked our Rust bindings to see whether it was listed. I don't see it here: https://docs.rs/aws-lc-sys/latest/aws_lc_sys/#functions

[manastasova]

From my understanding, only the functions defined in the include/openssl are consider external APIs.
These APIs should be internal, however, they are exported so that they can be used in our tests. However, I am not sure if there may be some customers that use these internal APIs as well.