Skip to main content

All weeks · Worksheet · Overview

Week 10 · Lecture slides

Week 10

Contents17 sections

Asymmetric & Hybrid Encryption

Security & Cryptography · Nutthakorn Chalaemwongwan

Today

  • Why asymmetric encryption alone can't encrypt bulk data
  • Hybrid encryption: wrap a symmetric key once, encrypt many messages
  • Asymmetric key exchange (ECDH) → ECIES as a concrete hybrid scheme
  • 🔬 Game: Nonce Detective — audit an AI-generated hybrid implementation

Recap — Week 6

  • AEAD (AES-GCM) = confidentiality + integrity + authenticity in one primitive
  • It closed the padding-oracle hole in unauthenticated CBC — no padding step, tag checked before any plaintext is released
  • Today: AES-GCM shows up again — genuinely correct, and still breakable
  • Same spine, new victim: even the primitive that fixed last week's failure has its own misuse mode

Why not just use RSA for everything?

  • RSA operations are orders of magnitude slower than AES for bulk data
  • RSA-4096-OAEP: roughly a 470-byte plaintext ceiling — can't fit a photo, let alone a video
  • OAEP padding eats further into the usable message size
  • The fix isn't "use a bigger RSA key" — it's "use RSA for something small"

Hybrid encryption: wrap once, encrypt many

  1. Bob generates a fresh AES-256 session key (CSPRNG)
  2. Bob wraps it with Alice's RSA public key (OAEP) — sent once
  3. Bob encrypts every message with AES-GCM under that session key
  4. Alice unwraps the session key with her RSA private key — once
  5. Alice decrypts every GCM ciphertext with the unwrapped key

Pay RSA's cost once per session, not once per message.

Public-key encryption in one slide

  • Alice publishes her public key; keeps the private key secret
  • Bob encrypts to Alice's public key — anyone can do this, it's public
  • Only the matching private key can invert the trapdoor function
  • Alice never has to share a secret with Bob in advance — that's the whole point over symmetric crypto

Asymmetric key exchange

  • Alice and Bob each generate an ephemeral elliptic-curve key pair
  • They exchange public keys over the open channel
  • Each combines their own private key with the other's public key → the same shared secret on both sides
  • An eavesdropper who sees both public keys cannot compute it (discrete-log hardness on the curve)

ECIES: hybrid encryption on ECDH

  • Sender generates an ephemeral EC key pair, runs ECDH against the recipient's public key
  • Shared secret → KDF (e.g. HKDF) → symmetric key — never the raw ECDH output used directly
  • Symmetric key encrypts the message with an AEAD cipher (AES-GCM / ChaCha20-Poly1305)
  • Ephemeral public key + nonce travel with the ciphertext, in the clear
  • No symmetric key is ever transmitted — Bob and Alice each derive it independently

A shallow review would stop here

A teammate's AI assistant wrote a Bob-to-Alice hybrid encryption module. A quick review checks:

  • Real algorithm, not textbook RSA? Uses RSA-OAEP. ✅
  • Symmetric key strong and CSPRNG-generated? AES-256, os.urandom-backed. ✅
  • Authenticated, not plain CBC/ECB? AES-GCM. ✅
  • Tag actually checked on decrypt, not ignored? ✅

It runs. The round trip succeeds. Alice decrypts everything Bob sends. Approved — right?

The failure mode: nonce reuse under GCM

  • AES-GCM needs a nonce — a value that must never repeat under the same key
  • NIST SP 800-38D §8.2: the (key, nonce) pair must be unique for every message
  • The failure shape to hunt for: a nonce that's a fixed constant, or derived from something that isn't actually unique per call, reused silently across an entire session
# illustrative shape of the bug — not the exact lab code
nonce = FIXED_VALUE                       # same nonce, every call
ciphertext = aesgcm.encrypt(nonce, plaintext, aad)
  • CWE-323 — Reusing a Nonce, Key Pair in Encryption

The break: what a passive eavesdropper gets

  • GCM's keystream depends only on (key, nonce) — same pair twice ⇒ same keystream
  • C1 = P1 ⊕ KS and C2 = P2 ⊕ KS ⇒ C1 ⊕ C2 = P1 ⊕ P2
  • No key needed, no interaction with Bob or Alice — just two intercepted ciphertexts, XORed
  • Any crib — a known word, a predictable header, language redundancy — peels apart both plaintexts

Worse: it's not just confidentiality

  • Week 6's padding oracle needed an active attacker — send a query, read 200 vs 403, repeat
  • Nonce reuse needs only a passive eavesdropper — two recorded messages, zero interaction
  • Reuse under GCM specifically also leaks the authentication subkey (Joux, 2006 — "the forbidden attack")
  • Recover that subkey and you can forge valid tags on messages nobody sent — both of GCM's guarantees fail from one mistake

The fix — general principle

  • Never reuse a (key, nonce) pair — give every message its own nonce
  • Two standard disciplines: a fresh CSPRNG-random 96-bit nonce per message, or a monotonic counter — pick one, don't mix
  • The nonce isn't secret — it travels with the ciphertext, in the clear
  • Untouched: RSA key size, OAEP padding, the AES-256 session key, per-session key wrapping
  • The bug lives in nonce discipline, not in any primitive — the fix stays local to the encrypt call

🔬 Signature game — Nonce Detective

  • You're handed a hybrid-encryption implementation that looks textbook-correct — proper RSA-OAEP, proper AES-GCM, nothing wrong at a skim
  • Find the one planted bug — then prove it: recover leaked plaintext from two intercepted messages, not just point at a line
  • It passes every "did they use the right primitives" checklist — the only way to catch it is to think like an attacker

Lab today

📋 Worksheet 10 — labs/week10-hybrid-encryption/worksheet.md (Parts 1–5) · start at audit_the_ai/README.md

  • Part 2 — 5 essay questions: bulk-data limits, hybrid encryption flow, public-key encryption, key exchange, ECIES
  • Part 3 (required) — Audit-the-AI: find the planted bug in broken_hybrid_encrypt.py, write a proof script recovering real leaked plaintext, explain the attack, submit fixed_hybrid_encrypt.py
  • Part 4 — EiPE (explain hybrid encryption, no jargon) + Prompt Problem (ask an AI to implement ECIES, critique its output against a checklist)
  • Part 5 — live viva spot-check, no notes
  • No Docker target, no flag — conceptual week, graded on your written reasoning and proof script

Key takeaways

  • Textbook-secure primitive, real-system failure — this week it's AEAD's turn
  • RSA-OAEP correct + AES-256 correct + AES-GCM correct does not imply a secure system
  • One reused nonce breaks both confidentiality and authenticity of GCM — and needs only a passive eavesdropper
  • "Did they use the right primitives?" is necessary — it is never sufficient. AI-generated crypto code needs the same scrutiny as any other AI-generated code

Questions?

Next week: Digital Signatures & Zero-Knowledge Proofs

All weeks in Security & Cryptography