Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test End-to-End Encryption Without Breaking Privacy
Transcript
- Lucas: So, Luna, quick one: when you send an encrypted message on your phone, how do you know it actually reached the other person without being read along the way? Luna: I assume they got it and it said 'delivered.' Honestly, I trust the little lock icon. But you're the QA guy — how do you test that? Lucas: That's the billion-dollar question, and it's way harder than it sounds. Today, I want to dig into end to end encryption testing — because the entire point of E2EE is that nobody in the middle can read the data, which includes the testers. So how do you verify something you're not supposed to see? Luna: That's a paradox. If the testers can't see the plaintext, how do they even know the encryption is working? Lucas: Right, and that paradox is why most teams just trust the crypto library and only test the plumbing around it. But as we'll see, that leaves real gaps. And before we go deeper — you know, a lot of people ask us how we keep this show going without ads. It's a choice. We deliberately don't run ads on these episodes. If that matters to you, supporting us is easy: buy me a coffee dot com slash fexingo. No pressure, just know that listener support is what keeps this ad-free. Luna: Yeah, and we genuinely mean that — it's a nice way to give back when the show helps you. Anyway, back to encryption: what's the first thing you actually test? Lucas: The first thing is the test vectors. A test vector is a known input and expected output pair, like a hash value for a given message. For encryption, you want a set of plaintexts, keys, and nonces with known ciphertexts. You can get them from the spec or generate them with a reference implementation if you have one. Luna: So you're not testing the encryption itself, you're testing that your code matches the standard. Lucas: Exactly. And that's a crucial distinction. You're not testing whether AES is secure — you're testing whether your wrapper, your key derivation, your encoding all handle the standard correctly. The crypto library is assumed to be correct; your job is to test how you use it. Luna: But what if you don't have a reference implementation? Like, you're building your own protocol on top of a library? Lucas: Then you generate your own test vectors, but you have to be careful. The classic mistake is to generate vectors from your own implementation and then test your implementation against them. That's circular. You're just confirming your code is consistent with itself, which tells you nothing about correctness. Luna: So you need an independent source. Like, if you're using Signal Protocol, you'd use their published test vectors? Lucas: Signal publishes official test vectors for their protocol, and they're a lifesaver. They cover key agreement, message encryption, and ratchet steps. If your code produces the exact same outputs for those inputs, you've got a strong signal that you're on the right track. But that's just the start. Luna: What's the next step? Because I imagine you also need to test the key exchange, not just the message encryption. Lucas: Key exchange is where it gets really interesting. In E2EE, two parties need to agree on a shared secret without anyone else learning it. The standard is something like Diffie-Hellman combined with long-term keys. The testing challenge is that the exchange is inherently interactive — it's a multi-step handshake, and each party's state changes. Luna: So how do you test that deterministically? If the exchange involves randomness, you can't just replay a script. Lucas: That's the key insight. You have to make the test deterministic. For testing, you can inject fixed random values — you override the random number generator with a known seed. That way, the ephemeral keys are predictable, and you can verify the shared secret against a precomputed value. It's fragile, but it's the only way to have a repeatable test. Luna: Ah, so the test itself becomes a controlled experiment. You're not testing the randomness, you're testing the protocol logic. Lucas: Precisely. And once you've got a deterministic key exchange, you can also test the ratchet. In Signal protocol, every message derives a new key, so you can simulate a long conversation and verify that the keys evolve correctly. That catches a whole class of bugs where state doesn't update properly. Luna: Let's talk about the human side. I assume testing the UI for E2EE is different from testing a normal chat app. Lucas: It is, and this is where privacy really matters. When you're testing the UI, you don't want to use real user data — obviously. But you also have to be careful not to accidentally log plaintext. I've seen testers dump message content into logs to debug an issue, and that completely undermines the encryption. Luna: So what do you do? You need to test the UI flows, but you can't see the messages. Lucas: You use synthetic test data that is obviously fake — like 'test message 1' — and you make sure your logging is strict about never recording message bodies. Also, you can test the UI behavior without the correct plaintext: for example, you can verify that the message is encrypted on the wire, even if you can't see it in the app. Luna: How do you verify encryption on the wire? That goes beyond unit tests. Lucas: You can use a network proxy like Wireshark or mitmproxy, but here's the catch: with E2EE, the proxy sees only ciphertext. You can't read the message, but you can verify that the bytes you see are not plaintext. For example, if you search for the phrase 'test message 1' in the captured traffic and find it, your encryption is broken. If you don't find it, that's evidence — not proof, but evidence — that the message is encrypted. Luna: That's clever. It's a negative test — you're looking for the absence of plaintext. Lucas: Exactly. And you can also check that the ciphertext changes between messages, even if the plaintext is the same. That's a property of proper encryption with random nonces or ratchets. If two identical messages produce identical ciphertext, you've got a serious problem — it indicates you're not using a random IV or you have a static key. Luna: That's a great point. I hadn't thought of that. So you're testing the encryption's properties, not just the protocol. Lucas: And that brings me to another crucial angle: backward compatibility. E2EE is notoriously hard to update. If you change your protocol or key derivation, old clients can't talk to new ones. So you need to test that a client from two versions ago can still exchange messages with the latest one. Luna: How do you simulate that? You don't have old devices lying around. Lucas: You can use deterministic test vectors from the old version. If you have a known message and key from the old algorithm, you can verify that the new client can still decrypt it. That's a form of regression testing for the protocol. It's a bit like having a golden master for your crypto. Luna: Have you seen a real bug that would have been caught by these techniques? Lucas: Oh, absolutely. There was a famous bug in a messaging app where the encryption key was derived from a timestamp with minute-level granularity. So if two users sent messages in the same minute, they'd use the same key. A test that checked for identical ciphertexts for identical plaintexts in the same minute would have caught it. But they didn't have that test. Luna: That's a sobering example. And it shows that thinking about properties, not just following the spec, is essential. Lucas: Right. And there's another property worth testing: forward secrecy. In E2EE, even if a long-term key leaks, past messages should remain secure. You can simulate a key leak in a test — you know, intentionally expose a private key — and then verify that an attacker who captured old ciphertexts cannot decrypt them. Luna: That sounds like a tough test to write. You need to capture ciphertexts, then leak the key, then try to decrypt. How do you do that in an automated test? Lucas: You can, in a unit test, generate a message, encrypt it, then destroy the key — or, more realistically, replace the key with a new one — and then attempt to decrypt the old ciphertext using the new key. It should fail. That's a simple test for ratchet advancement. It's not full forward secrecy testing, but it's a good start. Luna: Let's talk about test data. If you're testing an E2EE feature in a staging environment, you can't use real user messages. What do you actually put in the database? Lucas: You use synthetic data that is clearly fake, and you ensure that it's encrypted at rest as well. The key thing is that the test data itself must not contain any real personal data. That's a privacy requirement, not just a QA best practice. So you generate random strings, maybe with a prefix like 'test-' so you can spot them easily if they leak. Luna: And how do you manage test keys? I'd imagine you need separate keys per environment. Lucas: Absolutely. You should have a separate key management system for testing. Never use production keys in test, even if they're test-generated. And you need to rotate them regularly. Also, test key generation should be deterministic — you can use a fixed seed to generate a set of keys for a test suite, so tests are reproducible. Luna: One thing I'm curious about: how do you test the 'second device' scenario? Like when you add a new device to your account and messages sync? Lucas: That's a classic scenario in E2EE, and it's tricky. The new device needs to receive the user's private key securely. The typical approach is to show a QR code on the old device and scan it with the new one. Testing that involves simulating the QR code generation and scanning, plus verifying that the key is transferred without being exposed. Luna: And you can't just point a camera at a screen in an automated test, right? Lucas: You can, actually, with image recognition, but it's brittle. A better approach is to refactor the code so the QR code data is a pure function — you give it the public keys and it produces a string. Then you can test that function directly. The actual scanning is just an I/O layer that you can test with a mock. Luna: So the key is to separate the crypto logic from the UI. That seems like a common theme in E2EE testing. Lucas: Exactly. The more you can isolate the crypto into pure functions, the easier it is to test deterministically. And that's a good design principle anyway. If you can't test the core logic without a UI, you've made a design mistake. Luna: Let's wrap up with a concrete checklist. If someone is starting to test an E2EE feature, what are the three most important things to do? Lucas: One: use published test vectors from a trusted source, or generate your own from an independent reference. Two: make your tests deterministic by injecting fixed randomness. And three: test for properties, not just outputs — like that ciphertext changes with each message, and that old ciphertexts stay undecryptable after a key rotation. Luna: That's a solid framework. And it's a lot deeper than 'trust the library.' Lucas: It is. And honestly, E2EE testing is one of the most intellectually satisfying areas of QA — because you're verifying the unverifiable. Every test you write is a small act of cryptographic reasoning. And that's what keeps me coming back.