Latest / Encrypted Ambition: Where Ambition Meets Encryption - Inside The Minds Of Disruptors. / Why Gemma Is the Right Local AI Model for CMMC Compliance

Transcript

Why Gemma Is the Right Local AI Model for CMMC Compliance · Encrypted Ambition: Where Ambition Meets Encryption - Inside The Minds Of Disruptors.

• 0:14
  Host: Gemma 4 is a new open-weight AI from Google that can run entirely on a contractor’s own hardware, keeping controlled data inside the security boundary. How does that work, analyst?
• 0:25
  Analyst: Gemma 4 was released on March 31, 2026 in several sizes: E2B, E4B, 26B A4B, 31B Dense, and a 12B Unified added on June 3, 2026. It ships under an Apache 2.0 license.
• 0:43
  Host: So the license is standard open source, but what makes it practical for defense contractors?
• 0:49
  Analyst: Apache 2.0 is a commercially permissive license that eliminates a vendor-specific terms-of-use document. Legal teams already review it for other software, so the process is straightforward.
• 1:01
  Host: And the hardware? What do we need to run the biggest model?
• 1:04
  Analyst: The unquantized 31B Dense model fits a single 80GB NVIDIA H100 GPU. Quantized versions run on consumer GPUs, even on phones or Raspberry Pi.
• 1:18
  Host: That’s a wide range. How do the smaller models perform?
• 1:22
  Analyst: E2B and E4B use per-layer embeddings and run offline with near-zero latency on a phone or a Jetson Orin Nano. They also support 30-second audio clips natively.
• 1:34
  Host: What about context windows? How much text can each model handle?
• 1:39
  Analyst: E2B and E4B support 128K tokens. The 12B Unified, 26B A4B, and 31B Dense models support 256K tokens, giving a full technical document in one request.
• 1:57
  Host: That’s impressive. Why is local inference so critical for CUI?
• 2:02
  Analyst: When a contractor pastes CUI into a hosted AI, the data leaves the system security plan and requires additional controls. Local inference keeps prompts, documents, and outputs inside the existing boundary.
• 2:16
  Host: So it satisfies DFARS 252.204-7012 and NIST SP 800-171?
• 2:24
  Analyst: Exactly. The inference host sits on a segmented VLAN or air-gap, so the same access, audit, and encryption controls that protect other CUI apply to the AI stack.
• 2:36
  Host: How does that affect the System Security Plan?
• 2:38
  Analyst: Adding a model expands the SSP boundary to include the runtime, checkpoint storage, and inference cache. Those components must be mapped to the 110 controls of NIST SP 800-171.
• 2:53
  Host: And if a requirement isn’t met?
• 2:55
  Analyst: It goes into a Plan of Actions and Milestones, which is allowed under 32 CFR 170.21. The model file itself isn’t compliant; the deployment is what gets assessed.
• 3:08
  Host: Can you walk us through Petronella’s seven-stage method?
• 3:11
  Analyst: Sure. First, define the data boundary and applicable frameworks. Second, run a paid scoping engagement to build a prototype on the client’s data.
• 3:20
  Host: What does the prototype involve?
• 3:22
  Analyst: It measures context lengths, concurrency, and accuracy requirements, then records those metrics for hardware sizing.
• 3:30
  Host: So hardware sizing follows the prototype results?
• 3:33
  Analyst: Yes. The GPU cluster is sized based on the prototype’s performance. A 31B Dense model under concurrency requires an H100, while an E4B field utility might fit on a single RTX 5090.
• 3:47
  Host: After sizing, what’s next?
• 3:49
  Analyst: The cluster is isolated-either a VLAN or a full air-gap-so only approved traffic reaches the inference host.
• 3:57
  Host: How are access and audit controls layered?
• 4:00
  Analyst: Role-based access controls map to NIST 3.1, encryption at rest and in transit uses FIPS-validated cryptography, and audit logging satisfies NIST 3.3. All logs are preserved for the 90-day retention required by DFARS.
• 4:17
  Host: And the final stage?
• 4:19
  Analyst: Validate the deployment against the mapped controls, document the results in the SSP, and then hand off to a C3PAO for certification.
• 4:28
  Host: What about Section 1532 of the FY2026 NDAA?
• 4:32
  Analyst: Section 1532 defines covered artificial intelligence as AI developed by DeepSeek or by High Flyer and its affiliates. Gemma, developed by Google DeepMind, is not mentioned.
• 4:45
  Host: So Gemma isn’t restricted by that provision?
• 4:48
  Analyst: Correct. The law prohibits only the named developers; it does not reference the model family or its licensing.
• 4:55
  Host: Are there other Chinese-origin models that raise concerns?
• 4:59
  Analyst: GLM-5.3, for example, is owned by Zhipu AI, which is on the BIS Entity List. Exporting its weights to a hosted service would require a license, but self-hosting is permissible.
• 5:11
  Host: What about future legislation?
• 5:13
  Analyst: The FY2027 Senate bill S. 4784 would add Zhipu AI to the list of covered AI. That could force a mid-contract change if the model is used.
• 5:25
  Host: So risk management is key. How does Gemma compare to GPT-OSS?
• 5:30
  Analyst: Both are Apache 2.0, but GPT-OSS 20B runs on a 16GB GPU and GPT-OSS 120B on an 80GB GPU with MXFP4 quantization. They are text-only and have smaller context windows of 128K tokens.
• 5:50
  Host: Gemma offers more modalities and larger context windows, right?
• 5:54
  Analyst: Exactly. Every Gemma 4 size supports images, and E2B, E4B, and 12B Unified also support audio. The larger models provide 256K tokens, which is useful for long technical documents.
• 6:10
  Host: If a contractor needs a small, low-cost deployment, what’s the best choice?
• 6:15
  Analyst: The E2B model in the QAT mobile format fits in just 1GB of memory and runs offline on a phone or Raspberry Pi. That covers field laptops or edge devices.
• 6:27
  Host: And for a mid-size contractor with a data center?
• 6:30
  Analyst: A single 80GB H100 can run the 31B Dense model, which is the foundation for fine-tuning. That fits within a typical CUI enclave.
• 6:41
  Host: What about the legal review process for the model?
• 6:44
  Analyst: Because the license is Apache 2.0, the review is a standard open-source assessment. No separate policy document or click-through gate is required.
• 6:53
  Host: How does Petronella’s own deployment illustrate this approach?
• 6:56
  Analyst: Petronella runs a private AI cluster in a hybrid SOC that never sends CUI to a public cloud. The AI never closes tickets autonomously, and all actions are logged for CMMC and HIPAA audit.
• 7:10
  Host: What does a typical engagement look like in terms of duration?
• 7:13
  Analyst: A CMMC Level 2 readiness engagement spans 12 to 14 weeks: discovery, gap analysis, remediation sprint, and C3PAO handoff.
• 7:24
  Host: During remediation, the inference host is built and the model deployed, correct?
• 7:29
  Analyst: Yes, the chosen checkpoint is installed, access controls are applied, audit logging is connected, and the system is validated against the SSP.
• 7:39
  Host: Who is responsible for issuing the certificate?
• 7:42
  Analyst: Only a C3PAO can issue a Level 2 certificate. An RPO prepares the contractor but cannot conduct the assessment.
• 7:50
  Host: So the contractor’s C3PAO partner must be independent from Petronella?
• 7:55
  Analyst: Exactly. Independence rules prohibit a provider from performing both preparation and assessment on the same client.
• 8:02
  Host: What about change discipline when updating the model?
• 8:05
  Analyst: Each new checkpoint is recorded with the developer name, license, and version. The model is checked against Section 1532 before deployment.
• 8:15
  Host: And if a new version is released, how is it handled?
• 8:17
  Analyst: The change is logged, the SSP is updated, and the new model undergoes the same validation process before going live.
• 8:25
  Host: So the entire stack is mapped to NIST controls, and the boundary remains within the SSP.
• 8:30
  Analyst: Yes, the inference host inherits the same protection as the rest of the enclave-segmented access, audit, encryption, and incident reporting.
• 8:40
  Host: Now that we’ve covered the technical and compliance aspects, let’s talk about what organizations should do next.
• 8:46
  Analyst: First, organizations should map the Gemma 4 checkpoint to the existing System Security Plan boundary.
• 8:53
  Host: That means identifying which CUI assets will interact with the model and labeling them in the SSP.
• 8:59
  Analyst: Once the boundary is defined, the next step is a paid scoping engagement to prototype on real data.
• 9:05
  Host: During that prototype we measure context lengths, concurrency, and the accuracy required for the business use case.
• 9:12
  Analyst: Those measurements feed directly into the GPU sizing calculation for the inference cluster.
• 9:18
  Host: We saw that a 31B Dense model serving 256K tokens at moderate concurrency needs a single 80GB H100.
• 9:26
  Analyst: In contrast, an E2B model can run on a phone or a Raspberry Pi with a 1GB mobile checkpoint.
• 9:32
  Host: That flexibility lets a contractor span from field laptops to a hardened enclave server.
• 9:38
  Analyst: After sizing, the cluster is isolated, typically on a segmented VLAN or an air-gap, to enforce the same controls as the rest of the system.
• 9:47
  Host: Isolation ensures that only authorized traffic can reach the inference host, satisfying 3.13 System and Communications Protection.
• 9:56
  Analyst: Once isolated, the open-weight model is downloaded under the Apache 2.0 license, with no click-through gate.
• 10:03
  Host: That simplifies the legal review because the license is standard across the software stack.
• 10:08
  Analyst: After download, the weights are installed on the inference stack, and role-based access control is defined.
• 10:14
  Host: Role-based access maps to NIST 3.1 Access Control, ensuring only certain users can submit prompts.
• 10:22
  Analyst: Audit logging is then wired to the same collection point used for other CMMC controls.
• 10:28
  Host: This satisfies 3.3 Audit and Accountability, capturing who requested what and when.
• 10:34
  Analyst: Encryption at rest and in transit uses FIPS-validated cryptography, as required by 800-171.
• 10:43
  Host: The inference host also logs incident data for the 90-day preservation period mandated by DFARS.
• 10:49
  Analyst: Because the model runs locally, every prompt, document, and output stays inside the enclave boundary.
• 10:56
  Host: That eliminates the risk of data crossing into a third-party cloud, which would require additional agreements.
• 11:02
  Analyst: The next concrete step is to document the model checkpoint in the SSP, listing its version and license.
• 11:09
  Host: That documentation becomes part of the evidence repository during the C3PAO assessment.
• 11:14
  Analyst: If a new checkpoint is released, the organization must log the change, update the SSP, and re-validate.
• 11:21
  Host: Re-validation ensures the new model still meets the same NIST control mappings and performance metrics.
• 11:27
  Analyst: Common mistakes include adding the model to the SSP without detailing the inference host’s segmentation.
• 11:34
  Host: Another mistake is overlooking the cache memory requirement for long context windows.
• 11:39
  Analyst: The KV cache grows with context length, so a 256K token request can double the VRAM usage.
• 11:47
  Host: Failing to account for that can lead to out-of-memory errors during production.
• 11:51
  Analyst: Another pitfall is assuming the Apache 2.0 license covers all downstream use cases.
• 11:56
  Host: While it is permissive, you still need to verify that downstream partners can accept an open-weight model.
• 12:02
  Analyst: Organizations should also review the model’s provenance, ensuring the download source is a trusted channel like Hugging Face.
• 12:09
  Host: That way you can trace the weights back to the developer and confirm compliance with Section 1532.
• 12:15
  Analyst: Because Gemma 4 is not covered by Section 1532, it poses no statutory restriction for DoD contracts.
• 12:23
  Host: In contrast, the pending FY2027 bill could add Zhipu AI to that list, making GLM-5.3 risky.
• 12:32
  Analyst: Export controls also apply to Zhipu, since its entities are on the BIS Entity List.
• 12:37
  Host: That means any hosted-API usage would require an export license, but self-hosting avoids that.
• 12:43
  Analyst: But the security record of GLM-5.3 raises additional concerns, as advisory reports note its agentic capabilities.
• 12:52
  Host: Petronella’s approach is to keep the model local, so it stays under the same controls as the rest of the system.
• 12:58
  Analyst: That also simplifies incident response because the logs are already integrated into the existing SOC workflow.
• 13:05
  Host: Now, let’s discuss the hardware options for different business sizes.
• 13:10
  Analyst: A small contractor might deploy the E2B model on a single 80GB H100 if they need high throughput.
• 13:17
  Host: But if the workload is light, the 1GB mobile checkpoint can run on a Raspberry Pi without any GPU.
• 13:23
  Analyst: For a mid-size contractor, the 12B Unified model offers 256K context and runs on a single RTX 5090 or H200 class card.
• 13:36
  Host: The 31B Dense model is best suited for a data-center enclave with an 80GB H100 and 128GB unified memory per node.
• 13:45
  Analyst: Our benchmarks showed 63.5 tokens per second on an RTX PRO 6000 Blackwell with 192GB DDR5 when running the 31B Dense model.
• 13:58
  Host: When scaled to 32-way concurrency, the aggregate throughput reaches 2,173 tokens per second across the cluster.
• 14:06
  Analyst: Those numbers illustrate the performance gap between a single GPU and a clustered environment.
• 14:12
  Host: When choosing between Gemma 4 and GPT-OSS, consider modality and context requirements.
• 14:18
  Analyst: Gemma 4 supports text, image, and native audio, which is useful for scanned documents and voice notes.
• 14:25
  Host: GPT-OSS only handles text, so it may be sufficient for code analysis or structured drafting.
• 14:31
  Analyst: However, GPT-OSS 120B offers 128K tokens, which may limit document size compared to Gemma’s 256K window.
• 14:44
  Host: So the choice depends on the volume of text and the need for multimodal input.
• 14:48
  Analyst: For defense contractors, the security posture of the model is as important as its performance.
• 14:54
  Host: Because Gemma 4 is not covered by Section 1532, it avoids the statutory prohibition that applies to DeepSeek.
• 15:01
  Analyst: If you adopt GPT-OSS, you still stay within the same compliance boundary because the model runs locally.
• 15:08
  Host: But you must verify that the model’s license is Apache 2.0 and that no hidden clauses restrict use of CUI.
• 15:16
  Analyst: The seven-stage method also includes a final validation step where the model’s outputs are compared against ground truth.
• 15:24
  Host: That helps catch hallucinations and ensures the AI behaves as intended for mission-critical tasks.
• 15:30
  Analyst: A common oversight is to skip that validation, assuming the model’s performance metrics alone guarantee reliability.
• 15:37
  Host: That can lead to erroneous decisions in threat analysis or compliance reporting.
• 15:42
  Analyst: During the remediation sprint, teams should also plan for model versioning and rollback procedures.
• 15:47
  Host: Versioning ensures you can revert to a known good checkpoint if a new release introduces bugs.
• 15:53
  Analyst: Rollback procedures should be documented in the SSP and tested in a staging environment.
• 15:59
  Host: The Petronella team recommends using a separate staging cluster that mirrors the production hardware.
• 16:04
  Analyst: That way you can validate the new model without impacting live CUI processing.
• 16:09
  Host: After validation, the final step is to submit the evidence to the C3PAO for certification.
• 16:15
  Analyst: The C3PAO will review the SSP, the audit logs, and the model documentation before issuing the certificate.
• 16:23
  Host: Once certified, the contractor can confidently use Gemma 4 for CUI processing within the DoD ecosystem.
• 16:30
  Analyst: If a new version is released, you repeat the validation and re-documentation steps to maintain compliance.
• 16:37
  Host: That ensures the SSP remains current and the C3PAO can audit the system at any time.
• 16:42
  Analyst: Another question we hear is whether the model can be fine-tuned on proprietary data.
• 16:48
  Host: Fine-tuning is possible, but you need to keep the training pipeline inside the enclave and audit the process.
• 16:54
  Analyst: The training data must also be classified as CUI and handled under the same NIST controls.
• 17:00
  Host: If you use a third-party training service, that would require a separate data-processing agreement.
• 17:05
  Analyst: Petronella’s approach is to build the fine-tuning pipeline on the same hardware used for inference.
• 17:11
  Host: That keeps the entire lifecycle under your control and simplifies audit requirements.
• 17:16
  Analyst: Finally, we recommend establishing a change management process that tracks model updates, version numbers, and licensing status.
• 17:24
  Host: Recording those details in the SSP and evidence repository helps future assessors see the evolution of the AI stack.
• 17:32
  Analyst: That completes the practical roadmap for deploying Gemma 4 under CMMC Level 2 compliance.
• 17:38
  Host: Thanks for the deep dive into the compliance and deployment details.