Latest / Encrypted Ambition: Where Ambition Meets Encryption - Inside The Minds Of Disruptors. / Vmware Private AI Sovereign Cloud Alternative Ibm for Compliance

Transcript

Vmware Private AI Sovereign Cloud Alternative Ibm for Compliance · Encrypted Ambition: Where Ambition Meets Encryption - Inside The Minds Of Disruptors.

• 0:14
  Host: Today we’re looking at why private AI clusters on isolated hardware are becoming a must for defense and health firms.
• 0:21
  Analyst: Because public cloud AI services simply can’t meet the strict security baselines for CUI and PHI.
• 0:28
  Host: Can you explain the main regulatory hurdle that forces this shift?
• 0:32
  Analyst: The DFARS 252.204-7012 clause requires any cloud provider handling covered defense information to meet FedRAMP Moderate or equivalency.
• 0:44
  Host: And public cloud tenants aren’t assessed against that baseline?
• 0:48
  Analyst: Correct. Commercial productivity tenants never get a FedRAMP Moderate assessment, so they fall short of DFARS.
• 0:55
  Host: What about the DoD CIO CMMC FAQ clarifications regarding encrypted data?
• 1:01
  Analyst: Encrypted CUI is still treated as CUI. Encryption alone doesn’t remove the FedRAMP Moderate requirement.
• 1:08
  Host: So a defense contractor can’t just encrypt its data and use a public AI API?
• 1:13
  Analyst: Exactly. The data remains covered, so the cloud must still satisfy the baseline.
• 1:19
  Host: How does this affect healthcare organizations and HIPAA?
• 1:22
  Analyst: HIPAA’s Security Rule demands a risk analysis under 45 CFR 164.308(a)(1)(ii)(A).
• 1:31
  Host: And there’s no single HIPAA certification to rely on?
• 1:35
  Analyst: Right. Clients must perform their own risk assessments against the provider’s security posture.
• 1:41
  Host: What about NIST SP 800-171 and cryptography?
• 1:45
  Analyst: It requires FIPS-validated cryptography to protect CUI. Many public cloud setups use non-validated modules.
• 1:54
  Host: So that’s another compliance gap.
• 1:56
  Analyst: Yes, and it’s a common pitfall for organizations that assume any strong encryption suffices.
• 2:02
  Host: Shifting to hardware, what does a private AI cluster look like?
• 2:06
  Analyst: A 7B parameter model runs on a single NVIDIA A100 or H100 GPU.
• 2:13
  Host: What if the organization needs a larger model?
• 2:16
  Analyst: Larger models need 2 to 4 GPUs, making a multi-node cluster necessary.
• 2:21
  Host: Can you give an example of a cluster configuration?
• 2:24
  Analyst: The reference cluster uses GB10 Grace Blackwell nodes with 128GB unified memory each.
• 2:32
  Host: How are those nodes connected?
• 2:34
  Analyst: They’re linked over a QSFP112 400G interconnect to pool 256GB for larger models.
• 2:42
  Host: What about smaller deployments?
• 2:44
  Analyst: Single-box inference workstations use RTX 5090, RTX 6000, or H200 class GPUs.
• 2:52
  Host: So the hardware choice is a compliance decision too?
• 2:55
  Analyst: Yes. On-premise hardware means data never leaves the organization’s control.
• 3:00
  Host: And if it’s in the public cloud, the provider’s controls apply.
• 3:04
  Analyst: Exactly, and those controls may not meet the required frameworks.
• 3:08
  Host: What’s the process for mapping AI controls to the frameworks?
• 3:11
  Analyst: Petronella uses a six-stage deployment method.
• 3:14
  Host: Walk me through those stages.
• 3:16
  Analyst: Stage one is defining the data boundary and identifying applicable frameworks.
• 3:21
  Host: So you identify what’s CUI, PHI, or financial data?
• 3:26
  Analyst: Yes, then map those to CMMC, HIPAA, or DFARS as appropriate.
• 3:31
  Host: Stage two?
• 3:32
  Analyst: Sizing the GPU hardware based on model size and token volume.
• 3:37
  Host: Stage three?
• 3:38
  Analyst: Isolating the cluster on a segmented VLAN or full air gap.
• 3:42
  Host: So you physically separate the AI workloads?
• 3:45
  Analyst: That’s right. It prevents interaction with untrusted networks.
• 3:49
  Host: Stage four?
• 3:50
  Analyst: Deploying open-weight models on an inference stack you control.
• 3:53
  Host: Which models are supported?
• 3:55
  Analyst: Llama 3.1, Mistral, Qwen 2.5, and Phi are the families we benchmark.
• 4:01
  Host: Stage five?
• 4:02
  Analyst: Layering role-based access control, encryption at rest and in transit, and audit logging.
• 4:08
  Host: And mapping those controls to the framework requirements?
• 4:11
  Analyst: Exactly. That alignment is critical for audit readiness.
• 4:15
  Host: Stage six?
• 4:16
  Analyst: Validating the system against the mapped controls before production.
• 4:20
  Host: So you must prove compliance before going live?
• 4:23
  Analyst: Yes, validation ensures the cluster meets CMMC, HIPAA, or DFARS standards.
• 4:29
  Host: What frameworks does Petronella target?
• 4:31
  Analyst: CMMC Levels 1, 2, or 3; HIPAA; and DFARS 252.204-7012.
• 4:39
  Host: What’s the difference between CMMC Level 2 and Level 3?
• 4:42
  Analyst: Level 2 covers 110 NIST SP 800-171 requirements, Level 3 adds NIST SP 800-172 requirements.
• 4:55
  Host: And DFARS stays on NIST SP 800-171 Revision 2?
• 5:00
  Analyst: Yes, with a DoD class deviation for now.
• 5:03
  Host: How does the cost compare between private AI and public APIs?
• 5:07
  Analyst: At 500K+ tokens per day, private deployment breaks even within 6 to 12 months.
• 5:13
  Host: And at higher volumes?
• 5:14
  Analyst: At 5M+ tokens daily, private AI costs 60 to 80% less annually than API spend.
• 5:22
  Host: So the break-even point is tied to token volume?
• 5:25
  Analyst: Exactly. The higher the daily token count, the faster the ROI.
• 5:29
  Host: What about the risk of shadow AI?
• 5:31
  Analyst: The IBM 2025 Cost of a Data Breach report shows 20% of breaches involved shadow AI.
• 5:39
  Host: That’s employees using unapproved AI tools?
• 5:42
  Analyst: Yes, or private clusters not properly isolated from other network segments.
• 5:47
  Host: How do you mitigate that?
• 5:48
  Analyst: Implement egress control, prompt logging, and secrets handling as part of the hardening checklist.
• 5:54
  Host: What does the hardening checklist cover?
• 5:56
  Analyst: Network isolation, secrets handling, prompt and access logging, and egress control.
• 6:02
  Host: And how do you ensure audit readiness?
• 6:04
  Analyst: Use a compliance documentation platform to automate SSP authoring, POA&M tracking, and evidence repository.
• 6:13
  Host: Petronella offers that platform?
• 6:15
  Analyst: Yes, it’s called ComplianceArmor®.
• 6:17
  Host: So the platform manages documentation for CMMC and HIPAA?
• 6:21
  Analyst: It organizes evidence for audits and tracks remediation tasks.
• 6:25
  Host: What about the hybrid SOC model?
• 6:28
  Analyst: Petronella runs ten-plus production AI agents on the private cluster.
• 6:32
  Host: And the AI never closes a ticket on its own?
• 6:35
  Analyst: Correct. The AI never touches production systems without human authorization.
• 6:40
  Host: So every action is logged for audit?
• 6:43
  Analyst: Yes, logs are retained for CMMC and HIPAA audit purposes.
• 6:47
  Host: What’s the typical engagement model?
• 6:49
  Analyst: We provide a free 30-minute scoping consultation to map the data boundary.
• 6:54
  Host: So the first step is a scoping call?
• 6:56
  Analyst: Exactly. We identify data types, token volume, and applicable frameworks.
• 7:01
  Host: And then the six-stage deployment follows?
• 7:04
  Analyst: After scoping, we move through sizing, isolation, model deployment, control layering, and validation.
• 7:10
  Host: How do you handle the need for FIPS-validated cryptography?
• 7:14
  Analyst: All cryptographic modules in the stack are FIPS-validated, meeting NIST SP 800-171.
• 7:21
  Host: So the hardware, the isolation, the controls, and the documentation all align?
• 7:27
  Analyst: That’s the goal. Each element maps back to the specific framework requirement.
• 7:32
  Host: What’s the biggest mistake organizations make with private AI?
• 7:36
  Analyst: Assuming that simply running the cluster on isolated hardware automatically satisfies compliance.
• 7:42
  Host: Because the infrastructure still needs to be validated against the standards?
• 7:46
  Analyst: Yes, and the controls must be mapped, logged, and validated.
• 7:50
  Host: Another common error?
• 7:51
  Analyst: Using non-FIPS-validated encryption for CUI.
• 7:55
  Host: Which can lead to a compliance gap?
• 7:57
  Analyst: Exactly. The requirement is explicit: FIPS-validated cryptography.
• 8:02
  Host: What about the audit trail?
• 8:04
  Analyst: Audit logging must capture every prompt, access, and egress event.
• 8:08
  Host: So the logs become evidence during an assessment?
• 8:11
  Analyst: Yes, they demonstrate that the controls are in place and functioning.
• 8:15
  Host: What’s the timeline for a typical deployment?
• 8:18
  Analyst: It depends on data boundary complexity and framework scope, but the six stages guide the process.
• 8:24
  Host: How does the company ensure the cluster is truly isolated?
• 8:27
  Analyst: They use segmented VLANs or full air gaps, physically separating the AI workload.
• 8:33
  Host: And that’s part of the security baseline?
• 8:35
  Analyst: It satisfies the isolation requirement in CMMC and DFARS.
• 8:40
  Host: Are there any specific open-weight models that perform well on private hardware?
• 8:44
  Analyst: Llama 3.1, Mistral, Qwen 2.5, and Phi are benchmarked for performance and compliance.
• 8:52
  Host: Do they all support the same inference stack?
• 8:54
  Analyst: Yes, the inference stack is under the organization’s control, so they can manage it.
• 8:59
  Host: What about the role-based access control?
• 9:01
  Analyst: Roles are defined to limit who can deploy models, view logs, or modify configurations.
• 9:07
  Host: And the encryption at rest?
• 9:09
  Analyst: The storage volumes use FIPS-validated encryption modules.
• 9:13
  Host: So the entire stack is validated?
• 9:15
  Analyst: From hardware to software, each component is evaluated against the frameworks.
• 9:20
  Host: What about the cost of setting up a private cluster?
• 9:23
  Analyst: Hardware costs are upfront, but the ROI comes from reduced API spend and compliance risk.
• 9:29
  Host: And the break-even point is around 6 to 12 months?
• 9:32
  Analyst: Yes, at 500K+ tokens per day the system pays for itself within that window.
• 9:39
  Host: What if an organization is a small business with low token volume?
• 9:42
  Analyst: A single workstation with an RTX 5090 or RTX 6000 can serve as a proof of concept.
• 9:50
  Host: So they can test before committing to a full cluster?
• 9:52
  Analyst: Exactly. It lowers the entry barrier for experimentation.
• 9:56
  Host: What about the compliance documentation?
• 9:59
  Analyst: ComplianceArmor® automates the creation of the System Security Plan and tracks remediation.
• 10:06
  Host: So the platform reduces the administrative burden?
• 10:08
  Analyst: Yes, it keeps evidence organized and ready for audit.
• 10:12
  Host: How does the hybrid SOC work with the private cluster?
• 10:15
  Analyst: The SOC runs AI agents that analyze threat data, but the AI never modifies production systems.
• 10:22
  Host: So the human operator remains in control?
• 10:24
  Analyst: Exactly. The AI assists, but all actions require human authorization.
• 10:29
  Host: What about the risk of data leakage through the AI?
• 10:32
  Analyst: The isolation and egress controls prevent unauthorized data transfer.
• 10:37
  Host: And the logs capture any attempted exfiltration?
• 10:40
  Analyst: Yes, every egress event is logged and mapped to the framework controls.
• 10:45
  Host: What’s the next step for an organization that wants to adopt this approach?
• 10:49
  Analyst: First, they need to define the data boundary and determine which frameworks apply.
• 10:54
  Host: So they start with a scoping exercise?
• 10:56
  Analyst: Yes, a 30-minute consultation can guide them through that process.
• 11:00
  Host: After that, they would move into hardware sizing?
• 11:03
  Analyst: Correct. They’ll assess token volume and model size to choose the appropriate GPU configuration.
• 11:09
  Host: And then isolate the cluster?
• 11:11
  Analyst: On a segmented VLAN or full air gap to meet isolation requirements.
• 11:16
  Host: Then deploy the open-weight model?
• 11:18
  Analyst: Yes, benchmarking to ensure performance meets the use case.
• 11:22
  Host: Layer the controls and validate before production?
• 11:25
  Analyst: Exactly. That completes the six-stage deployment.
• 11:28
  Host: So what should organizations do about all of this?
• 11:31
  Host: So after that six-step framework, what are the immediate next steps for an organization that’s just starting to consider private AI?
• 11:39
  Analyst: First, they should schedule a scoping session to map out the exact data boundary. That means cataloging every piece of CUI, PHI, or other regulated data they plan to feed into the model.
• 11:51
  Host: That sounds like a lot of work. How long does that scoping usually take?
• 11:55
  Analyst: A focused 30-minute consultation can surface the key data types and the frameworks-CMMC, HIPAA, DFARS-that apply. From there you can draft a high-level scope.
• 12:06
  Host: Once the scope is clear, what comes next?
• 12:09
  Analyst: You move to hardware sizing. You need to estimate token throughput. If you’re looking at 500,000 tokens per day, a single NVIDIA A100 or H100 can handle a 7-B parameter model.
• 12:22
  Host: And if we need more capacity?
• 12:24
  Analyst: For 5 million tokens daily, you’d move to a small or large cluster. A small cluster uses one GPU; a large cluster uses GB10 Grace Blackwell nodes with 128GB unified memory each, linked over a QSFP112 400G interconnect to pool 256GB.
• 12:45
  Host: That covers the compute side. How about isolation?
• 12:49
  Analyst: Isolation is the third step. You can segment the cluster on a dedicated VLAN or, for the highest assurance, install a full air gap. That prevents any traffic from the cluster to untrusted networks.
• 13:02
  Host: I’ve heard about air gaps before. Is that mandatory?
• 13:06
  Analyst: Not strictly mandatory, but it’s the most robust approach. Many DFARS-compliant contractors choose it to satisfy the FedRAMP Moderate equivalency requirement for covered defense information.
• 13:19
  Host: What about the software stack?
• 13:20
  Analyst: You deploy open-weight models-Llama 3.1, Mistral, Qwen 2.5, or Phi-on an inference stack you own. That lets you benchmark performance against your specific use case before finalizing.
• 13:34
  Host: And the controls?
• 13:35
  Analyst: You layer role-based access control, enforce encryption at rest and in transit, and enable audit logging. For CUI, the encryption must be FIPS-validated per NIST SP 800-171.
• 13:50
  Host: So we can’t just use any encryption library?
• 13:52
  Analyst: Exactly. A standard AES implementation that isn’t FIPS-validated doesn’t satisfy the requirement. You need to verify every cryptographic module in the stack.
• 14:03
  Host: What about the compliance mapping?
• 14:05
  Analyst: You map each technical control to the exact framework requirement. For example, CMMC Level 2 requires 110 controls from NIST SP 800-171. Level 3 adds a subset of NIST SP 800-172.
• 14:24
  Host: Does that mapping need to be documented?
• 14:26
  Analyst: Yes, it’s part of the System Security Plan. You also need a Plan of Actions and Milestones to track remediation. ComplianceArmor is a platform that can automate that documentation.
• 14:37
  Host: Once everything is in place, how do we validate?
• 14:41
  Analyst: You perform a validation phase-run test cases, verify audit logs, confirm encryption keys, and ensure the cluster never contacts external networks. Only after passing validation do you move to production.
• 14:55
  Host: What are the typical pitfalls that organizations run into?
• 14:58
  Analyst: One big mistake is assuming that encrypting CUI in the cloud removes the FedRAMP Moderate requirement. The DoD CIO CMMC FAQ says encrypted CUI is still CUI.
• 15:11
  Host: So encryption alone isn’t enough.
• 15:13
  Analyst: Right. Another mistake is using non-validated cryptography. That violates NIST SP 800-171 and can lead to a compliance gap.
• 15:24
  Host: And shadow AI?
• 15:25
  Analyst: Shadow AI is a real risk. 20% of breaches involved shadow AI. If employees use unapproved tools, data can leak. You mitigate that with egress controls and prompt logging.
• 15:38
  Host: What about mapping controls to frameworks?
• 15:40
  Analyst: If you handwave the mapping, you can’t demonstrate compliance. Each control must be tied to a specific requirement-like a CMMC 2 control or a HIPAA sub-requirement.
• 15:52
  Host: So documentation is key.
• 15:53
  Analyst: Exactly. An automated compliance platform helps keep evidence organized and ready for audit.
• 16:00
  Host: What questions do you hear most often from clients?
• 16:03
  Analyst: They ask if a private AI cluster is the same as a sovereign cloud.
• 16:07
  Host: How do you explain the difference?
• 16:09
  Analyst: A private cluster is dedicated infrastructure you control, usually on-premises or in a private data center. A sovereign cloud is a cloud service that guarantees data stays within a jurisdiction, but it still needs FedRAMP Moderate equivalence for CUI.
• 16:25
  Host: Can they use a public cloud if they encrypt everything?
• 16:28
  Analyst: Not for CUI. Even encrypted CUI in a public cloud still requires FedRAMP Moderate or equivalence. So the public cloud provider must be assessed against that baseline.
• 16:39
  Host: What hardware do they need for a 7-B model?
• 16:42
  Analyst: One NVIDIA A100 or H100 GPU. That’s the minimum for a 7-B parameter model.
• 16:49
  Host: How long does the whole deployment take?
• 16:52
  Analyst: It depends on complexity-scope, hardware procurement, and validation. But the six-stage method gives a clear roadmap that can be completed in a few months.
• 17:01
  Host: What’s the financial upside?
• 17:03
  Analyst: At 500,000 tokens per day, private AI can break even in 6 to 12 months. At 5 million tokens daily, it’s 60 to 80 percent cheaper than API spend.
• 17:15
  Host: So the cost savings are significant.
• 17:17
  Analyst: Yes, especially when you factor in the compliance risk mitigation.
• 17:21
  Host: What if an organization wants to start small?
• 17:23
  Analyst: They can use a single-box inference workstation-RTX 5090, RTX 6000, or H200 class GPU-for proof-of-concept or low token volumes.
• 17:35
  Host: And then scale up?
• 17:36
  Analyst: When token volume increases or they need larger models, they move to a small or large cluster, adding GPUs and interconnect bandwidth accordingly.
• 17:46
  Host: What about the NIST AI Risk Management Framework?
• 17:49
  Analyst: The NIST AI RMF provides a voluntary framework for managing AI risks. The 2024 AI RMF 1.0 profile focuses on generative AI and can help organizations align their risk management with broader NIST guidance.
• 18:05
  Host: Is that mandatory?
• 18:06
  Analyst: No, it’s voluntary, but it’s increasingly referenced in assessments, so aligning early is prudent.
• 18:12
  Host: How do we ensure the system is audit-ready?
• 18:15
  Analyst: Implement RBAC, FIPS-validated encryption, and comprehensive audit logging. Use a compliance documentation platform to automate evidence collection and maintain a ready-for-audit state.
• 18:27
  Host: What about the DFARS 252.204-7012 clause?
• 18:32
  Analyst: That clause requires a cloud service provider handling covered defense information to meet FedRAMP Moderate equivalence. If you’re using a public cloud, you need to verify that equivalence.
• 18:44
  Host: So a private cluster sidesteps that requirement?
• 18:47
  Analyst: Yes, because the data never leaves the organization’s controlled environment, eliminating the need for that clause.
• 18:54
  Host: What about the DoD CIO CMMC FAQ clarifications?
• 18:58
  Analyst: They clarified that encrypted CUI remains CUI. So encryption alone doesn't satisfy the requirement; you must still meet FedRAMP Moderate or equivalence.
• 19:08
  Host: In practice, how do you demonstrate that the cluster is isolated?
• 19:12
  Analyst: You document the network segmentation, show VLAN or air-gap configuration, and provide evidence that no outbound traffic is possible from the cluster.
• 19:22
  Host: What about prompt logging?
• 19:24
  Analyst: Prompt logging records every input to the model and the model’s output. That satisfies audit requirements and helps detect anomalous behavior.
• 19:33
  Host: How do you handle secrets?
• 19:34
  Analyst: Secrets are stored in a dedicated vault with strict access controls. Only authorized personnel can retrieve them, and all access is logged.
• 19:44
  Host: Is there a standard checklist for hardening?
• 19:46
  Analyst: Yes, the blueprint hardening checklist covers network isolation, secrets handling, prompt and access logging, and egress control. It walks through eight steps to implement these controls.
• 19:58
  Host: That’s helpful. What about the risk of a breach?
• 20:01
  Analyst: The IBM 2025 Cost of a Data Breach report shows 16% of breaches involved attacker use of AI, and 20% involved shadow AI. That underscores the need for tight controls.
• 20:15
  Host: What about the Verizon 2025 DBIR findings?
• 20:19
  Analyst: It found ransomware in 88% of SMB breaches versus 39% at large organizations, and third-party involvement rose from 15% to 30%. That highlights the necessity of vendor risk management.
• 20:34
  Host: So the takeaway is that private AI can be both compliant and cost-effective.
• 20:39
  Analyst: Precisely. When you follow the six-stage method, you map every control to the relevant framework, use FIPS-validated cryptography, isolate the cluster, and validate before production. That gives you compliance, security, and savings.
• 20:54
  Host: Thanks for that deep dive, analyst.