Latest / Encrypted Ambition: Where Ambition Meets Encryption - Inside The Minds Of Disruptors. / Serve GLM-5.3-Flash With Full 1M Context on One DGX Station GB300

Transcript

Serve GLM-5.3-Flash With Full 1M Context on One DGX Station GB300 · Encrypted Ambition: Where Ambition Meets Encryption - Inside The Minds Of Disruptors.

• 0:14
  Host: Today we examine a GPU memory mishap that stalled a single-box AI lab on a DGX Station GB300.
• 0:21
  Analyst: It involved the GLM-5.3-Flash model from NVIDIA, a 191-gigabyte checkpoint that the team downloaded.
• 0:29
  Host: Petronella Technology Group ran this exact configuration on its own GB300 since September 25, 2026.
• 0:37
  Analyst: The model’s full 1,048,576-token context is a staggering 1.05 million tokens, but the real issue was a hidden page cache.
• 0:50
  Host: On the GB300, the GPU’s high-bandwidth memory sits on NUMA node one, while the system’s LPDDR memory is on node zero.
• 0:58
  Analyst: Heavy file I/O on the host, like downloading or copying the 191-gigabyte weights, fills node zero’s page cache.
• 1:07
  Host: When the kernel moves those pages into HBM, vLLM sees almost no free GPU memory and crashes.
• 1:13
  Analyst: The error message looks odd: it says it tried to allocate 151 GiB while reporting 159.43 GiB free.
• 1:23
  Host: That contradiction is a signature of the NUMA trap, and it can happen without you knowing.
• 1:28
  Analyst: The fix is surprisingly mechanical, involving three steps that keep the page cache out of the GPU memory.
• 1:35
  Host: First, before every start the pre-start script drops the page cache with sync and echo 1 to /proc/sys/vm/drop_caches.
• 1:43
  Analyst: That command forces the kernel to flush cached pages, freeing HBM for vLLM’s KV cache allocation.
• 1:52
  Host: Second, all bulk I/O operations run under numactl with membind zero, so they stay in LPDDR.
• 1:59
  Analyst: This includes the initial weight download with huggingface-cli and any future rsync or copy jobs that might fill the cache.
• 2:07
  Host: Third, the vLLM container itself is never pinned to node zero; pinning it causes CUDA to fail with Error 802.
• 2:14
  Analyst: By leaving the container free to use the entire machine, it can access the GPU memory it needs without restriction.
• 2:22
  Host: The service is managed by systemd, not Docker’s restart flag, which gives the pre-start script a chance to run each time.
• 2:29
  Analyst: The systemd unit starts the container with a short list of vLLM flags that set the context and concurrency.
• 2:37
  Host: Those flags include --max-model-len 1048576 and --max-num-seqs 256, allowing up to 256 sequences in flight.
• 2:48
  Analyst: GPU memory utilization is capped at 0.94, giving the system a 6% headroom for unexpected spikes.
• 2:56
  Host: Chunked prefill and prefix caching are turned on to handle very long prompts efficiently and reduce redundant work.
• 3:02
  Analyst: Tool calling is enabled with auto-tool-choice, and the parsers glm47 and glm45 match the model’s output format for structured calls.
• 3:13
  Host: The API surface is OpenAI-compatible, listening on port 8888 and exposing the model as glm-5.3-flash.
• 3:22
  Analyst: This compatibility lets existing client code redirect to the private endpoint with minimal changes to the base URL.
• 3:29
  Host: The KV cache uses fp8 precision, which comes from the checkpoint configuration and saves memory.
• 3:35
  Analyst: With fp8, the cache holds 5,933,986 tokens, enough for roughly 5.66 full 1-million-token requests.
• 3:47
  Host: So the service can support multiple long-context sessions simultaneously without exceeding GPU limits.
• 3:52
  Analyst: Startup takes about seven minutes from systemctl start to serving; weights load in about 77 seconds.
• 4:00
  Host: The remaining time is spent compiling and capturing CUDA graphs, which vLLM does once at launch for future requests.
• 4:07
  Analyst: If startup takes longer, the pre-flight check with grep FilePages on node one can reveal the page cache trap.
• 4:14
  Host: The restart policy in systemd restarts the service on failure every 60 seconds, capped at three starts per hour.
• 4:21
  Analyst: That cap prevents a crash loop from hammering the machine and losing logs, keeping the system stable.
• 4:27
  Host: When a container crashes, the pre-start script saves its logs before removing it, preserving evidence for debugging.
• 4:35
  Analyst: Docker itself has no restart flag in this setup, so systemd owns the entire restart lifecycle and ensures the pre-start runs.
• 4:44
  Host: The smoke test checks three things: model listing, a chat answer, and a get_weather tool call returning structured tool_calls.
• 4:52
  Analyst: Skipping the tool-call check leads to silent failures that break downstream agents, so the test is essential.
• 4:59
  Host: Petronella’s go-live record shows the first user decoding at 145.9 tokens per second, a solid baseline.
• 5:07
  Analyst: Time to first token for that single user is 0.061 seconds, giving a fluid interactive experience for end users.
• 5:16
  Host: When 32 users run concurrently, the aggregate throughput is 1,464 tokens per second, a respectable level.
• 5:24
  Analyst: That averages 55.6 tokens per second per user, with a 0.985-second median time to first token across the load.
• 5:33
  Host: No failures occurred at that load, proving the configuration can handle moderate concurrency without errors.
• 5:39
  Analyst: The peak aggregate measured was 2,517 tokens per second at 256 users, about 29 tokens per user, the ceiling of this setup.
• 5:50
  Host: Those numbers come from a burst test with 256-token answers and three repeats after a warm-up, ensuring consistency.
• 5:58
  Analyst: The long-context check involved needle retrievals at 968,377 prompt tokens, covering 90% of the maximum context length.
• 6:10
  Host: Each retrieval took between 33.5 and 37.0 seconds, confirming the model can process nearly a million tokens in a single request.
• 6:20
  Analyst: Quality gates before go-live included a tool-call battery of 38 out of 38 and an agent battery of 15 out of 15, both passing.
• 6:30
  Host: On the closed-book CMMC battery, the checkpoint scored 26 out of 35, the best result among models tested in that suite.
• 6:39
  Analyst: The total evaluation items reached 117 out of 127, indicating strong overall performance across diverse tasks.
• 6:48
  Host: However, the configuration has known limits that teams should be aware of before scaling.
• 6:53
  Analyst: Specifically, there is no speculative decoding; multi-token prediction fails to load with a tensor shape mismatch error.
• 7:01
  Host: A different checkpoint from RedHatAI supports multi-token prediction but loses measurable compliance recall, so it’s not recommended.
• 7:09
  Analyst: That checkpoint achieved 231 tokens per second for one user and 1,760 for 32 users on the same machine, but its recall dropped.
• 7:20
  Host: The RedHatAI checkpoint without multi-token prediction still reaches 144 tokens per second for one user, a modest drop.
• 7:27
  Analyst: Its 32-user throughput is 1,633 tokens per second, with prefill rates of 28,610, 33,076, and 32,511 tokens per second for 8K, 32K, and 128K prompts.
• 7:46
  Host: These prefill numbers are specific to the RedHatAI checkpoint, not the NVIDIA recipe, so they don’t directly compare.
• 7:53
  Analyst: Another limitation is the lack of API key authentication; vLLM binds to 0.0.0.0 with no auth by default.
• 8:03
  Host: Deploying it on a private network or behind a firewall with a reverse proxy is essential for security and compliance.
• 8:09
  Analyst: The recipe has only been tested on DGX Station GB300 hardware, so other GB300 systems should work but have not been measured.
• 8:19
  Host: For reference, the same model on a four-GPU H200 NVL server decodes at 238 tokens per second for one user.
• 8:28
  Analyst: That setup achieves 2,254 tokens per second aggregate at 32 users, faster than the single-box GB300 configuration.
• 8:38
  Host: On a four-unit GB10 pod, the model serves one user at 44.9 tokens per second, a lower throughput.
• 8:47
  Analyst: With two GB10 units, it supports only six sequences at 24.1 tokens per second per user, highlighting scaling limits.
• 8:56
  Host: Petronella recommends using at least four GB10 units for long-context workloads to reach acceptable performance.
• 9:02
  Analyst: The GB300’s advantage is its ability to hold the full one-million-token context on a single machine, ideal for branch deployments.
• 9:12
  Host: It’s ideal for desk-side or branch deployments that need that capacity without a multi-GPU cluster, simplifying operations.
• 9:19
  Analyst: Choosing vLLM over simpler servers is justified when you need structured tool calling, concurrency, and long-context support.
• 9:28
  Host: For smaller footprints, a multi-GPU RTX 6000 Pro setup might be more appropriate, depending on the workload.
• 9:35
  Analyst: If you’re evaluating a GB300 or similar hardware, the recipe provides a proven starting point that has run since September 2026.
• 9:44
  Host: But you should also consider whether the memory trap will reappear if you run heavy I/O after deployment, especially during updates.
• 9:52
  Analyst: The pre-flight check with grep FilePages can catch that before you start the container, preventing silent failures.
• 9:59
  Host: Organizations should also plan for authentication and network isolation to protect the endpoint from external access.
• 10:06
  Analyst: Additionally, monitoring the container logs, which are preserved by the pre-start script, helps diagnose future crashes and performance regressions.
• 10:15
  Host: Now that we understand the mechanics and limits, what steps should companies take to avoid this trap in their own deployments?
• 10:22
  Host: First, verify that the GB300’s page cache is empty before you start the container.
• 10:27
  Analyst: Run the grep FilePages command against node1’s meminfo; the output should be near zero, otherwise heavy I/O has already populated HBM with file pages.
• 10:39
  Host: If you see non-zero values, drop the cache with sync and echo 1 into drop_caches.
• 10:44
  Analyst: That command flushes the kernel’s page cache, freeing HBM for the vLLM KV cache allocation.
• 10:51
  Host: Next, download the 191-gigabyte checkpoint with numactl bind to node 0.
• 10:56
  Analyst: Use numactl --membind=0 huggingface-cli download nvidia/GLM-5.3-Flash-NVFP4 to keep file pages in LPDDR memory.
• 11:10
  Host: Avoid downloading with plain huggingface-cli; the kernel would then place pages in HBM and trigger the trap.
• 11:12
  Analyst: After the download, install the service files: copy glm53-flash-prestart.sh to /usr/local/sbin and glm53-flash.service to /etc/systemd/system.
• 11:29
  Host: Reload systemd and enable the unit so that it starts on boot.
• 11:33
  Analyst: The unit runs the pre-start script, which clears the cache and preserves prior logs before removing the container.
• 11:39
  Host: It also enforces the restart policy: 60-second intervals, maximum three starts per hour.
• 11:46
  Analyst: That policy stops an out-of-memory loop from hammering the machine and keeps logs accessible.
• 11:52
  Host: When you start the service, give it about seven minutes before it reports ready.
• 11:56
  Analyst: The first 77 seconds load weights, then the remaining time is spent compiling CUDA graphs.
• 12:02
  Host: After startup, run the smoke test to confirm the model is listed, returns a chat answer, and returns a structured get_weather tool call.
• 12:10
  Analyst: The smoke test script checks three endpoints; failing any of them indicates a configuration issue.
• 12:16
  Host: If the smoke test passes, you can be confident the service is healthy.
• 12:19
  Analyst: Now, for ongoing operations, keep all bulk I/O off node 1 by wrapping downloads in numactl or docker run with cpuset-mems 0.
• 12:32
  Host: Don’t pin the vLLM container to node 0; that causes CUDA initialization to fail with Error 802.
• 12:39
  Analyst: The container should have access to both nodes so it can allocate GPU memory freely.
• 12:45
  Host: Monitor the container’s logs regularly; they are preserved by the pre-start script before each restart.
• 12:51
  Analyst: If you notice repeated out-of-memory errors, re-run the grep FilePages check to verify HBM is clear.
• 12:58
  Host: Also watch the token throughput metrics; one user at 145.9 tok/s and 32 users at 1,464 tok/s are expected benchmarks.
• 13:09
  Analyst: If throughput drops significantly, check GPU memory utilization; the service is configured to use 94% of available memory.
• 13:18
  Host: Remember that the KV cache holds 5,933,986 tokens; only about 5.66 full-million-token requests can be served concurrently.
• 13:30
  Analyst: If your workload exceeds that, you’ll see requests queue rather than fail, so plan capacity accordingly.
• 13:36
  Host: Consider adding more GB10 units if you need higher aggregate throughput; a four-unit pod can support more sequences.
• 13:43
  Analyst: But if you require the full 1M context on a single machine, the GB300 is the only tested solution.
• 13:51
  Host: For authentication, bind the service to 0.0.0.0 but place a reverse proxy with API key enforcement in front.
• 13:58
  Analyst: Without authentication, anyone on the network can consume GPU resources and read model outputs.
• 14:05
  Host: Deploy the service behind a corporate firewall or private network to isolate it from external traffic.
• 14:11
  Analyst: If you need to run updates, perform them during low-usage windows to avoid triggering the page-cache trap.
• 14:18
  Host: Use the pre-flight grep before any update to ensure node 1’s FilePages remain low.
• 14:23
  Analyst: For scaling, keep the --max-num-seqs flag at 256 if you anticipate many concurrent long prompts.
• 14:33
  Host: If your use case is short prompts, you might lower --max-num-seqs to reduce memory footprint.
• 14:38
  Analyst: However, lowering --max-model-len changes the service’s purpose; only trim it if you truly need less context.
• 14:47
  Host: When you notice a sudden increase in GPU memory consumption, verify that no background process is writing large files to disk.
• 14:54
  Analyst: Large rsync or backup jobs can inadvertently fill HBM with file pages, triggering the same out-of-memory error.
• 15:02
  Host: To avoid that, schedule such jobs on node 0 or disable them during peak inference times.
• 15:08
  Analyst: Also, keep the vLLM container’s CPU affinity broad; pinning it to a single CPU socket can reduce scheduling flexibility.
• 15:17
  Host: If you observe CUDA graph capture failures, ensure the container has sufficient GPU memory headroom.
• 15:22
  Analyst: The 0.94 GPU memory utilization setting leaves enough headroom for graph compilation and runtime spikes.
• 15:30
  Host: For tool-calling workloads, validate that the parser names glm47 and glm45 match the container’s build.
• 15:38
  Analyst: If you upgrade the vLLM image, re-run the smoke test to confirm parser compatibility.
• 15:44
  Host: Avoid relying on Docker’s restart policy; systemd provides more granular control and a consistent pre-start step.
• 15:51
  Analyst: Docker’s --restart unless-stopped would skip the pre-start script, leaving the page-cache trap unresolved.
• 15:58
  Host: If you need to recover from a crash loop, use systemctl reset-failed before restarting the unit.
• 16:04
  Analyst: After resetting, double-check the FilePages count; a non-zero value indicates a persistent I/O issue.
• 16:12
  Host: When adding more GPUs, remember the GB10 pods have lower per-GPU throughput, especially with only two units.
• 16:19
  Analyst: A four-unit GB10 pod serves one user at 44.9 tok/s, while two units can handle only six sequences at 24.1 tok/s each.
• 16:31
  Host: For higher throughput, consider a four-GPU H200 NVL server; it achieves 238 tok/s for one user and 2,254 tok/s for 32 users.
• 16:43
  Analyst: But that configuration requires multi-GPU networking and is not a single-node solution like the GB300.
• 16:50
  Host: If you need the full 1M context and a single-machine deployment, the GB300 recipe is the only proven path.
• 16:57
  Analyst: When planning a new deployment, start with the provided repository, copy the service files, and follow the install script exactly.
• 17:05
  Host: Don’t skip the pre-start script; it is essential for clearing the cache before every start.
• 17:11
  Analyst: Also, avoid hard-coding the container’s image tag; pin by digest to ensure reproducibility.
• 17:18
  Host: If you need to upgrade the image, update the digest and re-run all tests before production use.
• 17:24
  Analyst: For monitoring, export vLLM metrics to Prometheus; look at token per second, queue length, and memory usage.
• 17:33
  Host: Set alerts for when token throughput drops below 100 tok/s for one user, indicating a potential memory issue.
• 17:39
  Analyst: Also alert on unexpected restart events; a spike in restarts often signals the page-cache trap re-appearing.
• 17:47
  Host: When scaling to multiple users, keep an eye on the 55.6 tok/s per user benchmark; falling below that suggests contention.
• 17:55
  Analyst: If you need to support more than 32 concurrent users, test at 256 users to verify the 2,517 tok/s peak.
• 18:05
  Host: Remember that the 256-sequence limit is enforced by the --max-num-seqs flag; exceeding it will queue requests.
• 18:13
  Analyst: If you need speculative decoding, note that the current checkpoint does not support multi-token prediction.
• 18:20
  Host: A different RedHatAI checkpoint offers that feature but at the cost of closed-book recall, so choose based on your use case.
• 18:27
  Analyst: If recall is critical, stick with the nvidia checkpoint and accept the single-token decoding speed.
• 18:34
  Host: For branch deployments, the GB300’s single-machine design reduces network complexity and maintenance overhead.
• 18:40
  Analyst: However, you must still enforce network isolation; the service listens on all interfaces by default.
• 18:47
  Host: Deploy a reverse proxy like NGINX or Envoy to add TLS, rate limiting, and API key authentication.
• 18:55
  Analyst: That layer also provides load balancing if you later add more GB300 machines behind a cluster.
• 19:02
  Host: When you add more machines, keep each one configured identically to ensure consistent performance.
• 19:08
  Analyst: Synchronize the model weights across all nodes using the same numactl download procedure to avoid cache traps.
• 19:15
  Host: If you run into memory fragmentation, consider rebooting the host to clear the kernel’s page cache.
• 19:21
  Analyst: Rebooting is a quick way to reset the NUMA topology, though it is less desirable than graceful restarts.
• 19:28
  Host: For development, use the smoke test after every change to confirm the service remains healthy.
• 19:34
  Analyst: Automate the smoke test in your CI pipeline; it catches regressions before they hit production.
• 19:40
  Host: Finally, document the entire deployment process, including the pre-start script, restart policy, and monitoring setup.
• 19:46
  Analyst: Documentation ensures that new operators can recover from the page-cache trap without hunting for obscure logs.
• 19:54
  Host: By following these steps, organizations can deploy the GLM-5.3-Flash model reliably on a GB300 and avoid the common pitfalls.
• 20:03
  Analyst: They’ll also be able to scale appropriately, maintain security, and monitor performance to meet their business needs.
• 20:10
  Host: Now that the recipe is clear, let's think about what that means for the day-to-day ops of a business.
• 20:16
  Analyst: The first implication is that the 191-gigabyte checkpoint is a one-time cost; once you have it locally, you avoid repeated downloads that trigger the page-cache trap.
• 20:26
  Host: So you can schedule the initial download during a maintenance window and keep the weights on a persistent volume.
• 20:32
  Analyst: Exactly. And because the KV cache holds 5,933,986 tokens, you can comfortably serve up to five or six full-million-token requests concurrently before you hit the 5.66 limit.
• 20:48
  Host: If a team needs more than that concurrency, what do they do?
• 20:51
  Analyst: They either increase the number of GB300 nodes or reduce the maximum sequence length to free up KV space, but the recipe is tuned for full context, so changing that changes the service's purpose.
• 21:04
  Host: What about the 7-minute warm-up? That seems long for a small office.
• 21:09
  Analyst: The weight load takes about 77 seconds, and the remaining 6 minutes are for CUDA graph capture and compilation; you can’t reduce that without sacrificing the pre-prefill and prefix caching that keep latency low.
• 21:22
  Host: So the startup is a known cost that you can plan for.
• 21:25
  Analyst: Right. And if a restart is needed, systemd will handle it automatically, but you should monitor the restart count to catch a loop early.
• 21:33
  Host: What are the most common mistakes people make when deploying this setup?
• 21:37
  Analyst: Three that we see: starting the container manually and skipping the pre-start cache drop, pinning the container to node 0, and forgetting to run bulk downloads with numactl.
• 21:48
  Host: Those all trigger the same out-of-memory symptom, correct?
• 21:51
  Analyst: Yes, the self-contradicting error message about 159.43 GiB free but 151 GiB allocation failing is the hallmark of the trap.
• 22:03
  Host: How can operators detect the trap before it kills the service?
• 22:07
  Analyst: Run a quick grep on /sys/devices/system/node/node1/meminfo for FilePages; if it's not close to zero, you’ve got stale page cache in HBM.
• 22:39
  Host: And if it is non-zero, you should drop the cache before starting again.
• 22:43
  Analyst: Exactly. The pre-start script does that automatically, but if you bypass it you’ll hit the trap.
• 22:49
  Host: What about security, especially since the API is open by default?
• 22:53
  Analyst: You should put the service behind a reverse proxy that enforces TLS and API key or token authentication; otherwise anyone on port 8888 can consume GPU resources.
• 23:06
  Host: That proxy can also do rate limiting, which is handy for protecting against misuse.
• 23:11
  Analyst: And it gives you a single point to log requests, which is useful for compliance audits.
• 23:16
  Host: Speaking of compliance, how does the model perform on closed-book tests?
• 23:21
  Analyst: On the CMMC battery it scored 26 of 35, the best among models we measured, and the total evaluation items hit 117 of 127.
• 23:32
  Host: So it’s suitable for organizations that need high recall of factual content.
• 23:37
  Analyst: Yes, but remember the no-speculative-decoding limitation; if you need multi-token prediction for speed, you’ll need a different checkpoint that may trade off accuracy.
• 23:48
  Host: So there’s a trade-off between throughput and compliance.
• 23:51
  Analyst: Precisely. The RedHatAI checkpoint offers higher throughput but lower recall, so you must align the choice with business priorities.
• 24:00
  Host: How do you monitor the service once it’s live?
• 24:02
  Analyst: Use systemctl status to check health, journalctl for recent logs, and docker logs for container output; we also recommend sending metrics to a monitoring system like Prometheus.
• 24:14
  Host: What metrics are most valuable?
• 24:16
  Analyst: Token throughput per user, time to first token, and the number of concurrent sequences; also watch for any OOM events in the logs.
• 24:26
  Host: If the service starts to slow, what steps should you take?
• 24:29
  Analyst: First, check the current number of active sequences; if you’re near 256, you’re at the concurrency ceiling. Next, examine the cache usage; if you’re close to 5.93 million tokens, you’re near the KV limit.
• 24:44
  Host: Could you throttle user requests to stay within limits?
• 24:47
  Analyst: Yes, you can enforce a soft limit in your application layer or use the proxy to cap request rates per IP or API key.
• 24:56
  Host: What about scaling horizontally?
• 24:58
  Analyst: Add another GB300, keep the same weight download process, and use a load balancer in the proxy layer; the service remains identical across nodes, so you just increase aggregate throughput linearly.
• 25:11
  Host: Do you recommend any particular load-balancing algorithm?
• 25:14
  Analyst: Round-robin is simple and effective when each node has identical capacity; for uneven loads you could use least-conn or weighted round-robin based on recent throughput.
• 25:26
  Host: What about caching the model weights across nodes to avoid disk duplication?
• 25:30
  Analyst: You can mount a shared NFS or use a distributed file system, but remember that each node must still drop the page cache before starting to avoid the trap.
• 25:41
  Host: So the pre-start script must run on every node.
• 25:44
  Analyst: Exactly. It’s a small script but critical for stability.
• 25:48
  Host: What if a node fails during a request?
• 25:50
  Analyst: The proxy can detect the failure and redirect the request to another healthy node; vLLM will handle the new context from scratch, but the latency will include a warm-up.
• 26:02
  Host: Is there a way to preserve context across restarts?
• 26:05
  Analyst: Not with the current checkpoint; you would need a stateful cache or a different architecture that supports checkpointing of KV state.
• 26:13
  Host: What about tool calls-how reliable are they?
• 26:16
  Analyst: The smoke test checks for a structured get_weather tool call; on our deployment we hit 38 of 38, so tool calls are handled correctly as long as you use the specified parser names glm47 and glm45.
• 26:32
  Host: If a new tool is added, do you need to adjust the parser?
• 26:35
  Analyst: Yes, you’d need to update the tool-call parser or add a custom parser that matches the new tool’s output format.
• 26:43
  Host: How do you handle version upgrades of the checkpoint?
• 26:46
  Analyst: Download the new checkpoint with numactl, drop the cache, and restart the service; the pre-start script will preserve old logs and apply the new weights.
• 26:56
  Host: Does the new checkpoint affect the KV token count?
• 26:59
  Analyst: Potentially; the new checkpoint may change the fp8 configuration, so you’d need to recalculate the cache capacity and adjust max-num-seqs if necessary.
• 27:09
  Host: What if the new checkpoint introduces a different parser name?
• 27:13
  Analyst: You’d have to update the service flags to match the new parser; otherwise the model will output raw text that the client cannot parse.
• 27:21
  Host: What questions do listeners frequently ask about this deployment?
• 27:24
  Analyst: Common ones include: how to handle more than five concurrent million-token requests, whether the setup works on other GPUs, and how to secure the endpoint if it’s exposed to the internet.
• 27:36
  Host: And the answers are essentially what we’ve covered: scale horizontally, test on the target hardware, and add a proxy with authentication.
• 27:44
  Analyst: Correct. Another question is about the impact of speculative decoding; we advise against it because it’s unsupported for this checkpoint.
• 27:52
  Host: What if an organization wants to use speculative decoding anyway?
• 27:56
  Analyst: They’d need to switch to a checkpoint that supports it, like the RedHatAI NVFP4, but they should be prepared for reduced compliance recall.
• 28:05
  Host: So the trade-off is clear.
• 28:07
  Analyst: Yes, and that trade-off should be documented in the architecture plan.
• 28:11
  Host: Do you have any final recommendations for operators who are just starting?
• 28:15
  Analyst: Start with the single-node recipe, verify the smoke test, monitor metrics, and only then consider scaling; also keep the pre-start script and systemd unit in version control so you can roll back quickly.
• 28:29
  Host: And always remember to keep the page cache clean before each start.
• 28:32
  Analyst: That’s the single most important step to avoid the out-of-memory loop that can be hard to diagnose otherwise.
• 28:39
  Host: What about resource limits-do you recommend cgroups?
• 28:42
  Analyst: Using cgroups to cap CPU and memory ensures the container doesn’t starve the host, but the GPU memory is managed by vLLM through the 0.94 utilization flag.
• 28:54
  Host: So you’d set CPU shares to match the workload?
• 28:57
  Analyst: Exactly, and you can use docker run --cpus to limit the container to a fraction of the CPU, but leave GPU allocation to vLLM.
• 29:07
  Host: That balances host stability with model performance.
• 29:11
  Analyst: And don’t forget to keep the system updated; kernel patches can affect NUMA behavior, so monitor release notes for any changes that might re-introduce the page-cache trap.
• 29:21
  Host: Any last technical detail that could trip people up?
• 29:24
  Analyst: Make sure you’re not using the Docker restart flag; systemd must own the restart policy, otherwise the pre-start script won’t run on a crash, and the trap will reappear.
• 29:35
  Host: Got it-systemd is the single source of truth for restarts.
• 29:38
  Analyst: With a well-documented, monitored, and secured deployment, your organization can reliably serve GLM-5.3-Flash at full context on the DGX Station GB300.
• 29:51
  Host: Thank you for the detailed walkthrough and practical guidance.