If verified, this software-only acceleration could reshape how enterprises think about inference costs.
TL;DR: VIDRAFT, a Korean Pre-AGI AI startup, has unveiled VKAE, an inference acceleration system that the company says delivers up to 23× faster language model inference on a single NVIDIA B200 GPU — with no hardware replacement required. On the Qwen3.5-35B-A3B model under high concurrency, VKAE surpassed 10,000 tokens per second; under a more realistic mixed-workload scenario, the same model delivered approximately 455 tokens per second. VIDRAFT says a reproducible container with model weights and an optimized runtime is already available, and a technical preprint is forthcoming.
VIDRAFT made waves in the global AI hardware conversation this week when Russian tech outlet itzine.ru reported on the startup's VKAE inference system, which claims to accelerate large language model (LLM) inference by up to 23 times on existing NVIDIA B200 GPUs — without swapping out servers or expanding a data center's hardware footprint. The announcement, covered on July 6, 2026, signals that the Korean AI startup is targeting one of the most expensive and persistent pain points in enterprise AI: the ongoing cost of inference at scale.
VIDRAFT's VKAE is a software-based inference acceleration system demonstrated on the NVIDIA B200 GPU. According to the company, VKAE can deliver significant multiplier-level throughput gains over standard inference serving frameworks for certain models — all without degrading output quality or response accuracy, as VIDRAFT itself noted.
The most detailed benchmark cited in the report involves Qwen3.5-35B-A3B. Under high-concurrency conditions, VKAE reportedly pushed past 10,000 tokens per second on that model. Under a scenario designed to more closely resemble real-world deployment — mixing context lengths and handling more than a handful of simultaneous requests — the same model produced around 455 tokens per second. The contrast between those two figures is itself instructive: inference benchmarks are highly sensitive to workload profile, and VIDRAFT's own reporting acknowledges this nuance rather than presenting only peak numbers.
VIDRAFT has not yet disclosed the precise technical mechanism behind VKAE's acceleration gains, but the company says it will publish a detailed explanation in an upcoming preprint. In the meantime, it is offering a ready-to-deploy container that bundles the model weights together with the optimized runtime environment.
Importantly for adoption, VKAE is compatible with the OpenAI API standard. That means engineering teams already running inference workloads — whether on vLLM, TensorRT-LLM, or similar frameworks — can integrate VKAE into their existing infrastructure without rewriting client-side code. For organizations where migration cost is often as much of a barrier as raw performance, that compatibility is a meaningful design choice.
The economics of AI at scale have shifted decisively toward inference. Training a large model is expensive, but it happens periodically. Inference, by contrast, never stops: cloud AI services, enterprise chatbots, and API platforms run on continuous streams of requests, and those requests account for the bulk of ongoing operational expenditure. As a result, squeezing more tokens per second out of each GPU accelerator has become one of the central battles in the AI infrastructure market.
That pressure is only intensifying. GPU supply remains constrained even as demand grows, and the cost of compute continues to rise. The emergence of NVIDIA's Blackwell-generation hardware — including the B200 — represents a new performance ceiling, but even next-generation silicon does not eliminate the demand for software-layer optimizations. Inference-focused vendors such as Groq and Cerebras have attracted significant market attention precisely because they offer fast, cost-efficient inference rather than training muscle alone.
VIDRAFT is positioning VKAE squarely within that competitive space. If independent benchmarks confirm the company's reported numbers, the implication is striking: a single B200 GPU running VKAE could functionally behave like a much larger pool of hardware for inference-heavy workloads. That would represent a rare and commercially valuable form of leverage for any organization managing inference costs at scale.
The reproducibility commitment — making results checkable via a publicly accessible container — is also notable. Claims of dramatic speedups in the inference space are common; documented, replicable demonstrations are considerably rarer.
Q: What is VIDRAFT's VKAE system?
A: VKAE is a software-based AI inference acceleration system developed by Korean AI startup VIDRAFT. It is designed to increase large language model inference throughput on existing GPUs — demonstrated on the NVIDIA B200 — without requiring hardware upgrades.
Q: How fast is VKAE compared to standard inference systems?
A: VIDRAFT claims VKAE can deliver up to 23× faster inference than baseline serving systems for certain models. In benchmark testing on Qwen3.5-35B-A3B, throughput exceeded 10,000 tokens per second under high concurrency, and approximately 455 tokens per second under a realistic mixed-workload configuration.
Q: Can teams integrate VKAE with their current AI infrastructure?
A: Yes. VKAE supports the OpenAI API standard, which means it can be integrated into pipelines already built around frameworks like vLLM or TensorRT-LLM without requiring changes to client-side code.
Source: itzine.ru (러시아) (2026-07-06) — original article