Beyond LLM Decode: Why the Future of AI Inference Needs More Than GPUs

The Next Evolution of AI Inference

For years, the AI industry has relied heavily on GPUs to accelerate increasingly large neural networks. NVIDIA GPUs have become the foundation of modern AI infrastructure, offering massive parallel computing capability, high memory bandwidth, and an extensive software ecosystem. However, as AI applications evolve from simple chatbots into autonomous agents, robotics systems, and real-time decision-making platforms, the requirements of inference are changing. The industry is beginning to recognize that not every stage of AI inference benefits equally from the same processor architecture.

One increasingly important approach is prefill/decode disaggregation: separating the two major phases of LLM inference and executing them on independently optimized computing resources. This creates an opportunity to combine NVIDIA GPUs with specialized AI processors. At LENZO, we believe our TokenProcessor™, based on our proprietary CPU-Grounded Linear Array (CGLA) architecture, is particularly well suited to this emerging computing model. But there is an important distinction.

TokenProcessor is not simply a decode accelerator. It is a programmable dataflow processor capable of accelerating both AI inference and the non-AI computations that increasingly dominate the execution paths of autonomous agents.

This article explores why that distinction matters.

1. Why Prefill and Decode Should Be Treated Differently

LLM inference consists of two fundamentally different computational phases.

Prefill: A Parallel Computing Problem

During prefill, an LLM processes the input prompt and generates the key-value (KV) cache used during subsequent token generation. Because multiple input tokens can be processed together, this phase exposes substantial parallelism. Large matrix multiplications dominate much of the computation, making prefill particularly suitable for GPUs.

NVIDIA GPUs are highly effective at these operations because they provide:

• Massive parallel execution through tensor-oriented computing units

• High memory bandwidth

• Efficient processing of large matrix operations

• Mature software infrastructure for LLM inference

For sufficiently large workloads, GPUs can achieve high utilization by processing many operations concurrently. In other words, prefill is primarily a throughput-oriented computational problem.

Decode: A Sequential, Memory-Sensitive Problem

Decode behaves differently. Once the KV cache has been generated, the LLM produces output tokens autoregressively. Each new token depends on the preceding tokens. This introduces a sequential dependency that fundamentally limits parallelism across the output sequence. Although individual decode steps still contain parallel operations, the overall process must advance token by token. For many practical workloads, particularly small-batch inference, decode becomes highly sensitive to memory bandwidth, data movement, and execution latency. Consider a simplified example. An LLM generating 100 output tokens must execute approximately 100 sequential decoding steps.

Even if the processor has enormous theoretical computing throughput, it cannot eliminate the autoregressive dependency between those steps. Furthermore, each decoding step may require accessing a substantial portion of the model weights and the growing KV cache. This creates a very different optimization objective. Instead of maximizing the number of matrix operations executed simultaneously, the system must minimize the time and energy required to generate each additional token.

Prefill rewards massive parallelism. Decode rewards efficient, low-latency execution with minimal data movement.

These two objectives do not necessarily require the same hardware architecture.

2. Why the Industry Is Moving Toward Prefill/Decode Disaggregation

Traditionally, both prefill and decode have been executed on the same GPU infrastructure. This approach is straightforward and benefits from NVIDIA's mature inference software ecosystem. However, it also creates several challenges.

Different Resource Requirements

Prefill and decode place different demands on computing resources. Prefill typically benefits from high computational throughput. Decode, especially at small batch sizes, often depends more heavily on memory access efficiency and latency. When both phases share the same resources, optimizing one phase can compromise the other.

Resource Contention

Prefill workloads can involve large bursts of computation. Decode workloads require continuous, latency-sensitive execution. Combining them on the same GPU can introduce interference, particularly when multiple requests with different input and output lengths are processed simultaneously. This can make it more difficult to deliver predictable time-to-first-token (TTFT) and inter-token latency.

Independent Scaling

An application processing long documents may require substantial prefill capacity but relatively little decode capacity. A conversational application generating lengthy responses may have the opposite characteristics. Separating prefill and decode allows infrastructure operators to scale these resources independently. This is one of the primary motivations behind disaggregated inference architectures.

Specialized Hardware

Once prefill and decode are separated, another possibility emerges.

Why should both phases necessarily run on GPUs?

A heterogeneous architecture could assign prefill to NVIDIA GPUs while delegating decode to a processor specifically optimized for token-by-token execution. Such a system could potentially improve energy efficiency, latency, or infrastructure cost, depending on the workload and implementation. This is the opportunity LENZO is pursuing. It is important to recognize that disaggregation itself is not automatically beneficial. Transferring KV-cache data between processors introduces communication overhead, and heterogeneous systems require careful scheduling and memory management. The benefits must therefore be evaluated at the system level, not merely by comparing processor specifications.

3. TokenProcessor: A Different Architecture for LLM Decode

LENZO's TokenProcessor is based on CGLA, a proprietary dataflow computing architecture developed to minimize unnecessary data movement while maintaining programmability. Unlike conventional architectures that rely primarily on instruction-driven execution, CGLA organizes computation around the movement of data through a linear array of processing elements. This architecture offers several characteristics relevant to LLM decoding.

3.1 Streaming Dataflow Execution

CGLA is designed to execute computation through dataflow-oriented pipelines. Processing elements can receive data, perform operations, and forward results to subsequent processing stages. This approach is particularly attractive for computations that can be organized into streaming execution patterns. In LLM decoding, where the same computational structures are repeatedly executed for successive tokens, efficient dataflow execution can help reduce unnecessary control and data movement overhead.

3.2 Minimizing Data Movement

Data movement is one of the major contributors to power consumption in modern processors. For memory-intensive inference workloads, moving model weights and intermediate data can consume substantial energy. TokenProcessor's architecture emphasizes localized communication, narrow data paths, and near-memory computation. These design choices aim to reduce the energy spent moving data rather than performing useful computation. This is especially relevant for decode workloads, where the arithmetic intensity can be relatively low at small batch sizes.

3.3 Batch=1 and Low-Latency Inference

GPU inference systems often improve throughput by batching multiple requests together. However, batching can introduce additional latency and may not always be appropriate for interactive applications. Autonomous agents, robots, and real-time systems frequently require immediate responses to individual events. TokenProcessor is designed with low-latency, small-batch execution in mind. Rather than relying on large batches to achieve efficient execution, CGLA seeks to provide useful computational efficiency even when processing individual inference streams. This makes Batch=1 performance an important design objective.

3.4 Programmability Beyond Fixed-Function Accelerators

Some specialized AI accelerators achieve efficiency by restricting the operations or execution patterns they support. TokenProcessor takes a different approach. CGLA uses a programmable array of processing elements, allowing different computational kernels to be mapped onto the architecture. Its linear topology also constrains the mapping problem compared with more general two-dimensional coarse-grained reconfigurable arrays. This is intended to support relatively fast compilation while retaining flexibility across workloads. The result is an architecture designed not only for efficient neural network inference, but also for a broader range of data-intensive computation.

4. Combining NVIDIA GPUs and TokenProcessor

A heterogeneous inference system could combine NVIDIA GPUs for prefill with TokenProcessor for decode. The basic execution flow is straightforward.

Step 1: Prefill on NVIDIA GPU

The GPU receives the input prompt, executes the transformer prefill phase, and generates the initial KV cache.

Step 2: KV-Cache Transfer

The system transfers the required KV-cache state and associated execution metadata to the TokenProcessor-based decode environment.

Step 3: Decode on TokenProcessor

TokenProcessor executes the autoregressive decoding loop, generating output tokens sequentially.

Step 4: Output and Agent Execution

Generated tokens are delivered to the application, where they may trigger further operations such as tool calls, searches, or actions. This architecture allows each processor to focus on the workload characteristics for which it is best suited. The GPU handles large-scale parallel computation. TokenProcessor targets efficient, low-latency token generation. However, several engineering challenges must be addressed to make such a system practical. These include KV-cache format compatibility, model partitioning, quantization consistency, communication latency, scheduling, and synchronization between the two processors.

For short outputs, the overhead of transferring the KV cache may outweigh the benefits of moving decode to another device. For longer outputs or repeated decode operations, the balance may be different. Therefore, the key performance metric is not simply tokens per second. It is the end-to-end latency, throughput, energy consumption, and cost of the complete heterogeneous inference system. This is an important area for future system-level evaluation.

5. The Bigger Opportunity: AI Agents Need More Than LLM Acceleration

So far, we have discussed TokenProcessor primarily as a decode accelerator. But this is only part of the story. As AI evolves toward autonomous agents, a growing portion of application execution takes place outside the neural network itself. Consider an AI agent performing a complex task.

It may need to:

1. Interpret a user request using an LLM.

2. Search documents or structured data.

3. Traverse a graph of related entities.

4. Compare or filter candidate results.

5. Execute application-specific algorithms.

6. Call external tools and process their outputs.

7. Perform additional LLM inference.

8. Repeat the process until the task is complete.

Only some of these operations involve transformer inference. Many are traditionally executed on CPUs. In fact, the overall responsiveness of an agent may depend as much on these surrounding operations as on the LLM itself. This is where TokenProcessor's broader programmability becomes particularly interesting.

Text Search and Data Filtering

AI agents frequently search large collections of documents, logs, or structured records. These operations may involve pattern matching, filtering, comparisons, and streaming data processing. Such workloads do not necessarily require tensor-oriented computing. They often benefit from efficient data movement and pipelined execution. TokenProcessor's programmable dataflow architecture provides an opportunity to accelerate suitable search and filtering kernels.

Graph Processing

Graph-based computation is increasingly relevant to AI agents. Knowledge graphs, dependency analysis, workflow planning, and relationship discovery can all involve graph-oriented algorithms. These workloads often feature irregular memory access and control flow, making them challenging for traditional GPU execution. While not every graph algorithm maps efficiently to CGLA, suitable graph-processing kernels can potentially benefit from programmable dataflow execution.

Cryptographic and Hash Computation

AI agents increasingly interact with distributed services, secure data pipelines, and authenticated information. Cryptographic and hash operations are therefore common components of modern computing workflows. TokenProcessor's architecture is not restricted to neural network arithmetic. LENZO has also explored computational kernels such as SHA-based hashing, demonstrating the broader applicability of CGLA to non-AI algorithms.

Signal Processing and Physical AI

Physical AI systems must process more than language. They interact with cameras, microphones, sensors, and robotic control systems. Signal-processing operations such as FFT and digital filtering are essential in many of these applications. These workloads are often well suited to streaming execution. TokenProcessor's ability to support both neural network computation and suitable signal-processing kernels creates opportunities for integrated Physical AI acceleration.

6. From an AI Accelerator to an Agentic Computing Processor

This leads to a broader architectural question. If an AI agent spends substantial time executing both neural network inference and non-neural-network computation, should these workloads always be assigned to separate GPUs and CPUs? Traditional computing systems generally divide responsibilities between the two. GPUs accelerate tensor computation. CPUs handle orchestration, control flow, searching, graph operations, and general-purpose execution. This division works well in many situations, but it can also introduce data transfers and synchronization overhead. TokenProcessor offers a different possibility.

By supporting multiple classes of computational kernels on a programmable dataflow architecture, it could accelerate selected workloads that would otherwise execute on CPUs. Imagine an AI agent performing document analysis. The GPU processes a large input prompt. TokenProcessor generates output tokens. The agent then performs text matching, data filtering, and graph-related computation. Instead of automatically sending every non-AI operation back to the CPU, suitable kernels could execute on TokenProcessor. The result could be a more integrated execution pipeline with fewer unnecessary transfers between computing resources. Importantly, this does not mean replacing the CPU entirely.

General-purpose operating systems, complex branching, dynamic scheduling, network stacks, and many other tasks will continue to benefit from conventional CPUs. The opportunity is to offload selected computationally intensive, dataflow-friendly operations that are traditionally assigned to CPUs. This distinction is essential. TokenProcessor is not intended to be a universal replacement for GPUs or CPUs. It is intended to complement them.

7. Exploring CGLA Through LENZO's Tutorials

One of LENZO's objectives is to make CGLA accessible to developers before dedicated ASIC silicon becomes available. Our software development environment and tutorials are intended to demonstrate how different classes of computational workloads can be expressed and executed using the CGLA architecture. The broader tutorial collection includes examples spanning several computational domains.

These examples illustrate an important architectural principle. The same underlying processor architecture can be applied to computational problems that are traditionally associated with different categories of hardware. This flexibility is particularly relevant to the next generation of AI applications. Rather than building a processor that only executes one specific neural network operation, LENZO is developing a programmable architecture intended to support a wider range of workloads.

For developers interested in the separation of LLM inference phases, our dedicated tutorial provides a starting point:

Explore the Prefill/Decode Split Tutorial The tutorial should be understood as a technical exploration of the execution model, rather than a claim of production-ready heterogeneous GPU-to-ASIC performance.

8. What We Need to Measure Next

The potential advantages of heterogeneous inference are compelling, but architectural claims must ultimately be validated through measurements. For a GPU-prefill and TokenProcessor-decode system, several metrics are especially important.

Time to First Token (TTFT)

How quickly can the system produce its first output token, including prefill execution and any transfer overhead?

Inter-Token Latency (ITL)

How quickly can TokenProcessor generate subsequent tokens, particularly at Batch=1?

Energy per Generated Token

How much energy does the complete system consume for each output token?

KV-Cache Transfer Overhead

How much latency and bandwidth are required to move the cache between devices?

End-to-End Agent Latency

How much time is spent in neural network inference versus search, graph processing, data transformation, and other operations?

Performance per Watt Across the Entire Workflow

Does accelerating both decode and selected non-AI kernels improve total application efficiency? These are the measurements that will determine where heterogeneous architectures provide meaningful advantages over GPU-only systems. At LENZO, our goal is to validate these benefits progressively through software, FPGA-based evaluation, and ultimately ASIC implementations.

9. The Future Is Heterogeneous—and Programmable

The future of AI computing is unlikely to be defined by a single processor architecture. NVIDIA GPUs will continue to play an essential role in large-scale parallel computation, including training and prefill. CPUs will remain indispensable for general-purpose execution and system orchestration. But increasingly sophisticated AI applications create opportunities for specialized processors that can execute latency-sensitive and data-intensive workloads more efficiently. LLM decoding is one such opportunity. Agentic computation extends that opportunity much further.

LENZO's TokenProcessor is designed around a simple principle:

Minimize unnecessary data movement, preserve programmability, and execute computation through efficient dataflow.

These characteristics make TokenProcessor a promising candidate for dedicated LLM decoding. But its ability to execute non-AI computational kernels—including search, graph processing, hashing, and signal processing—may ultimately be just as important. As AI systems evolve from generating tokens to taking actions, accelerating the model alone will not be enough. We must accelerate the computation surrounding the model as well.

The next generation of AI infrastructure will not simply be about generating tokens faster. It will be about executing complete AI workflows more efficiently.

And that is the future LENZO is building toward.

Learn more about LENZO's CGLA architecture and software development environment at lenzo.com and explore our technical tutorials.