Perplexity Lily: Apple Silicon Engine Reviewed

• Open-sourced 2 September 2026 under Apache 2.0, one day after Perplexity shipped Hybrid Compute, the Mac-app feature Lily powers, on 1 September 2026.
• Deliberately narrow by design. Lily supports exactly one architecture, Qwen3.6-35B-A3B in MLX affine 4-bit format with group size 64, and rejects every other checkpoint shape (dense Qwen, BF16, GGUF, AWQ, GPTQ, int8, fp8) at load time.
• 1.23x prefill, 1.35x decode versus MLX-LM's fastest direct-generation path, averaged across ten lengths from 256 to 128K tokens on a 40-core, 128GB Apple M5 Max.
• Numerically close to MLX-LM despite the speed gain: a teacher-forced check across 192 positions found Lily's perplexity only 0.04% higher, with the same top-ranked token chosen 96.35% of the time.
• No PyTorch, no MLX. A Rust runtime owns the checkpoint, session state and generation loop; hand-written Metal kernels do the compute; an OpenAI-compatible chat-completions API is the only interface.
• Powers Hybrid Compute, which splits a Perplexity Computer task between cloud reasoning and a local model, gated by Privacy Gate and its open-sourced PII-Tracer classifier.
• Genuinely reproducible. Both performance reports publish the exact measurement contract, hardware, software versions and SHA-256 hashes of every binary, dependency lockfile and checkpoint revision involved.
• Narrow in scope by design: no speculative decoding, no streaming, no tools, no multimodal input, no sampling beyond greedy decoding, and a hard requirement for an M5-generation or newer chip on macOS 26.

The two announcements landed a day apart and are best read as one story. On 1 September 2026, Perplexity introduced Hybrid Compute in the Perplexity Computer Mac app: a mode that, in the company's own words, "now splits a task between cloud models and a local model on your Mac". The framing addresses a structural problem agentic assistants have always had — the context that makes them genuinely useful, deal documents, privileged files, client records, is exactly the context users cannot responsibly send to a cloud endpoint. Hybrid Compute's answer is to keep the cloud model doing research, planning and reasoning, while routing anything touching private files or on-device actions to a compact model that never leaves the Mac.

The following day, Perplexity open-sourced the engine behind the local half of that split: Lily. The blog post announcing it is explicit about why the split needs a dedicated engine rather than an off-the-shelf one: "For this division of labor to feel seamless, local inference must keep pace with the rest of the task. That requires an engine that can process prompts quickly and sustain a high token-generation rate." A standalone demo of Lily, decoupled from the Mac app, is published in the pplx-garden repository, Perplexity's open source home for inference technology, which also holds an RDMA transfer library (fabric-lib) and a unigram tokenizer.

The default way to run an LLM on a Mac is MLX, Apple's open-source array framework, plus MLX-LM, its companion library for loading and generating text with a wide range of architectures. MLX-LM already ships a competent Qwen implementation, grouped expert dispatch, a fused recurrent kernel, GQA-aware attention, but its operations must stay reusable across every model architecture MLX-LM supports. Perplexity's argument is that this reusability is exactly what caps its ceiling: "An engine dedicated to Qwen can specialize at the model and runtime level, coordinating kernels, data movement, and scheduling around the model's fixed structure."

Lily implements that specialisation end to end in a single process: a Rust runtime loads the checkpoint and manages session state and the generation loop, an OpenAI-compatible chat-completions API accepts requests and streams tokens, and custom Metal kernels execute every Qwen-specific operation. Neither PyTorch nor MLX sits anywhere in the execution path. The target model, Qwen3.6-35B-A3B, is what makes this worthwhile: it is a sparse, hybrid architecture with 35 billion total parameters but only around 3 billion activated per token, using mixture-of-experts routing across 256 experts (eight selected per token, plus one always-on shared expert), 10 full-attention layers with grouped-query attention (16 query heads sharing 2 KV heads), and 30 Gated DeltaNet layers that compress earlier context into a fixed-size recurrent state instead of a growing cache.

Those three structures, uneven MoE expert groups, attention over a growing KV cache, and a sequential recurrence, create genuinely different computational shapes, and Perplexity's central claim is that a model-specific runtime can coordinate all three around Apple silicon's actual hardware paths (the M5's Neural Accelerators for dense matrix work, its vector ALUs for the bandwidth-bound remainder) more tightly than a general-purpose library ever safely could.

Prefill processes many prompt tokens at once, and Perplexity's optimisations here target three specific costs. First, dequantising weights inside the matrix multiply itself: the checkpoint stores weights as 4-bit codes with a shared bfloat16 scale and bias per group of 64, shrinking the model from roughly 70GB in bfloat16 to a 19.4GB checkpoint, but Metal 4's tensor operations need bfloat16 operands. Lily reconstructs each weight tile inside the grouped GEMM itself, in on-chip threadgroup memory, rather than expanding the full array into unified memory first; in Perplexity's own ablation, this raised end-to-end prefill throughput by 77.4% at a 512-token prompt.

Second, keeping MoE routing entirely on the GPU. Assigning token rows to experts requires a histogram, a prefix scan, a scatter and a block map; an ablation that instead pauses for the CPU to inspect these intermediates showed the cost directly: keeping the whole sequence in one GPU command buffer added 89% to prefill throughput at 512 tokens by removing CPU–GPU synchronisation inside every MoE layer. Third, matching tile size to actual expert load (a 32-row tile with four simdgroups added 13.2% at 2K tokens over a fixed 16-row baseline) and keeping the Gated DeltaNet recurrent scan resident in registers rather than round-tripping through memory (+5.6% at 2K tokens). Long prompts are processed in bounded chunks throughout, so temporary activations never compete with resident weights, recurrent state and the KV cache for unified memory as a prompt grows.

Batch-1 decode, generating one token at a time for a single user, has almost no weight reuse, so throughput is set almost entirely by memory bandwidth rather than compute. Perplexity found one recorded decode step launched 795 GPU kernels forming 555 sequential stages, with Metal's serial execution mode running every one of them in order regardless of whether they actually depended on each other. Recording real data dependencies in a concurrent Metal pass, so independent kernels can overlap, was one fix; another was removing the per-token CPU round trip entirely by having the GPU write its selected token straight into the next step's input slot, rather than sending it to the CPU and back.

On the memory side, coalescing attention-cache reads lifted key bandwidth from 33.8 to 47.9 GB/s and value bandwidth from 42.0 to 61.8 GB/s, improving end-to-end decode by 2.1% at a 3,840-token context. GQA packing, grouping four query heads into one threadgroup so each KV cache row loads once and is reused across all four instead of being fetched independently, improved decode throughput by 23.8% at a 32K-token context. And switching to a fixed-block attention layout once the context passes 32K tokens, rather than always using the general path, improved decode by 7.7% at 32K, 27.4% at 64K, and 40.2% at 128K tokens, the single largest individual decode win in either report and a clear signal that Lily's advantage widens specifically as conversations get longer.

Perplexity's headline claim, an average of 1.23x MLX-LM's prefill throughput and 1.35x its decode throughput, comes from the engineering blog post and is an arithmetic mean across ten equally weighted lengths from 256 to 128K tokens. Crucially, Perplexity did not stop at that summary figure: it published two dated, independently reproducible reports directly in the pplx-garden repository, each with the full per-length table, the exact measurement contract, and SHA-256 hashes of the Lily binary, the Cargo lockfile, the matrix-runner script and the MLX test harness used.

Selected rows from Perplexity's 2 September 2026 report ("Lily vs MLX 0.32.2"), Apple M5 Max, 40-core GPU, 128GB unified memory, batch 1, greedy decoding. Full ten-row table in the source report.

The honest picture in that table is more nuanced than the headline average suggests. Decode is a clean win at every single measured context, ranging from 1.236x to 1.319x across the two dated reports. Prefill is not uniformly a win: it is strongest at short-to-medium prompts, peaking around 4,096 tokens, but on the later 0.32.2-dated report it actually dips below parity at 32,768 tokens (0.991x) and further still at 65,536 (0.902x) and 131,072 tokens (0.801x), meaning MLX-LM prefills faster than Lily at the very longest prompt lengths in that specific run. The earlier 0.31.2-dated report, by contrast, showed Lily ahead on every prefill row, including the long ones. Perplexity does not paper over this: both reports are published side by side with their exact MLX version numbers, letting anyone see that the comparison shifted somewhat between one MLX point release and the next.

On accuracy rather than speed, Perplexity ran a teacher-forced comparison against MLX-LM across 192 positions, feeding both engines the same reference prefix at each step so that earlier differences could not compound into later ones. Lily's perplexity came out 0.04% higher than MLX-LM's, and the two engines picked the identical top-ranked token 96.35% of the time, evidence that the specialised kernels are not trading accuracy for speed in any meaningful way.

Perplexity's writeup is unusually candid about dead ends, which is itself useful signal about how mature the optimisation effort is. Speculative decoding, using a smaller draft model to propose tokens for Qwen3.6-35B-A3B to verify, made batch-1 decode 18% slower rather than faster: verification processed groups of two to five rows, an inefficient shape for Apple silicon, and those rows frequently routed to different experts, increasing the amount of expert-weight data that had to be read per step. Shrinking the drafter's output vocabulary improved the drafter's own throughput by 4.7–5.1%, but did not make the full speculative loop faster overall. Perplexity is careful to flag this as workload-specific rather than a universal verdict on speculative decoding, noting its own batched Qwen deployment on Nvidia Blackwell datacenter GPUs uses speculative decoding successfully under different batch and hardware conditions.

Other experiments that did not pay off included reducing the total number of GPU kernel launches further, overlapping entire inference phases rather than individual kernels, using larger prefill tiles than the tuned 32-row size, applying broader operator fusion beyond the four chains that did help, accelerating the MoE router specifically, and combining the output projection with token selection. Measurements of the underlying hardware ceiling explain why: the MoE GEMMs and GEMVs were already reaching 97.9% and 90.3% of the fastest sustained weight-read rates achievable for their access patterns, and removing arithmetic entirely from the sparse GEMV changed throughput by only 0.2%, confirming that weight reads, not computation, are the limiting resource in decode. Prefill's matrix multiplication separately reached 93% of the theoretical matrix throughput limit in isolation and 80–86% inside the full model.

Lily is the compute engine; Privacy Gate is the policy layer that decides what actually reaches it versus the cloud. In Hybrid Compute, a task typically begins in the cloud, but any step that touches sensitive material is diverted locally, with Privacy Gate scanning content for identifiers such as government ID numbers, financial account numbers and credentials before anything crosses the boundary outward, and choosing to keep it local, mask it, refuse the step, or ask the user, according to independent reporting on the launch.

Perplexity open-sourced the classifier behind that gate alongside Lily: PII-Tracer is a compact 0.6B-parameter bidirectional encoder adapted from a Qwen3 backbone, replacing the model's original causal attention mask with padding-aware bidirectional attention over a 4,096-token window. A linear tagging head emits 37 labels, one "outside any span" label plus BIOES-format position labels across nine distinct PII categories, and a second, auxiliary head separately predicts whether a conversation contains sensitive material at all. Perplexity trained it for three epochs on roughly 714,000 samples, with a constrained Viterbi decoder resolving the final label sequence at inference time.

Alongside the model, Perplexity published PII-TRACE, a benchmark of 13,148 synthetic conversations spanning 13 languages and 10 writing systems, containing 37,431 character-level identifier mentions in total. Its organising insight is that finding most instances of PII in a long conversation is a meaningfully easier and less useful task than finding every single copy of it, since a single unmasked mention defeats the entire purpose of the gate. Across 12 competing detectors, PII-Tracer recorded the highest character-level F1 score (0.629) and placed second on span-overlap and span-containment F1, behind GPT-5.6-sol specifically. Where it pulled clearly ahead was consistency: PII-Tracer found every mention of a recurring identifier 79.4% of the time and every mention of a cross-turn identifier 77.6% of the time, against 57.0% and 55.1% for GPT-5.6-sol. In the hardest bucket, identifiers repeated six to ten times in one conversation, PII-Tracer scored 0.691 against 0.464 for GPT-5.6-sol, 0.073 for GLiNER2-PII, and 0.045 for Claude Opus 4.8, according to MarkTechPost's coverage of the release.

Hybrid Compute itself launches with three local models available, according to Perplexity's own announcement: Gemma 4 E4B, Qwen3.6-35B-A3B (the model Lily is built for), and a Perplexity model specifically post-trained for Perplexity Computer tasks. The in-app setup flow separately points to a one-click download of a smaller PPLX Qwen 3.8 27B model for users on lower-memory Macs, per 9to5Mac's reporting.

Running the open-source Lily server directly, rather than through the Hybrid Compute product, means meeting a strict and narrow bar. Per the repository's own README, Lily requires an Apple GPU family 10 or later chip (M5 and newer), macOS 26 or later for Metal 4 tensor operations, and Rust 1.92 as pinned by the project's toolchain file. It validates the exact 35B-A3B architecture and quantisation layout at load time and will refuse anything else: dense Qwen checkpoints, smaller Qwen models, BF16 weights, GGUF, AWQ, GPTQ, int8 and fp8 formats are all explicitly unsupported. The release benchmark and test suite pin an immutable checkpoint revision, mlx-community/Qwen3.6-35B-A3B-4bit revision 38740b847e4cb78f352aba30aa41c76e08e6eb46, downloadable via the Hugging Face CLI, and at 19.4GB it makes 32GB or more of unified memory the realistic minimum for the standalone server.

The API surface is deliberately minimal. The server exposes POST /v1/chat/completions, GET /v1/models and GET /health, accepts only text-only system/user/assistant messages, max_tokens, and an optional prompt_cache_key, and always decodes greedily with the checkpoint's thinking mode disabled. Streaming responses, sampling parameters, tool calls, response-format options, multimodal content and speculative decoding are all rejected outright by the request validator. A fixed two-entry LRU cache of decode states lets a repeated prompt prefix skip reprocessing, reporting reused tokens back in usage.prompt_tokens_details.cached_tokens. This is a research-grade reference server, not a general local-inference product; Perplexity is explicit that the shipping Hybrid Compute feature in the Mac app has its own separate, more permissive requirements (macOS 15, 24GB minimum, 32GB recommended) precisely because it can fall back to lighter local models when Lily's bar isn't met.

• One model, one hardware family. Lily runs only Qwen3.6-35B-A3B on M5-generation-or-newer Apple silicon; it is not a general local-inference runtime and explicitly rejects every other checkpoint format.
• Prefill advantage narrows and can invert at very long prompts. The 2 September dated report shows Lily falling behind MLX-LM on prefill specifically at 32K, 65K and 131K-token prompts (0.991x, 0.902x, 0.801x), even though decode stays ahead throughout.
• Steep hardware floor. M5-or-newer, macOS 26, and 32GB or more of unified memory for the standalone server rule out the large installed base of older Apple silicon Macs.
• Minimal API by design. No streaming, no sampling, no tool calling, no multimodal input, no speculative decoding, greedy decoding only, single-request batch size 1.
• Speculative decoding made things worse, not better, on this specific batch-1, single-model workload, which limits one common path to further speedups without a different approach.
• No weights included. Users must separately obtain the specific pinned Qwen3.6-35B-A3B checkpoint revision; Lily is the engine, not the model distribution.

Against Apple's own MLX-LM, the comparison Perplexity chose to publish, Lily wins decisively on decode throughput at every measured context and on prefill at short-to-medium prompt lengths, but the two dated reports together show that lead is not permanent or universal: it can shrink, and on the later report even invert, at the longest prefill lengths. MLX-LM remains the far more general tool, supporting essentially any architecture the MLX ecosystem has implemented, streaming, sampling, and ongoing community maintenance, none of which Lily offers.

Against other local and open-weight approaches to Apple silicon covered on this site, such as Liquid AI's LFM2.5-DSpark draft models, which speed up decoding through speculative decoding rather than a rewritten kernel stack, Lily's approach is the more radical one: rather than accelerating a general engine with a smaller helper model, it rewrites the engine itself around one specific target model. Against Qwen3.8-27B and Qwen 3.8 Max, both open-weight releases from Qwen's own maker, Alibaba, Lily is not a competing model at all; it is complementary infrastructure that could in principle be adapted to run other Qwen checkpoints, though today it validates and accepts only the single 35B-A3B architecture and quantisation layout.

Against fully local, agent-oriented open releases such as Meta's Muse Glimmer 30B, the comparison is again more about philosophy than a head-to-head number: Muse Glimmer is a downloadable, Apache 2.0 open-weight model users can run on whatever local stack they choose, while Lily is a highly specialised engine tied to one specific upstream model that Perplexity does not itself control the weights for.

Worth trying now if you are building or researching local, privacy-preserving inference specifically on recent Apple silicon (M5 or newer) and specifically want to run Qwen3.6-35B-A3B: Lily is a real, reproducible, meaningfully faster alternative to MLX-LM for that exact combination, and the two dated performance reports with published hashes make its claims unusually easy to check rather than take on faith. Engineers interested in Metal kernel design for MoE routing, Gated DeltaNet, or GQA on Apple's Neural Accelerators will also find the source and ablation writeups genuinely instructive.

Not the right tool if you want a general local-inference server for arbitrary models, need streaming responses, tool calling, sampling controls or multimodal input, or are running anything older than an M5-generation Mac. Most everyday Mac users who simply want the benefit described here should instead use the shipping Hybrid Compute feature in the Perplexity Computer Mac app, which wraps Lily (or a fallback local model on lower-memory machines) behind a one-click setup and Privacy Gate, rather than building and running the standalone server themselves.

Lily is a narrow, unusually well-documented piece of infrastructure rather than a general product launch, and that narrowness is the entire point: by giving up on supporting every model and every Mac, Perplexity built something that beats Apple's own general-purpose framework on the one combination it targets, decode throughput up 1.23–1.35x on average and ahead at every tested length, prefill ahead at most lengths but not unconditionally so at the very longest prompts. The two dated, hash-verified performance reports are a genuinely higher bar for transparency than most vendor benchmark claims in this space clear.

The more interesting long-term story may be Privacy Gate and PII-Tracer rather than raw throughput: a 0.6B open-source classifier that beats larger frontier models specifically at not missing repeated mentions of the same identifier is a genuinely useful, independently reusable building block for anyone building a cloud/local split of their own, regardless of whether they ever run Lily itself.

Perplexity's official engineering blog post and the Lily source and performance reports on GitHub carry the full detail referenced throughout this article.