IronSight: Turning 2D Video Into 4D Replays

IronSight is not a shipped commercial product; it is a public engineering log, published in instalments as a "dev diary" series on X and Substack, of a single creator fusing several genuinely hard computer-vision problems into one working tool. The result: point a couple of pairs of smart glasses (or phones, or GoPros) at a shooting range, and IronSight will reconstruct the whole session as a navigable 4D scene, automatically score every shot as a hit or a miss, and let you replay it from an angle no camera was ever actually at - including, in the project's most attention-grabbing feature, a view that tracks a target's position even when a wall or fence physically blocks the camera's line of sight.

What makes it notable for this site is less the shooting-range use case and more the assembly: the project stitches together structure-from-motion geometry, 3D Gaussian splatting, and a frontier multimodal model (Gemini) acting as an automated judge, and it was substantially "vibe coded" - built through natural-language collaboration with an AI coding agent rather than hand-written line by line - with Claude Fable 5 specifically credited with accelerating the build once it became available.

• What it is: a 4D scene reconstruction tool built from ordinary multi-camera 2D video, with automated hit/miss scoring.
• Who built it: Bilawal Sidhu, a creator and former Google spatial-computing product manager, published as a public dev-diary series.
• The stack: COLMAP/pi3 structure-from-motion, 3D Gaussian splatting, audio-waveform sync, Gemini as multimodal judge, built with heavy use of Claude Fable 5.
• What's next: a free, open-source "God's Eye View v1" release targeted for late July 2026.

At its core, IronSight is a desktop web application that ingests footage from two or more cameras filming the same shooting-range session and outputs an interactive 4D spatial replay - "4D" meaning a full 3D scene that also plays back correctly in time, so you can pause, rewind and fly around a specific moment rather than just watching a flat video. In the creator's own framing: "video becomes a place, and the camera becomes just one way into it."

Once reconstructed, a session can be explored the way you would explore a 3D game level: orbit around a shooter mid-draw, drop into a target's-eye view, or switch to an overhead "God's Eye" perspective that shows every shot fired, every target hit, and the shooter's position at every instant - all derived purely from footage that was never filmed from that angle.

IronSight is a solo/small-team side project from Bilawal Sidhu, a creator whose YouTube and X channels (tracked by this site) cover spatial computing, generative AI and applied computer vision to an audience of more than two million subscribers. Before his current work as a full-time creator, TED technology curator and a16z venture scout, Sidhu spent six years at Google as a senior product manager leading spatial-computing initiatives - including Google Maps' Immersive View, the ARCore Geospatial API and YouTube VR Capture - giving him direct professional background in exactly the 3D-reconstruction and camera-geometry problems IronSight tackles.

That background matters for how to read the project. This is not a hobbyist stumbling into computer vision; it is a specialist who spent years shipping 3D mapping features to more than a billion users, now demonstrating what the same category of problem looks like when a single person can attack it with current-generation AI coding tools instead of a Google-sized engineering team.

The defining constraint of the project is what Sidhu calls a "pixels only" rule: no LiDAR scanners, no depth cameras, no inertial measurement units. Every reconstruction is recovered purely from ordinary 2D video shot on consumer hardware - Meta Ray-Ban smart glasses, GoPros and iPhones, mixed and matched across a session. Early sessions fused footage from two separate pairs of Ray-Bans; later ones added a third-person GoPro observer angle and a rifle-mounted first-person view.

This is a meaningfully harder problem than reconstruction from dedicated depth sensors, because the system has to recover camera position, orientation and scene geometry entirely from how pixels move and correlate across frames - the same fundamental challenge that makes structure-from-motion and SLAM (simultaneous localisation and mapping) active, difficult areas of computer-vision research. Choosing the hard path deliberately is also the point: consumer smart glasses are the camera most people already own and wear, so a reconstruction pipeline that works from Ray-Ban footage alone is far more broadly applicable than one that requires specialised capture rigs.

The technical pipeline runs in stages, each solving a distinct problem:

• Synchronisation: because the cameras are not hardware-linked, IronSight aligns them after the fact using audio - detecting gunshot reports in each camera's waveform to establish a single shared timeline across every angle.
• Camera geometry: COLMAP, a widely-used open-source structure-from-motion toolkit, recovers each camera's trajectory and unifies every feed into one coordinate frame, using masking to ignore moving foreground subjects while solving for the static scene.
• 3D reconstruction: the aligned frames train a 3D Gaussian splat - a scene representation built from millions of small, semi-transparent, oriented "blobs" positioned in space, which can be rendered from any virtual viewpoint far faster than traditional mesh-based photogrammetry. Sidhu also tested alternative feed-forward mappers (pi3) as a comparison point against COLMAP's more established but slower pipeline.
• Shot detection and scoring: classical audio analysis flags each gunshot event; Gemini then reviews the corresponding cropped video frame to call it a hit or a miss, with genuinely ambiguous cases (a spinning target mid-impact, for instance) escalated to a human review queue rather than silently guessed.

A fifth stage handles the housekeeping that makes the other four usable: a purpose-built annotation tool keeps shot labels synchronised across every camera angle, so a hit tagged from the shooter's first-person Ray-Ban feed carries over correctly to the third-person GoPro observer view and the reconstructed God's-Eye perspective, rather than requiring each angle to be labelled separately. It is an unglamorous piece of tooling, but it is exactly the kind of infrastructure work that separates a one-off demo from something that can be rerun session after session without manual clean-up.

One documented failure case is instructive: fast-spinning "star" targets can physically separate from their tracked position in the static Gaussian reconstruction after impact, because the splat represents a moment-in-time geometry rather than tracking the target's continued motion - a reminder that even a working 4D pipeline is reconstructing a scene, not truly simulating physics.

A striking part of the project is how much of it is dedicated to proving the reconstruction is trustworthy rather than merely showing it off. IronSight's interface includes a "match photo" verification mode - visible in the screenshot above - that wipes between the live Gaussian-splat render and an actual camera frame from the same instant, so any drift or hallucinated geometry is immediately visible rather than hidden behind a polished render.

That rigour extends to formal engine comparisons: the image above documents a head-to-head test of three different geometry-tracking approaches - COLMAP's classical global mapper, the feed-forward pi3 model, and an internal tool referred to as lingbot-map - scored on frame registration rate, reprojection error and trajectory drift against the same source footage. COLMAP produced the crispest reconstruction with full registration and sub-pixel error but took longest to run; the faster alternatives traded some accuracy for throughput. Publishing a benchmark like this - not just the flashy replay, but the unglamorous accuracy numbers behind it - is unusually disciplined for a solo side project, and closer to how an applied research team would validate a computer-vision pipeline before shipping it.

The most conceptually interesting piece of IronSight is arguably the smallest in the pipeline: using a frontier multimodal model as an automated scoring judge rather than a chatbot. Once a shot's timestamp is located from the audio track, Gemini receives a cropped video frame around the relevant target and returns a hit/miss classification - a task that requires genuine visual reasoning (spotting a fresh bullet mark, a knocked-over target, or a spinning "star" target's new resting position) rather than simple object detection.

This pattern - a capable multimodal model doing the specific, bounded judgement call inside a larger deterministic pipeline, rather than trying to run the whole system end-to-end - is a useful template well beyond shooting ranges. It shows up in the sports-replay and volumetric-capture tools referenced in Sidhu's wider research (his August 2026 posts on 4D Gaussian splatting for live sport, built with Arcturus, use a related capture-and-replay approach), and increasingly in agentic coding and QA tooling generally: use the model for the judgement a human would otherwise have to make, and keep the surrounding system classical, fast and auditable.

The part of this story most relevant to how AI tools are actually used day to day is how quickly IronSight came together once Claude Fable 5 became available to Sidhu. His own account is blunt about the timeline: an early shot-counter prototype existed as a rough proof of concept in mid-2025, built with an earlier Gemini model handling media understanding plus a hand-written After Effects script for the heads-up-display overlay. The full multi-camera 4D IronSight system, by contrast, is explicitly described as "a weekend build with Fable" - work that, in Sidhu's estimation, would have been a publishable SIGGRAPH-calibre research paper only a few years earlier.

"Vibe coding" - working with an AI coding agent through natural-language iteration rather than writing every line by hand - is often discussed abstractly. IronSight is a concrete, technically substantial data point: a former Google spatial-computing PM, with real domain expertise, using it to compress a multi-week computer-vision engineering effort (camera calibration, structure-from-motion integration, Gaussian splat training, multimodal scoring, and a full interactive replay UI) into what he describes as a few days of focused work.

IronSight's development has been documented in near-real time rather than announced as a finished product. Dev Diary 1 (4 July 2026) introduced the basic 4D fusion of two Ray-Ban feeds. Dev Diary 2 (7 July) added the through-wall AR tracking capability and cross-camera hit scoring. By 15 July, Sidhu was demonstrating a full "vibe coded terminator vision" replay system with no LiDAR or IMU at all, built entirely from Ray-Bans, GoPros and iPhone footage, alongside the formal engine benchmark and verification tooling covered above.

Reading the diary chronologically is a better way to understand modern AI-assisted engineering than any single demo: it shows the false starts (an early full-loss trajectory tracker that produced a usable track but smeared the point cloud with drift), the deliberate benchmarking, and the compounding effect of a capable coding agent removing the friction that used to make each incremental feature its own multi-day task.

IronSight sits alongside a related, earlier Sidhu project called God's Eye View (formerly "WorldView"), which took a similar reconstruction approach and applied it more broadly - an orbital, satellite-style visualisation layer that has separately gone viral on social media. Sidhu has said the underlying code is being prepared for a public, open-source release, with a first version targeted for around the end of July 2026.

A roadmap document Sidhu published alongside the project sketches a longer-term "Phase 3" vision explicitly framed as a wishlist rather than a commitment: full 3D point-cloud reconstruction at world scale, proper SLAM-based camera trajectory recovery, multi-camera synchronisation against a shared spatial reference, a native desktop app, and even post-run AI coaching that reviews a session and suggests what to improve. Whether all of that ships is genuinely uncertain - the document itself is labelled "someday" - but the direction is clear: from a single weekend demo toward a general-purpose spatial-reconstruction platform.

It is easy to read IronSight as a novelty - a fun demo for a niche hobby. The more interesting reading is what it demonstrates about the current state of three converging technologies: consumer smart glasses as an ambient capture device, 3D Gaussian splatting as a fast, increasingly accessible reconstruction technique, and AI coding agents capable enough to let a domain expert skip most of the traditional engineering overhead of gluing those pieces together.

Sidhu's own framing captures this well: he describes the project as sitting at "the convergence of augmented reality and artificial intelligence," and explicitly shares his workflows so others can replicate and remix them with their own agents. That open, journal-style approach to building - publish the failures and the benchmarks alongside the wins, and let the audience follow the actual engineering process - is arguably as significant a signal about where AI-assisted development culture is heading as the reconstruction technology itself.

There is also a practical, near-term application well beyond hobbyist shooting sessions. The same pipeline - multi-camera fusion, audio synchronisation, Gaussian splat reconstruction, multimodal event scoring - maps fairly directly onto sports coaching and officiating (reconstructing a contested play from broadcast and sideline footage), industrial safety review (replaying a workplace incident from whatever cameras happened to be present), and search-and-rescue or forensic reconstruction from bystander phone footage. None of those are things IronSight itself does today, but the fact that one person assembled the core capability from consumer hardware and off-the-shelf models in about a week is the more important signal than the specific shooting-range application it currently ships with.

• Not a shipped product. IronSight is a personal research project with no announced commercial release, pricing or support - treat it as a demonstration of what is now possible, not a tool you can currently buy.
• Static-scene assumption. Gaussian splats reconstruct geometry at a point in time; fast-moving elements like spinning targets can visibly detach from the reconstruction after impact, a limitation the project's own documentation is candid about.
• Engine trade-offs are real. The published bake-off shows classical COLMAP still wins on accuracy over faster feed-forward alternatives - there is no free lunch between reconstruction quality and processing speed yet.
• Open-source timeline is not guaranteed. The "God's Eye View v1" release and the broader Phase 3 roadmap are stated targets and a wishlist respectively, not commitments with contractual dates.

Professional volumetric-video and sports-replay systems - the kind of multi-camera rigs used in broadcast "freeze-frame" replays or dedicated venue-scale capture stages - have existed for years, but typically require large, fixed camera arrays, significant compute budgets, and specialist operators. IronSight's distinguishing feature is doing a version of the same job with hardware most people already own, processed by one person on consumer compute, largely built through conversational AI coding rather than a studio engineering team.

That trade-off is honest, not magic: professional rigs still win comprehensively on reconstruction fidelity, camera count and reliability. What IronSight demonstrates is how far the accessible end of that spectrum has moved - a gap that, on this evidence, is closing faster than most people tracking either AI coding tools or spatial computing in isolation would expect.

Sidhu's own back catalogue makes the comparison concrete. His reporting on Arcturus's venue-scale 4D Gaussian splatting - built for broadcasting entire sporting events from a "god's eye" view, tested first-hand in a headset - and his separate coverage of a 2017-era Intel volumetric capture stage both used the same underlying representation IronSight relies on, but required a purpose-built studio full of fixed cameras to produce. Watching essentially the same output category emerge from two pairs of consumer smart glasses and a weekend of AI-assisted development is the clearest single illustration of how much of that infrastructure gap software has now closed.

IronSight is a small project doing a lot of useful signalling at once: that 3D Gaussian splatting has matured enough for a solo creator to deploy it convincingly from consumer smart-glasses footage, that multimodal models like Gemini are ready to act as bounded, auditable judges inside larger pipelines rather than just chat interfaces, and that AI coding agents - Claude Fable 5 specifically credited here - can compress what used to be weeks of specialist computer-vision engineering into a weekend, in the hands of someone who already understands the problem deeply.

None of that means the underlying problems are solved - the engine bake-offs and the spinning-target failure case are refreshingly honest about what still breaks. But as a real-time public record of what one domain expert can now build with current AI tooling, it is one of the more concrete demonstrations available of where AI-assisted engineering actually stands in mid-2026, well outside the usual coding-benchmark framing.