The Rendering of Reality: Between Geometry and Appearance

Photogrammetry and Gaussian splatting are both techniques designed to reconstruct and visualize three-dimensional scenes from two-dimensional images. At first glance, they appear to share a common foundation: the idea that a 3D structure can be inferred from many 2D photographs taken from different perspectives. This underlying principle — known as Structure from Motion (SfM) — has been central to computer vision and computational photography for decades. But while both approaches share this conceptual root, they diverge radically in how they interpret and represent the resulting 3D data. Photogrammetry, rooted in geometry, aims to reconstruct explicit surfaces — typically as meshes with textures — suitable for CAD, VR, and inspection workflows. In contrast, Gaussian splatting takes a fundamentally different route: instead of aiming for a mesh, it produces a volumetric rendering from dense, point-based representations (3D Gaussians) optimized for photo-realistic real-time viewing. It is more closely aligned with neural rendering than traditional reconstruction.

In essence, photogrammetry tries to recover the shape; Gaussian splatting captures the appearance. Both aim to "depict" a 3D object from multiple images, but their methodologies and outputs serve different strengths and applications.

So where does this specific idea to re-create parts of our reality from images come from? Let's take a brief look at their shared past.

After photography was invented and began to evolve in the early 19th century — notably through the pioneering work of Nicéphore Niépce, Louis Daguerre, and William Henry Fox Talbot — it opened a new window on the world. By the 1840s, the camera had become more than a curiosity; it was increasingly embraced as a scientific instrument. Early astronomers, archaeologists, and military surveyors began using photography not only to document but to measure, catalog, and study the world^[1][2][3].

This transformation laid the groundwork for photogrammetry — the idea that spatial information could be extracted from images. By the 1850s, engineers were beginning to experiment with deriving measurements from photographs taken at different angles, giving birth to image-based 3D reconstruction.

The foundations of photogrammetry were laid in the mid-19th century by Aimé Laussedat, a French military officer and engineer. His 1851 paper Mémoire sur l'emploi de la photographie dans la levée des plans^[4] introduced the use of photographs for topographic mapping, earning him recognition as a pioneer in the field.

Laussedat's breakthrough came from combining the emerging technology of photography with traditional surveying methods. He developed a method called "iconométrie" or "metrophotography," using precisely oriented photographs taken from different viewpoints to create accurate topographic maps^[5]. His process involved using a specially modified camera-theodolite (illustrated below), an instrument he designed to ensure geometric accuracy in the photographs. Initially working with ground-based photos of Paris architecture, he later expanded to balloon photography for wider terrain coverage^[6].

Between 1851 and 1859, Laussedat refined his techniques through extensive experimentation at the Invalides in Paris. His method involved taking photographs from at least two different positions, measuring angles and distances to control points, and using geometric principles to reconstruct the three-dimensional positions of features visible in multiple images^[7]. The success of these experiments led to the French Army's adoption of his techniques for military mapping, marking the first institutional recognition of photogrammetry as a practical surveying tool^[8].

The turn of the century brought significant technological advances. In 1901, Carl Pulfrich at Carl Zeiss developed the stereocomparator^[9], enabling precise measurements from stereoscopic photograph pairs. This innovation led to the development of stereophotogrammetry, which would prove crucial for military applications.

Eduard von Orel's invention of the stereoautograph in 1909 at the Military Geographic Institute in Vienna^[10] marked another milestone, allowing direct plotting of contour lines from stereo photographs. This technique remained fundamental until the digital era.

World War I transformed photogrammetry from an experimental technique into an essential military tool. The British Royal Flying Corps established dedicated photographic sections that systematically documented the Western Front through aerial reconnaissance^[11], developed systematic methods for aerial survey and map production. Their process involved:

• Grid-pattern aerial photography flights
• Glass plate negative photography using specialized cameras
• Mobile darkroom development
• Stereoscopic image analysis
• Mechanical plotting for topographic map creation

By 1918, Allied forces were processing tens of thousands of aerial photographs weekly, producing detailed maps that proved crucial for military planning^[12].

The interwar period saw further technical advancement with the introduction of new analog plotting instruments. The Wild A5, introduced in 1937^[13], represented a significant advancement in stereoplotting technology. These mechanical-optical devices enabled operators to trace contour lines and features while viewing stereo pairs, producing highly accurate topographic maps.

Heinrich Wild, the Swiss engineer who revolutionized precision instrument design, significantly advanced photogrammetric capabilities through his innovations at Wild Heerbrugg. His stereoplotting instruments dramatically improved the efficiency of map production compared to traditional field surveying methods^[14].

The transition to digital photogrammetry marked a pivotal moment. Software like Agisoft Metashape (formerly PhotoScan) and RealityCapture democratized 3D reconstruction, allowing anyone with a decent camera to create detailed 3D models. This democratization laid the groundwork for applications we now take for granted:

• Building Information Modeling (BIM)
• Archaeological documentation
• Virtual reality environments
• Film and game production
• Industrial quality control

The concept of using Gaussian functions for point-based graphics emerged from early computer graphics research in the 1980s, building on fundamental work in volume rendering and surface representation.

In 1983, researchers at the University of North Carolina introduced the concept of splatting for volume rendering. Dr. Lee Westover's seminal paper Interactive Volume Rendering^[15] presented the fundamental ideas of representing 3D points as Gaussian kernels.

Dr. James F. Blinn's work in the late 1980s on "blobby models" and soft objects laid additional groundwork for what would become Gaussian splatting^[16]. However, these early attempts faced significant limitations:

• Computational Power: The processing requirements far exceeded available hardware
• Memory Constraints: Storing and manipulating large point clouds was impractical
• Limited Graphics Capabilities: Real-time rendering of complex splats was impossible
• Lack of Efficient Algorithms: Optimization techniques were still primitive

The breakthrough by researchers at ETH Zürich^[18] coincided with three key developments:

1. Shared Foundations, Divergent Paths
While Gaussian splatting and photogrammetry both utilize Structure-from-Motion (SfM) for initial camera pose estimation from image collections, they diverge fundamentally in their processing:

2. The Depth Map Dilemma
Modern implementations reveal an important nuance: while Gaussian splatting can incorporate depth maps derived from photogrammetry (through tools like Polycam's conversion pipelines)^[19], this represents an optional hybrid approach rather than a requirement. The ETH Zürich team demonstrated that splatting achieves best results when trained directly on raw images/video, particularly for challenging materials like:

• Translucent surfaces
• Specular reflections
• Volumetric effects

3. Platform Synergies
The perceived "convergence" stems from commercial tools creating interoperability bridges:

• Polycam allows export of splatting-ready point clouds from photogrammetry scans^[20]
• Luma AI enables mesh extraction from Gaussian splat reconstructions
• NERFstudio supports photogrammetry-derived depth as training constraints

Marc Levoy, reflecting on these developments in 2024, noted^[21]: "What we're seeing isn't convergence in the algorithmic sense, but rather the emergence of multi-modal workflows where photogrammetry provides metrological validation while splatting handles perceptual realism."

The complementary nature of these technologies becomes particularly valuable in virtual production:

Virtual Production Workflow

1. Photogrammetry: Captures measurable set pieces/digital doubles
2. Gaussian Splatting: Renders dynamic environments in real-time
3. Hybrid Validation: Mesh models verify splatting reconstructions metrologically

This separation of concerns addresses a key industry challenge: While traditional photogrammetry meshes struggle with real-time rendering of complex materials, Gaussian splatting's neural radiance fields (NeRFs) can represent these effects naturally but lack inherent measurability. Recent Mo-Sys Engineering tests showed Gaussian splatting achieving 92% faster render times for reflective surfaces compared to photogrammetry pipelines^[22].

The journey from Laussedat's camera-theodolite to today’s neural rendering tools spans nearly two centuries of innovation. What began as a military mapping aid has become a cornerstone of visual computing — shaping everything from cultural heritage to cinematic production.

Photogrammetry and Gaussian splatting, though rooted in the same principle, now offer distinct strengths: one captures shape with geometric rigor, the other appearance with immersive realism. Together, they form a dual lens through which we model, measure, and imagine space.

As these technologies become increasingly accessible, artificial intelligence is playing a growing role in shaping their future. Gaussian splatting already relies on gradient-based optimization and neural radiance fields^[23], while modern photogrammetry tools use machine learning to enhance feature matching, depth estimation, and object segmentation^[24]. From real-time rendering in neural pipelines to adaptive automation in photogrammetric modeling, AI is redefining how 3D reconstruction is built — and who can build it.