Mesh refinement

Clean a noisy SLAM mesh, densify it via Loop subdivision, and texture-map vertices from RGB frames with view-angle, depth, and occlusion awareness.

The triangle mesh emitted by a SLAM system is usually noisy: floaters, small holes, low vertex count, and either no colours at all or coarse per-vertex colours from a one-off integration step. MeshRefiner bundles a four-stage pipeline that fixes those issues using the rest of the recording — the per-frame RGB images and the camera trajectory.

from stera.data import MCAPReader
from stera.processing import MeshRefiner

session = MCAPReader("recording.mcap")
refined = MeshRefiner(session).refine()
# refined.vertices / .faces / .vertex_colors / .vertex_normals

Install the optional dependencies first: pip install "stera-sdk[mesh]". This pulls in pymeshlab, pyransac3d, scipy, and matplotlib. The colorize-only path works on a base install.

Pipeline

#	Stage	What it does
1	`clean`	pymeshlab cascade — dedup verts/faces, repair non-manifold, drop floaters (small components by face count + diameter), close small holes, one Laplacian smoothing step.
2	`densify`	Loop subdivision — each iteration roughly quadruples the face count, giving the colorize stage finer texture-sampling granularity.
3	`colorize`	For each enabled frame: project verts into RGB + depth, bilinear-sample the image, weight by view-angle and inverse-distance, reject occluded samples via the depth buffer.
4	`brighten`	Linear brighten + lift darks — meshes textured from FPV footage often need it.

Two more optional, off-by-default scene-cleaning passes ride on top of the same RANSAC plumbing:

#	Stage	What it does
5	`strip_table_clutter`	Find the kitchen-counter plane (largest horizontal plane that isn't floor or ceiling), drop every vertex sitting on it. Walls attached to the counter are protected by kd-tree proximity.
6	`fill_table`	Drop a flat triangulated patch on the counter plane. Region defined by a 2D occupancy mask of inlier density (handles L-shaped counters; doesn't over-extend).

All six stages are exposed as separate methods on MeshRefiner. The one-shot refine() chains whichever are enabled.

Quick recipes

refined = MeshRefiner(session).refine()

Clean + ×16 subdivision + colorize at color_speed=0.5. ~37 s on a 11k-frame MCAP.

refined = MeshRefiner(session, color_speed=0.0).refine()

Every 30th frame, hard cap of 400 frames, no occlusion. ~30 s.

refined = MeshRefiner(session, color_speed=1.0).refine()

Every frame, depth-buffer occlusion test on. ~100 s, sharpest texture.

refined = MeshRefiner(
    session,
    subdivision_iters=0,
    colorize=False,
).refine()

Just the pymeshlab cleanup cascade — useful for downstream tools that do their own texturing.

refined = MeshRefiner(
    session,
    strip_table_clutter=True,   # drop objects on the kitchen counter
    fill_table=True,            # add a clean flat patch where they were
).refine()

Adds two extra scene-cleaning passes. Geometry heuristics; tune the knobs below per scene.

Speed dial

color_speed is a float in [0, 1]. Lower = faster, higher = sharper texture detail.

`color_speed`	every_n	max_frames	occlusion
`0.0`	30	400	off
`0.25`	13	none	off
`0.5` (default)	5	none	on
`0.75`	2	none	on
`1.0`	1	none	on

every_n interpolates geometrically (round(30 ** (1 - q))) so the dial is roughly log-linear in cost. Per-knob overrides win over the dial:

MeshRefiner(session, color_speed=0.5, color_every_n=2)   # speed=0.5, but use every 2nd frame anyway
MeshRefiner(session, color_speed=1.0, color_use_occlusion=False)   # max quality without occlusion

How the colorizer weighs frames

For each vertex, every enabled frame contributes a colour sample with weight

w  =  cos²(view_angle) / distance²

where view_angle is the angle between the surface normal and the camera ray, and distance is the depth at projection. Head-on, close-up views dominate; grazing-angle and far views barely register. The weighted average across frames lands in vertex_colors.

With color_use_occlusion=True (the default at speed ≥ 0.5), the depth map at that pixel is checked first — if a closer surface is in front, the sample is rejected. This stops walls bleeding colour through one another.

Verts that no frame ever saw are filled with mid-grey (128, 128, 128).

Strip clutter from the counter

The two scene-cleaning passes are aimed at a recurring problem in indoor FPV scans: the kitchen / desk plane is reconstructed with everything that was sitting on it — cups, plates, laptops — turned into a lumpy approximation rather than the flat surface that's actually there.

refined = MeshRefiner(
    session,
    strip_table_clutter=True,
    fill_table=True,
).refine()

What happens:

Multi-RANSAC fits all major planes. Horizontal planes (normal close to world_up) are sorted by height.
The largest horizontal plane that isn't the floor (lowest) or ceiling (highest) is picked as the counter.
Every vertex inside the counter's convex-hull footprint AND clutter_height_min-clutter_height_max metres above the plane is classified as clutter and dropped along with any face touching it.
Walls attached to the counter are protected: any vertex within wall_protect_radius of an RANSAC vertical-plane inlier is excluded from the clutter set, so the wall surface — including the wall-to-counter junction — stays intact.
fill_table drops a triangulated flat patch on the plane. The patch region is defined by a 2D occupancy mask over the inlier density (so it follows L-shaped counters and doesn't extend past where the surface actually was).

These are geometry heuristics. The auto-detect picks the right plane in most kitchen / counter scenes, but for unusual layouts you may need to tune clutter_height_max, wall_protect_radius, or fill_occ_min_count. See the API reference for the full knob list.

Wire it into the visualizer

The Visualizer constructor takes a mesh_refine= flag that runs the same pipeline before the mesh hits the .rrd:

from stera.viz import Visualizer

viz = Visualizer(
    session,
    map_3d="both",                              # log mesh + PC (PC hidden by default)
    mesh_refine={"color_speed": 0.5},           # any MeshRefiner kwarg
)

map_3d="both" logs the refined mesh, the unrefined raw mesh (world/mesh_raw, hidden by default), and the point cloud (also hidden by default). All three sit in the entity tree with eye-icon toggles — flip any one on to compare.

Use the granular methods

The stages are usable independently:

refiner = MeshRefiner(session, color_speed=0.5)

verts, faces, _ = session.mesh()
verts, faces = refiner.clean(verts, faces)              # pymeshlab cleanup only
verts, faces = refiner.densify(verts, faces)            # ×16 face count
from stera.processing import compute_vertex_normals
normals = compute_vertex_normals(verts, faces)
colors = refiner.colorize(verts, normals)
colors = refiner.brighten_colors(colors)

refine() is just sugar for the above with the constructor flags.

Saving the result

import trimesh
refined = MeshRefiner(session).refine()
trimesh.Trimesh(
    vertices=refined.vertices,
    faces=refined.faces,
    vertex_colors=refined.vertex_colors,
).export("scene.ply")