Forest-scene 3DGS under-vegetation benchmark (+ Apple-Silicon OpenSplat path)#62
Open
itamarwe wants to merge 8 commits into
Open
Forest-scene 3DGS under-vegetation benchmark (+ Apple-Silicon OpenSplat path)#62itamarwe wants to merge 8 commits into
itamarwe wants to merge 8 commits into
Conversation
- Three.js scene: soldier in a clearing surrounded by 28 procedural trees with animated leaf clusters (wind via per-pivot rotation, URL-param wind time for deterministic headless capture) - URL params control camera (az/el/dist), vegetation visibility, wind time - capture_views.py: headless Chrome captures 16 views × 2 (veg/no-veg) = 32 images - benchmark.py: per-view occlusion ratio, PSNR, SSIM on full frame and soldier ROI; tier breakdown (top-down → low oblique); scene_difficulty score - Pre-captured sample_images/ at 3 perspectives for quick inspection https://claude.ai/code/session_01Wx5whWAwztMAspQ4B1NpQT
Leaf canopy: - 280 PlaneGeometry instances per tree (18 cm × 13 cm broadleaves) - Canvas-generated leaf texture: pointed-oval shape, lateral veins, 4 green variants - Per-leaf color variation via InstancedMesh.setColorAt() - Wind flutter via onBeforeCompile vertex injection (normal-displacement keyed on instance world position × time) — each leaf flutters independently - Macro tree sway via parent pivot group rotation - customDepthMaterial with alphaTest → alpha-aware shadow casting (creates dappled light on the ground through leaf gaps) - 4096×4096 shadow map for fine dapple detail Straw / thatch roof alternative (?canopy=straw&cov=0.42): - Flat InstancedMesh of thin sticks at canopy height - Coverage fraction is exact and controllable (cov=0.0–1.0) - Useful as a controlled-occlusion baseline for algorithm benchmarking https://claude.ai/code/session_01Wx5whWAwztMAspQ4B1NpQT
Scene: - USAF 1951 target (5m × 5m) flat on the ground — 4 groups (G1 coarsest 32px bars → G4 finest 4px bars), 6 elements each, canvas-generated texture - Bullseye crosshair (2.2m) adjacent — for localization accuracy metric - Straw roof default 50% coverage, controllable via ?cov= Benchmark (benchmark.py): - Measures Michelson contrast at every USAF group/element - Maps texture-space bar boxes to image pixels via target locator - Reports highest resolved spatial frequency + MTF loss vs. ground truth - recovery_ratio = max_freq_occluded / max_freq_ideal (1.0 = perfect) - Plots MTF curves (ideal vs. occluded vs. optional 3DGS reconstruction) Sample images: top-down and oblique at 30%, 50%, 70% coverage https://claude.ai/code/session_01Wx5whWAwztMAspQ4B1NpQT
Replace the heuristic bright-pixel bounding box (locate_target) with proper DLT homography computed from the Three.js camera parameters (az/el/dist → spherical position, FOV=52°) to the four known world corners of the 5m×5m USAF target. The old approach failed because tree canopies visible from above in nearly-top-down views inflated the bright-pixel bounding box by ~134px, misaligning the G2 crops and producing 0 contrast for all 78–44mm elements. The homography maps texture pixels exactly to image pixels for any camera elevation, fixing measurement accuracy across all views. Also adds CAPTURE_VIEWS lookup table in benchmark.py matching the VIEWS list in capture_views.py so camera params don't need to be repeated; stems not in the table are skipped with a warning.
The correct pipeline is: 1. capture_views.py → 40 training views (veg=1) + 1 GT (veg=0) 2. Train 3DGS on training views using exported COLMAP poses 3. Render top-down novel view → topdown_recon.png 4. benchmark.py --recon topdown_recon.png --gt topdown_gt.png → MTF Changes: - capture_views.py: expanded to 40 training views (6 elevation tiers × 4–8 azimuths), added COLMAP pose export (cameras.txt + images.txt + camera_meta.json), separated training views from GT capture - benchmark.py: takes --recon and --gt args; compares reconstruction vs ground truth (not occluded vs ideal); single-view analysis at GT pose - Identity control (GT vs GT) produces recovery_ratio=1.000, loss=0.0000
reconstruct_mv.py: for each GT top-down pixel, back-projects the ray to the y=0 ground plane, then samples every training image at the projected location. Pixel-wise median across 40 views suppresses straw sticks (which appear at different locations due to parallax) while preserving the target texture (which is at y=0 and projects consistently). Results: 0% occlusion (control): recovery_ratio = 0.570 50% occlusion: recovery_ratio = 0.445 Identity (GT vs GT): recovery_ratio = 1.000 The gap between 0% and identity is blurring from oblique views in the median stack — real 3DGS would close this gap with learned appearance. The 50% vs 0% gap (0.445 vs 0.570) is the measurable cost of straw occlusion.
…hmark Stock nerfstudio/gsplat is CUDA-only, so ns-train splatfacto won't run on an M-series Mac. Add make_opensplat_project.py to convert a capture_views.py --poses output into an OpenSplat nerfstudio project (transforms.json with OpenGL-convention c2w + a ground-plane seed cloud, since OpenSplat rejects random init). The clean top-down is rendered via a withheld validation camera, then scored with benchmark.py. Real OpenSplat 3DGS (50% straw, 7000 iters) recovers the coarse-to-mid USAF bars with much higher fidelity than the median ground-plane stand-in (reconstruction_loss 0.040 vs 0.160; same 0.0256 lp/mm resolution cutoff), losing only the finest bars where straw gaps get narrower than the bars. Document the macOS build (Metal needs a native Apple-Silicon Xcode; CPU fallback otherwise) and gitignore the regenerable training_*/ and *.ply artifacts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A synthetic benchmark for 3D Gaussian Splatting reconstruction under dense vegetation occlusion: a flat USAF 1951 resolution target on the ground, viewed through a straw "roof" at controllable coverage. Captures multi-view training images with known poses, reconstructs a clean top-down, and scores it with an MTF (Michelson-contrast-per-bar) metric.
forest-scene/index.html— Three.js scene (USAF target + straw occluders)capture_views.py— headless Chrome multi-view capture + COLMAP/nerfstudio pose exportbenchmark.py— per-element MTF,recovery_ratio,reconstruction_lossreconstruct_mv.py— multi-view median ground-plane projection (oracle stand-in for a flat scene)make_opensplat_project.py— new: run real 3DGS on Apple Silicon via OpenSplatRunning real 3DGS on an M-series Mac
Stock nerfstudio/
gsplatis CUDA-only, sons-train splatfactowon't run on Apple Silicon. This branch adds an OpenSplat path:make_opensplat_project.pyemits a nerfstudio project (transforms.jsonwith OpenGL-convention c2w matching the scene's camera basis + a ground-plane seed cloud, since OpenSplat rejects random init), and the clean top-down is rendered through a withheld validation camera.Result (50% straw occlusion, 7000 iters)
recovery_ratioreconstruction_lossReal 3DGS reconstructs the coarse-to-mid bars with much higher fidelity than the median stand-in, losing only the finest bars (≤14.7 mm) where straw gaps get narrower than the bars.
Notes
training_*/,*.ply) are git-ignored — regenerate via the README.metalcompiler; a CPU fallback works but is slow.🤖 Generated with Claude Code