Pointmap Association and Piecewise-Plane Constraint for Consistent and Compact 3D Gaussian Segmentation Field

Wenhao Hu1, Wenhao Chai2, Shengyu Hao1, Xiaotong Cui1, Xuexiang Wen1, Jenq-Neng Hwang2 Gaoang Wang1
1Zhejiang University, 2University of Washington
Paper arXiv

Abstract

Achieving a consistent and compact 3D segmentation field is crucial for maintaining semantic coherence across views and accurately representing scene structures. Previous 3D scene segmentation methods rely on video segmentation models to address inconsistencies across views, but the absence of spatial information often leads to object misassociation when object temporarily disappear and reappear. Furthermore, in the process of 3D scene reconstruction, segmentation and optimization are often treated as separate tasks. As a result, optimization typically lacks awareness of semantic category information, which can result in floaters with ambiguous segmentation. To address these challenges, we introduce CCGS, a method designed to achieve both view Consistent 2D segmentation and a Compact 3D Gaussian Segmentation field. CCGS incorporates pointmap association and a piecewise-plane constraint. First, we establish pixel correspondence between adjacent images by minimizing the Euclidean distance between their pointmaps. We then redefine object mask overlap accordingly. The Hungarian algorithm is employed to optimize mask association by minimizing the total matching cost, while allowing for partial matches. To further enhance compactness, the piecewise-plane constraint restricts point displacement within local planes during optimization, thereby preserving structural integrity. Experimental results on ScanNet and Replica datasets demonstrate that CCGS outperforms existing methods in both 2D panoptic segmentation and 3D Gaussian segmentation.

Method

Example Image

Differences in mask association: Video vs. Pointmap. Video segmentation often struggle to maintain consistency during significant changes in camera views. In contrast, constructing a unified 3D point cloud field can ensure segmentation accuracy by leveraging spatial information.

Example Image

The pipeline of our method. (a) We first construct a unified point cloud field and establish correspondences between pixels using the pointmaps. (b) Leveraging these relationships, we construct a cost matrix for instance masks across two frames. The Hungarian algorithm is then applied to optimize the cost matrix, ensuring consistent mask association. By merging all frames, we obtain a point cloud enriched with consistent segmentation information. (c) This point cloud serves as the initialization for 3D Gaussians. To achieve compact 3D segmentation, we employ a piecewise-plane constraint, restricting point displacement within local planes through plane regularization and split projection.

2D Segmentation

RGB

CCGS (Ours)

Gaussian Grouping

RGB

CCGS (Ours)

Gaussian Grouping

3D gaussian segmentation

Note: You can interact using mouse controls: drag to rotate, scroll wheel to zoom in/out, and right-click drag to pan.

For visualization clarity, only the positional and categorical information of the trained Gaussian points is presented.

Mesh
Ground Truth
CCGS (Ours)
Gaussian Grouping
OpenGaussian (Coarse)
OpenGaussian (Fine)

Downstream Tasks

Before Deletion

CCGS (Ours)

OpenGaussian

Gaussian Grouping

Before Movement

CCGS (Ours)

OpenGaussian

Gaussian Grouping

BibTeX

@article{hu2025pointmap,
      title={Pointmap Association and Piecewise-Plane Constraint for Consistent and Compact 3D Gaussian Segmentation Field},
      author={Hu, Wenhao and Chai, Wenhao and Hao, Shengyu and Cui, Xiaotong and Wen, Xuexiang and Hwang, Jenq-Neng and Wang, Gaoang},
      journal={arXiv preprint arXiv:2502.16303},
      year={2025}
    }