Beyond Point-Attached Semantics:
Object-Centric Semantic Fields for Generalizable Manipulation

Zheng SUN1 Lerong ZHANG1 Zhihao LI1 Zhuo LI1 Quentin ROUXEL1 Fei CHEN1
1The Chinese University of Hong Kong, China
Paper Video Code

We condition a continuous semantic field on observed object geometry and query part-aware embeddings at explicit 3D locations, producing semantic point clouds for downstream manipulation policies.

Abstract

Generalizable robot manipulation requires stable 3D understanding of functional object parts, such as handles, tool heads, openings, and graspable regions. Raw point clouds provide geometry but lack explicit part semantics, and their sampled points vary with viewpoint, sensor configuration, and object instance. Existing 2D feature lifting and discrete 3D point-wise features enrich point clouds with semantics, but the resulting features remain attached to observation-dependent samples. We propose an object-centric continuous semantic field that conditions on an object point cloud and reads part-aware semantic embeddings at explicit 3D query locations. The field is trained from part-annotated object models and then frozen to generate semantic point clouds as object-level conditioning for manipulation policies. Experiments on RoboTwin simulation tasks and real-world bimanual object manipulation show that our representation provides more stable functional-part cues and improves policy performance over raw point-cloud, 2D feature lifting, and 3D point-wise feature baselines.

Core Idea

Move from point-attached semantics to an object-conditioned semantic field that can be queried at controlled 3D locations.

Teaser of object-centric continuous semantic field
Figure 1. Existing 2D feature lifting and discrete 3D point-wise features attach semantics to observation-dependent samples. We instead condition a queryable semantic field on the object point cloud and read part-aware embeddings at explicit 3D query locations, producing semantic point clouds for downstream policy learning.

Method Overview

Object conditioning and semantic readout are separated: support points describe the instance, while query locations decide where semantics are evaluated.

Overview of object-centric continuous semantic field
Figure 2. Support points condition an object-specific tri-plane cache; 3D query locations read out part-aware embeddings and training-time part logits. The frozen field is then queried at resampled object locations to export semantic point clouds for downstream manipulation policies.
1

Condition on the Object

Observed support points encode the current object instance and construct a queryable object-level field.

2

Query Explicit Locations

Semantic embeddings are read at controlled 3D coordinates instead of being tied to sensor samples.

3

Export Semantic Point Clouds

Coordinates and embeddings are paired as policy-ready semantic point clouds for manipulation.

Results

Across simulation and real-world evaluations, queryable object-centric semantics improve success on tasks that require functional-part localization.

RoboTwin Simulation

Hang Mug37%
Beat Hammer84%
Open Microwave35%
Put Cabinet83%

Real Robot

Grasp Mug17/20
Beat Cube17/20
Stir Mug10/20
Pour Water10/20

Representation Analysis

Cross-instance feature colors are more consistent for corresponding functional parts.

Cross-instance feature visualization on real observations
Figure 4. Colors are obtained from a shared PCA per method and category. Compared with 2D lifting and 3D point-wise features, our field yields more consistent part-level colors across mug and hammer instances.

Tasks & Setup

The evaluation spans simulation tasks, real-world held-out object splits, and a bimanual robot platform with calibrated RGB-D sensing.

Simulation task examples used in evaluation
Figure 5. Simulation task examples used in our evaluation. These tasks require localizing functional object parts, such as mug handles, hammer heads, articulated handles, and placement regions, to complete object-centric manipulation.
Real-world tasks and object splits
Figure 3. Policies are trained on training object instances and evaluated on held-out test instances for mug grasping, tool-object contact, grasping and stirring, and pouring-related manipulation.
Real-world experimental setup
Figure 6. The bimanual robot platform is equipped with two calibrated ZED 2i RGB-D cameras, and the evaluated objects include mugs, hammers, stirring tools, and containers used in the real-world manipulation tasks.

Real-World Videos

Representative rollouts from the provided video set, grouped by task.

Grasp Mug

Mug 0

Mug 1

Mug 3

Mug 6

Beat Hammer

Hammer 1

Hammer 2

Hammer 5

Hammer 6

Stir Mug

Stirring 1

Stirring 3

Stirring 4

Stirring 5

Pour Water

Pour Water

Pour Water 2

Pour Water 3

Pour Water 4

Resources

Paper PDF Local manuscript file Videos All provided real-world rollouts Code GitHub repository

BibTeX

@inproceedings{sun2026beyondpointattached,
  title={Beyond Point-Attached Semantics: Object-Centric Semantic Fields for Generalizable Manipulation},
  author={Sun, Zheng and Zhang, Lerong and Li, Zhihao and Li, Zhuo and Rouxel, Quentin and Chen, Fei},
  booktitle={Proceedings of the 10th Conference on Robot Learning (CoRL)},
  year={2026}
}