Sensor Fusion & Control Connecting Multiple Sensors and Mobility State into One System
Autonomous mobile systems use data from DVS, RGB cameras, LiDAR/Depth, IMU, and Odometry, each with different update rates and coordinate systems. If these data are not aligned accurately, overall system decisions can become unstable even when each sensor operates correctly.
Robowink’s Sensor Fusion & Control research focuses on aligning heterogeneous sensor data to common time and coordinate references, combining them with mobility-state information, and creating consistent state representations and control interfaces for higher-level systems.
DVS event timestamps, camera frame times, LiDAR scan times, IMU samples, and robot-state update cycles are different. Before fusion, they must be aligned to a common time axis.
Collects each sensor timestamp together with the system reception time.
Identifies time offsets caused by differences in sensor clocks and communication latency.
Maintains a recent data window in a buffer and matches sensor samples by time.
Interpolates state values when necessary to construct sensor and mobility information for the same point in time.
Flags data outside the allowed timing range or adjusts its fusion weight.
When sensors are mounted at different positions and orientations, the same object is observed in different coordinates. Extrinsic parameters and coordinate transforms are used to align observations into a common Robot Frame.
Calibrates camera and optical-system intrinsic parameters to reduce geometric distortion in image and event coordinates.
Defines the mounting-position and orientation relationships of DVS, RGB, LiDAR, and IMU in a common reference frame.
Transforms sensor observations consistently among Robot, Base, and Map Frames.
Compares sensor observations of the same object to check temporal and spatial alignment error.
Rather than combining every sensor in the same way, we study architectures that select the fusion stage according to data characteristics, computational cost, and latency requirements.
Combines raw or low-level sensor data that can be aligned in time and coordinates into a common representation to preserve as much spatial and temporal information as possible.
Combines ROI, motion features, depth, state vectors, and other features extracted from each sensor into a unified feature space.
Maintains sensor-specific outputs and confidence values while applying weighted combinations or priority rules at the system level.
External sensor information alone is not enough to fully understand the motion state of an autonomous mobile system. Internal states such as speed, heading, Odometry, IMU, and operating mode are combined to construct system context.
Uses current position and displacement together with sensor observations to maintain spatial context.
Combines vehicle speed and heading so the same external motion can be interpreted from a relative perspective.
Uses acceleration and angular-velocity information to supplement short-term changes in orientation and motion state.
Distinguishes the meaning of sensor data and interface conditions according to system modes such as driving, stopping, and task execution.
Sensor fusion requires more than simple averaging. The system must select which information to use based on sensor validity, environmental conditions, data freshness, and inference confidence.
Detects conditions that reduce sensor reliability, such as missing data, delay, saturation, or increased noise, and reflects them in the fusion process.
When the reliability of a particular sensor decreases, the weighting of other sensors and mobility-state information is adjusted to maintain a usable system-state representation.
This research delivers fused results and mobility state through standardized interfaces that can be used by higher-level Motion Planners and Robot Controllers.
State information in a common coordinate frame
Confidence and freshness validation
Message / register conversion
Motion / system input
State, error, and execution feedback
We study an integrated architecture that manages data formats and update cycles among sensors, edge modules, and robot controllers while monitoring processing latency and data loss.
Defines common fields required for fusion, including Timestamp, Frame ID, Pose, and Confidence.
Evaluates communication interfaces appropriate for the target system, including Ethernet, CAN-class, and Serial.
Manages queues and buffers according to data rate and priority so stale data does not remain in the control path.
Stores sensor, fusion, and control-interface logs and replays the same situations to analyze integration issues.
Evaluation focuses on how accurately multiple data sources are aligned and how reliably they are delivered in the actual system, rather than on individual sensor performance alone.
Sensor Fusion & Control is the final integration layer that connects the preceding three research technologies to a real robotic system.
Handles event sensing, calibration, preprocessing, and spatiotemporal data representation.
Converts event sequences into spike-based temporal features and AI inference information.
Executes SNN models on RISC-V/MCU, dedicated accelerators, and memory architectures.
Aligns and fuses all sensor and state information and connects it to real mobility-system interfaces.
We welcome joint research and PoC opportunities in DVS, RGB, LiDAR, and IMU alignment, state estimation, real-time sensor fusion, and robot-system interfaces.
