3-D Sensing

1. An active light emitter outputs a pattern
2. Objects deform the pattern
3. Image sensor senses the pattern and, using the distortion, computes the depth map
4. Good for indoors and short range, because they require seing the projected patterns.
5. Can be interfered with when, for example, there are multiple devices outputting patterns

1. Two images taken some known fixed distance apart
2. By comparing the images taken from known different angles, the depth can be known
3. The farther an object is from the sensors, the more similar the two images are
4. No active light emission is required so they do not interfere with each other.
5. Range is limited by distance between the two sensors.
6. Active IR can be used to create known features and enhance the accuracy

• LiDar
• Measure depth by emitting light and timing how long it takes to reach the sensor
• Very long range

1. A depth map provides the distance from the camera in pixel coordinates
2. Using a camera calibration, this depth map is converted into real-world x,y,z coordinates
3. The pixels can be superimposed with data taken from an RGB camera to provide a 3D image

• The Intel Realsense D435i is an active stereo depth camera
• The realsense drivers and library are not yet released for Ubuntu 20.04 and the ros package is not yet released for ROS noetic
• Follow my instructions for installation

There are 3 ways of using the realsense in ROS. Official documentation is here for using SDK 2.0: https://dev.intelrealsense.com/docs/docs-get-started

1. Use the ROS node: For example, roslaunch realsense2_camera rs_camera.launch filters:=pointcloud align_depth:=true.
   • This is the easiest version to use for the most part, and also makes the realsense data available any ROS node
   • Usage instructions are on the github page https://github.com/IntelRealSense/realsense-ros
   • There is also Developer Documentation
   • Visualize the pointcloud by adding a PointCloud2 object in rviz
2. Use librealsense python wrappers: wrappers around librealsense for more advanced usage in python
3. Use librealsense from C++
   • This is for advanced and fast point cloud processing

• In ROS, point clouds are sent using the sensor_msgs/PointCloud2 message
  • You can access the message fields directly but the format is not the most intuitive
• The pcl_ros package is the primary ROS package for manipulating pointclouds
• It includes several stand-alone filters that can be loaded as nodelets to perform common operations

• The pcl_ros package also lets you convert point cloud messages into C++ data types for use with the Point Cloud Library
  • While this is the most common way of handling point clouds in ROS, it is also a C++ only way

• The sensor_msgs package has python tools for manipulating Pointloud2 data
  • See the point_cloud2 API
  • See This ROS answer for some details on how to index into the pointcloud.
• The ros_numpy package lets you convert many ROS message types (including PointCloud2) into easy-to-use numpy arrays
  • You can use this package to convert a point cloud into a numpy array, manipulate, and then convert back into a pointcloud message
• You can install python bindings for pcl using pip (pip3 install python-pcl) or as an Ubuntu package (sudo apt install python3-pcl)
  • The python-pcl package mostly parallels the C++ pcl API
  • This Answer demonstrates how to convert data from a PointCloud2 message into a pcl.PointCloud object