Image Processing with ROS

1. The usb_cam package can be used with many cameras
2. The camera is one of the /dev/videoX devices, and can be specified with the video_device parameter
3. Run the usb_cam node. A command with parameters that work well with the webcam on the System76 laptops are
   • ros2 run usb_cam usb_cam_node_exe --ros-args -p framerate:=30.0 -p pixel_format:=yuyv
   • Cameras have many settings that are important for machine vision. Use v4l2-ctl to view and tweak them
   • The camera_info_url parameter can be passed to provide a camera calibration file that for most cameras you will need to generate.
     • Without this file images can still be processed but pixel coordinates in the images cannot be converted to real world coordinates via ROS
4. After running you can view the image in several ways
   1. Use rviz2 and add an Image view
   2. ros2 run usb_cam show_image.py (Image might be laggy at high resolutions)
   3. ros2 run rqt_image_view rqt_image_view
   4. ros2 run image_view image_view --ros-args -r image:=/topic_image_is_published_on to view the image

• The v4l2_camera package is designed to expose many v4l2 camera options to ROS via parameters
  • It automatically enumerates all available v4l2 options (which vary by camera model) and exposes them
• It can be used with ros2 run v4l2_camera v4l2_camera_node

These camera drivers have limited or no ROS 2 support

1. cv_camera A camera driver that uses OpenCV to open the camera. So if you can use your camera in open CV this node should work
2. libuvc_camera (not released for noetic)

The Wiki articles are for ROS 1, however the concepts are the same for ROS 2, just replace rosrun with ros2 run (and a few node names are slightly different)

image_pipeline contains several packages relating to image manipulation in ROS.

• The idea is to chain various calibration and image processing steps together to complete computer vision tasks.
• The pipeline also applies to stereo-vision cameras and 3D point clouds as well as monocular images.

Camera Calibration finds the intrinsic parameters of a camera using a checkerboard pattern and OpenCV.

• See this tutorial (replace cameracalibrator.py with cameracalibrator)
• More information about camera calibration CameraInfo
• Some more information on calibration matrices for the pinhole camera model
• Matlab tutorial on camera calibration

• The calibration can be stored and used to provide a camera_info topic so that the calibration can be used by other nodes.
  • The msg/CameraInfo Definition has useful information about camera calibration
• Calibrations can be loaded by setting the camera_info_url parameter
  • Each driver may be different, camera_info_url is a ROS convention however
• It is also possible to perform a stereo calibration for two cameras, which results in receiving depth information

• The image_proc package handles rectifying and debayering images.
• Rectification is a method to compensate for camera lens distortion (which can make parallel lines appear curved in an image)
• Debayering is a compensation that accounts for the individual color pixel layout on color cameras More Info
• Run image_proc with ros2 launch image_proc image_proc.launch.py
  • This launchfile launches several component nodes (that is nodes that all run in the same process) that subscribe to /image_raw and /camera_info to create the rectified and debayerd images
  • TODO: For performance purposes, you may be able to run the usb_cam node inside the same image_proc container (I have not tried this yet…)
  • Usually, when doing computer vision, you want to use the rectified image
• image_proc publishes the following
  1. image_mono: A monochrome unrectified image
  2. image_rect: A monochrome rectified image
  3. image_color: A color unrectified image. These images are de-omsaiced (that is it accounts for the rgb pixel pattern of the camera)
  4. image_rect_color A color rectified image.
  5. The subtopics (e.g., image_rect/compressed) are various compressed versions of the image. These are referred to as transports.
  6. Transmitting uncompressed images takes a lot of bandwidth (even with modern networks), so compressed images are preferred
  7. However, there is no convenient way in python of subscribing to compressed images

depth_image_proc Processes depth image

• In a depth image, each pixel corresponds to the depth (i.e., how far light has to travel before hitting something) in the scene.
• Used by RGB-D cameras such as the intel RealSense.
• Used to make point clouds (basically 3 dimensional pixelized images) from calibrated depth cameras

1. Web Video Server used to create an http stream of video, useful if interfacing your robot to a website.

1. opencv_apps Each "app" is a node that performs a single image processing function using opencv. Thus you can chain together many computer vision algorithms by connecting some nodes together in a launchfile.

1. The official ros apriltags package is called apriltag_ros
2. Choose a Tag Family
   • Download images here: https://github.com/AprilRobotics/apriltag-imgs
   • Resize the images and print them (for example, use gimp):
     • Choose Image->Scale Image
     • Set the Size of the image (choose the resolution and then set the size in real-world units (e.g. meters)
     • Choose "None" for interpolation
     • Print it out
     • It is important that the tag's scaling is correct or the tag will not work.
     • April tags must have a boarder around them. There are some tag families that have a white border and others that have a black border
3. - See README.md for instructions