Final Project (Fall 2018)

For the final project in this year's course, we are going to split up into 5 teams of 5 students and one team of 4 students, and have a Baxter/Sawyer manipulation competition. The guidelines for what you must accomplish are intentionally only loosely specified. Judges will be judging creativity, complexity, and reliability.

Below are some ideas that you could possibly incorporate into your projects to serve as inspiration.

GUI

It would be great to build a nice GUI to interact with your project. This GUI could allow changing of settings, control of the system state, or user interactivity. This could be done with something like rqt or interactive markers in rviz, or you could write a custom GUI using something like Tkinter or PyQt.

Web Interface

You could use something like Robot Web Tools and rosbridge_suite to build a web-based interface for communicating with your robot. You could even set this up to work over the internet (not just local network).

Simulation

You could base some of the technical content of the project largely around building a complex simulation environment in Gazebo or V-REP. You could even have the outputs of the simulation feeding into the decisions of the real robot (a simple example would be to use V-REP for solving IK).

IK

You could solve IK by implementing your own algorithms or adapting the Modern Robotics library. An interesting piece of a project would be to compare the IK solutions from pre-configured IK tools like IKFast, Baxter's IK service, trac_ik, bio_ik, and KDL.

Force/Torque Control/Sensing

You could do some very interesting things using the robot's torque control modes, or you could use the estimate of end-effector force to try and control the arm to apply specific forces to an object or to estimate the inertia of an object. This could be used ensure the arm is compliant in one or more directions, for keeping pressure on some object constant (e.g. drawing on a surface), or for implementing advanced object placing strategies (e.g. when inserting a peg into a hole you could measure when forces applied to the peg). Sawyer's motion interface might be useful for this.

Responsive Grasping

You could try and estimate whether grasps have been successful by utilizing forces measured by the grippers.

Joint Velocity Control

You could have the arm stabilize to target EE poses or trajectories by writing a joint velocity controller (note that Chapter 11 of Modern Robotics covers this).

Visual Servoing

You could implement a controller that continuously tracks a target object and adjusts the control as the estimate of the object's relative pose evolves. This is different from simply measuring where an object is, planning a path to get there, and then blindly executing that path.

Automatic Calibration

You could implement a routine that allows the robot to automatically calibrate to the surrounding environment. For example, imagine that you have an external camera, and that you are relying on knowing the pose of the camera relative to a fixed frame on the robot. Your calibration could involve taking pictures of a tag with the external camera and one of the robot's cameras. Then, since both cameras have an estimate of the tag location in their respective camera frames, you could calibrate the pose from the robot to the camera. This idea could also be used for calibrating the pose from the robot to a "work cell" (e.g. a table that the robot will be working with).

Face Communication

You could use the face display to animate/illustrate/indicate the how your demo is functioning. For example, you could display the output of motion planning schemes or image processing algorithms directly on the face, or you could use the display to print out the status/operation mode of your demo.

Button Interfaces

You could use the robot's built-in arm buttons for implementing some sort interactivity.

Voice Command

Tools such as pocketsphinx or gspeech could be used to implement voice-based control (would likely want an external, directional microphone).

Face Recognition

You could recognize/track users with tools such as face_recognition, face_detector, or OpenCV's face recognition module.

Dynamic Reconfigure

You could use dynamic_reconfigure to create parameters that are easy to modify during a running demo.

Other Robots

If a team was ambitious, they could likely come up with a demo where Baxter/Sawyer was somehow "teaming" with another robot (e.g. a TurtleBot or Jackal).

For the final project, each student will receive four separate grades.

1. Each member of a team will receive a proposal grade. Theses points will be completion points. Details of the proposal will be posted in Canvas.
2. All students will receive a group grade. This grade will be assigned based on the effort put forth in the competition and final submission. All members receive the same grade.
3. Each student will receive a group assessment grade. You will be asked to assess your own group. This grade will depend on the quality of your assessment.
4. Each student will receive individual points for their effort on the final project. Ideally every student will receive full marks for this individual grade. However, if I hear reports of group members not fully contributing, they may not receive full credit.

Tags

There are a variety of tag tracking packages in ROS. These packages are usually designed to provide a pose estimate from a camera frame to the real-world location of some sort of visual tag (AR tags, QR codes, etc.). Some of the packages can also read metadata off of the tags. These packages may be a great way to calibrate your system, identify graspable objects, etc. They are also, generally, pretty easy to use. Typically all you need is a calibrated camera, some topic remapping, and a few configuration files.

• Mini Project: During this course in 2014, two students did a fantastic mini project where they provided tutorials on how to use several different tag tracking packages, and they compared and contrasted the pros and cons of each.
• Project from 2014: This project last year used a couple of different tag tracking packages with Baxter. This may be a good resource to see how to use ar_track_alvar and visp_auto_tracker with these robots.
• AprilTags: These days, I'm seeing more and more people using AprilTags, and my experience with them so far has been quite positive. The best ROS package I've encountered to work with AprilTags is apriltags2_ros.

Perception

For many of your demonstrations, you may be interested in using some form of external world perception. Below are several tools that may be helpful for perception.

• OpenCV: This is by far the most widely-used computer vision library. This package integrates easily with ROS via the cv_bridge package. I will provide one or two introductory lectures on this in the coming weeks.
• PCL: The Point Cloud Library "is a standalone, large scale, open project for 2D/3D image and point cloud processing." It is nicely integrated with ROS via the perception_pcl and the pcl_ros packages. If you are interested in using a 3D camera such as a Kinect, this is how to do it.
• A few years ago a group was doing a project where they wanted to use PCL to extract point cloud clusters for objects sitting on a table, and then calculate the centroid of each of those clusters. I threw together the nodelet_pcl_demo that uses a series of nodelets from pcl_ros to do pass-through filtering, downsampling, plane fitting, coordinate transformations, and cluster extraction. There is also a simple custom message that allows easy passing of a small number of points from C++ to Python. It isn't as versatile as a sensor_msgs/PointCloud2 message, nor is it as space efficient. It is however, much easier to parse into a Numpy array.
• Camera Calibration: The lenses on cameras tend to distort images. In order to accurately use a camera for perception, it is often desirable to use a calibration procedure to remove these distortions. Additionally, geometric information of the camera is required to determine how to map pixel coordinates of an object to real-world coordinates. Most of the tag tracking tools require calibrated cameras. Camera calibration is easily done in ROS using the camera_calibration package. The YAML files produced by this calibration are then fully compatible with the ROS image_pipeline. I've run into occasions in the past where teams wanted to load up a YAML file from a calibrated camera, but it was a bit annoying to do with the robots that we have because they have their own internal calibrations that are already published. So, I created the camera_info_manager_node package that contains a single node that allows you to easily publish the data from a YAML file at an arbitrary rate and on an arbitrary topic. This way you can easily change the calibration information that your camera is using.

Motion Planning

Motion planning for a robot arm in ROS is accomplished almost exclusively with MoveIt!. Many of your demonstrations can likely be accomplished with quite simple motion planning strategies. However, if your team chooses to leverage MoveIt!, you will certainly have significantly enhanced capabilites (at the expense of higher complexity). I will lecture on this in the coming weeks.

Simulators

• Gazebo: Rethink has released a baxter_simulator package that could be useful for developing some of your demonstrations. It is not currently on apt-get for Melodic, and my first attempt at building from source failed, but my instinct is that the issues would be pretty easy to work around. Check out the main documentation page for more. Similarly, Sawyer also has a sawyer_simulator package I was able to build and run this with little effort.
• V-REP: If you'd like to investigate using V-REP instead-of or in-addition-to Gazebo, feel free to do so. Check out Jon Rovira's demo for getting started. Both Sawyer and Baxter have similar support.

State Machines

Likely a good ROS-y way of implementing the behaviors that you want are through actions (you certainly don't need actions, they are somewhat complicated). ROS provides the SMACH package to quickly build a hierarchical state machine that is implemented via actions. Here is a Baxter Demo Manager state machine a student wrote a few years ago using SMACH. You could also use a behavior tree implementation such as behavior_tree or pi_trees.

Setup and Calibration

Likely a big concern you should have is how to quickly guarantee that the environment is configured properly for your code. You could certainly rely on manual calibrations (jigs, tape measures, etc.). Alternatively, you could write some calibration routines that allow your system to adjust based on the current environment setup. Obviously the automatic solution is more advanced, but a manual procedure will likely work fine. Be sure that you know how to get calibrated quickly so that you don't waste your time during the competition.

Trajectory Action Server

The Baxter/Sawyer software provides a Joint Trajectory Action Server that allows the robot to execute motions through a sequence of joint positions and possibly velocities/accelerations. While this interface is not the easiest to use, I've had much better luck achieving high precision motions when properly using this tool because the robot can do a much better job of accounting for its own internal dynamics if the trajectory is known ahead of time.