Final Project (Fall 2015)
Table of Contents
Competition is Thursday, December 10 from 3-5 PM
Final submission due Sunday, December 13 at 5 PM
1 Introduction
For the final project in this year's course, we are going to split up into 5 teams of 4 students, and have a Baxter part-sorting competition. The scoring (described below) is setup to allow a variety of different strategies to be successful. Ultimately, it will be up to each team to decide how they want to try and score points. I'm going to invite the graduate students and faculty of the NxR lab to watch the competition; so this is also a good opportunity to impress some people! I'm also going to try and recruit several guest judges to help decide the winner.
2 Specifications
Your goal is to use the Baxter robot to autonomously pick objects off of a work surface and place them in a container. A finite number of points will be awarded based on the number of objects successfully picked and the time taken to pick them. These points will then receive a number of multipliers based on the level of autonomy that the team used and the complexity of the pick-and-place problem that they tackled. Additionally, the judges will have a set of points that they may add to this score based off of a set number of categories.
2.1 Timing
Each team will have 20 minutes run their trials in the competition and 3-5 minutes of setup time to transition between teams. During the 20 minute window, each team can attempt an unlimited number of trials. The team must signify to the judges when the timing for a particular trial should begin. Once a trial has begun, it is concluded when either (a) 6 objects have been successfully picked and placed, or (b) the team indicates to the judges that the trial has concluded, or (c) the total team time reaches the end of the 20 minute window. Only the highest-scoring trial from the 20 minute window will count in the final team rankings.
During an individual trial, a team may elect to physically intervene is something went wrong. For example, let's say a pick maneuver is attempted, but the robot fails to grasp the target object (maybe the gripper hits the object, or maybe the object falls out of the gripper). If the team physically intervenes, they must first tell the judges that they wish to intervene. During the intervention the time keeps running and a two minute penalty time is added per intervention. Only three interventions are allowed per trial. It is up to the judges' discretion to decide if an intervention is fair. A fair intervention includes pausing Baxter's motion, and then using the cuff buttons to re-open and close the grippers to allow re-insertion of a dropped part. An unfair intervention would be using Baxter's gravity compensation mode to execute a pick maneuver through manual manipulation. If the judges declare an intervention unfair, that trial may not be counted.
2.2 Environment
Teams may use whichever Baxter they prefer, and they are free to adjust the relative positioning of Baxter and the table. We have several tables with adjustable heights, and teams should feel free to adjust the height of the table. Baxter may be mounted on his pedestal at two different heights. The shorter height is optimal for work surfaces between 29 and 33 inches (74-84 cm), and the taller height is better for work surfaces between 32 and 36 inches (81-91 cm). If teams are having trouble with workspace height, we can either adjust the tables or we can adjust Baxter's mount height. You should not expect to adjust Baxter's height during the competition.
If an external sensor is desired, that is fine, and it can be placed anywhere within Baxter's reachable workspace (it does not need to be mounted on Baxter or the table; e.g., you could put a Kinect on a tripod). No permanent modifications to Baxter may be made. If you would like to strap a Kinect to Baxter's head, that is probably okay, but we need to check that it wouldn't interfere with the other teams.
You may select the area on the table where the parts are originally located, but it must be clearly delineated from the zone where parts are to be deposited. You may also select where to deposit the parts, but that zone must also be clearly delineated. The deposition zone must have an area no larger than a 12 inch by 12 inch square (30 cm x 30 cm). At the end of the trial, only parts that began in the pick zone and ended up in the deposition zone will be counted. If you successfully place a part, but then later knock it out of the deposition zone, then that part will not count. Feel free to drop parts into the zone. You may use some sort of container to define your the deposition zone (e.g. a cardboard box). You may also define these zones using markings on the table.
We are only going to use the grippers provided with the Electric Parallel Gripper kits sold by Rethink. The grippers in the kit have a 44 mm throw, and a variety of fingers and fingertips that can be used. Changing to a different finger width or length, or changing the tips on the grippers is quite easy. If necessary, you will be able to make adjustments to your grippers before running your trials.
2.3 Scoring
Measured Points
- Simple Pick Points (max 6 points)
- Every object that is successfully picked from the pick zone and placed into the deposition zone, that remains in the deposition zone throughout the trial will earn the team 1 point. The simplest version of earning these points likely involves a set of hard-coded object locations or arm configurations. You can only earn up to 6 points in this category. You may place more than 6 objects on the table in pick zone.
- Timing Points
- Teams will be awarded points for speed. These points will be normalized by the number of Simple Pick Points received. Ideally, teams that pick more objects will always beat teams with faster times. However, it is feasible for a team to pick very few object, but do it very fast and end up with a higher score than a team with more objects. Please don't do this.
Multipliers
Below is a discussion of a variety of ways that teams may add multipliers to their scores. Only the multipliers used in an individual trial count towards the trial score. If you have other ideas for multipliers, please let me know! It is up to the judges to decide if you have successfully satisfied the multiplier requirements. Make sure the judges know what multipliers you are attempting to use in a given trial.
- Dynamic Part Carrier (\(m=1.25\))
- A good strategy for easily picking up all 6 Simple Pick Points would be to fabricate a simple jig or part carrier that guarantees that the relative pose of all of your objects is constant. Then you only need to calibrate the pose between the carrier and Baxter, and then Baxter should be able to pick up all parts. You will be awarded this multiplier if, at the beginning of the trial, Baxter is able to automatically estimate the pose of a part carrier. In other words, I could place the carrier anywhere I want in your pick zone, and Baxter would still be able to pick up the parts.
- Online Dynamic Part Carrier (\(m=1.25\))
- If you are awarded this multiplier, you are also automatically awarded the Dynamic Part Carrier multiplier. The primary modification is that I should be able to arbitrarily move the carrier between picks, and Baxter will adjust, on-the-fly.
- Unknown Part Locations (\(m=2.0\))
- This multiplier is awarded if the translation between your pick objects can be chosen arbitrarily at the start of the trial.
- Unknown Part Locations and Orientations (\(m=1.25\))
- If this multiplier is awarded, you are also automatically awarded the Unknown Part Locations multiplier. The modification is that the orientation of the parts may also be selected at random at the beginning of the trial.
- Dynamic Part Locations (\(m=1.25\))
- If this multiplier is awarded, you are also automatically awarded the Unknown Part Locations multiplier. The modification is that Baxter should be able to adjust to translations of the parts between picks.
- Dynamic Part Locations and Orientations (\(m=1.25\))
- If this multiplier is awarded, you are also automatically awarded the Unknown Part Locations multiplier and the Dynamic Part Locations multiplier. The modification is that Baxter should be able to adjust to random translations and rotations of the parts between picks.
- Multiple Object Types (\(m=1.5\))
- This multiplier is awarded if your demo is capable of simultaneously picking two substantially different object geometries. For example, your collection of parts is made up of spheres and cubes (not different sized cubes).
- Sorting Parts w/ Metadata (\(m=1.5\))
- This multiplier is awarded if you are able to distinguish between two different parts and sort them into two different deposition zones with the assistance of "metadata". So, for example, you may place tags (AR, QR, etc.) on the objects to allow Baxter to easily tell which "type" of object each object is.
- Sorting Parts w/ Perception (\(m=2.0\))
- This is very similar to the Sorting Parts w/ Metadata multiplier, but it is more challenging because Baxter must be able to distinguish between the types of objects using only characteristics of the objects and sensors. For example, Baxter must be able to distinguish between two parts of different colors or different shapes.
- Dual Arm (\(m=2.0\))
- To be awarded this multiplier, you must simultaneously use both arms in a way that is more efficient than using a single arm at a time. For example, one route to accomplishing this goal would be to use one arm to capture and process images to decide the pose of the next object while simultaneously planning and executing a pick-and-place motion with the other arm. You can not get this multiplier by simply toggling which arm you use to pick and place objects.
- User Interface (\(m=2.0\))
- You can receive this multiplier by implementing a user interface that allows an operator to control the operation of the picking and placing operations. For example, you could provide an interface that allows a user to select which object should be picked or placed next. Another example would be providing an interface that allows a user to correct mistakes in perception. For example, let's say you try and estimate the orientation of an object on the table, and your image processing algorithm misjudges the orientation by 30 degrees. The user interface could allow the user to correct this angle to improve grasp reliability. Making a meaningful and useful interface is likely a bit tricky, but it is also likely a good strategy for increasing the level of robustness that you are able to achieve. Note that creating an nice, high-level interface that allows a user to completely direct Baxter (with no on-board perception or autonomy) may disqualify you from other multipliers. Correcting a misjudged orientation is fine; completely specifying the orientation with an interface will disqualify the orientation-specific multipliers from above.
Judged Points
The judges have several categories, where they can assign bonus points if they choose. The categories are roughly defined below, but ultimately, it is up to the judges' discretion and interpretation of the categories to decide how and if to allocate these points.
- Ambition (max 5 points)
- Points awarded for attempting a particularly ambitious task or solution strategy.
- Creativitiy (max 5 points)
- Points for solutions to common problems that are unique or creative.
- Technical Implementation (max 5 points)
- Points for uniquely solid technical implementations of solutions. This could include techniques that work very robustly, or strategies that require higher levels of technical competence to implement.
Overall Score
The overall score will be calculated with the following equation:
\[ S_t = \left( N + \frac{N}{T + P}\right) M_{tot} + J \]
where
- \(S_t\) ~ The total score received for a trial. This is the score that will be used for ranking teams.
- \(N\) ~ The number of Simple Pick Points received (max. of 6)
- \(T\) ~ The total time of the trial being scored (expressed in minutes).
- \(P\) ~ The penalty time for interventions (expressed in minutes). This time is calculated by multiplying the number of interventions in the trial (max. of 3) by 2 minutes.
- \(M_{tot}\) ~ This is the total multiplier received for the team during the execution of the trial. This is calculated by adding up all multipliers that were employed during trial execution (as determined by the judges).
- \(J\) ~ The total points awarded by the judges.
3 Deliverables
For the final project, each student will receive three separate grades.
- All students will receive a group grade. This grade will be assigned based on the effort put forth in the competition and final submission. All members receive the same grade.
- Each student will receive a group assessment grade. You will be asked to assess your own group. This grade will depend on the quality of your assessment (mostly these are completion points).
- Each student will receive individual points for their effort on the final project. Ideally every student will receive full marks for this individual grade. However, if I hear reports of group members not fully contributing, they may not receive full credit.
3.1 Competition
Your team is expected to show up to the competition and try their best during their allotted time. You should also expect to pay attention and cheer on the other teams when you are not setting up for your turn.
3.2 Write-Up
Each team will be expected to submit a single GitHub repository via Canvas. This repository should include well-written documentation about how to run your demos, what the important nodes and topics are, high-level conceptual overviews, etc. I will be looking at how well you used Git as a team, how well your package (or meta-package) conforms to ROS standards, and the quality of your Python code. For the write-up and the competition portion of the final project, each group member will be receiving the same grade.
3.3 Team Assessment
In addition to a single group write-up each team member will individually be responsible for submitting a single group assessment via Canvas. Each assessment will be kept completely confidential, and I will use the assessments to help me determine how to allocate individual points.
4 Resources and Considerations
- Tags
- There are a variety of tag tracking packages in ROS. These packages
are usually designed to provide a pose estimate from a camera frame to
the real-world location of some sort of visual tag (AR tags, QR codes,
etc.). Some of the packages can also read metadata off of the tags. These
packages may be a great way to calibrate your system, identify graspable
objects, etc. They are also, generally, pretty easy to use. Typically all
you need is a calibrated camera, some topic remapping, and a few
configuration files.
- Mini Project: During this course last year, two students did a fantastic mini project where they provided tutorials on how to use several different tag tracking packages, and they compared and contrasted the pros and cons of each.
- Project from Last Year: This project last year used a couple of
different tag tracking packages with Baxter. This may be a good resource
to see how to use
ar_track_alvar
andvisp_auto_tracker
with Baxter.
- Perception
- For many of the multipliers you are likely going to be wanting
to use some sort of perception. Below are several obvious tools for
perception.
- OpenCV: This is by far the most widely-used computer vision library.
This package integrates easily with ROS via the
cv_bridge
package. I will provide one or two introductory lectures on this in the coming weeks. - PCL: The Point Cloud Library "is a standalone, large scale, open project for 2D/3D image and point cloud processing." It is nicely integrated with ROS via the perception_pcl and the pcl_ros packages. If you are interested in using a 3D camera such as a Kinect, this is how to do it.
- Camera Calibration: The lenses on cameras tend to distort images. In order to accurately use a camera for perception, it is often desirable to use a calibration procedure to remove these distortions. Most of the tag tracking tools require calibrated cameras. Camera calibration is easily done in ROS using the camera_calibration package. The YAML files produced by this calibration are then fully compatible with the ROS image_pipeline.
- OpenCV: This is by far the most widely-used computer vision library.
This package integrates easily with ROS via the
- Motion Planning
- Motion planning for a robot arm in ROS is accomplished almost exclusively with MoveIt!. This project can be accomplished with quite simple motion planning strategies. However, if your team chooses to leverage MoveIt!, you will certainly have significantly enhanced capabilites (at the expense of higher complexity). I will lecture on this in the coming weeks.
- Examples
- Here are a few good pick-and-place Baxter demos that you may
find useful. These may be good starting places, but they are far from
perfect. I strongly encourage you to use these for inspiration not
solutions. If you use code from any examples online, you must cite your sources.
- Jon Rovira Pick-Place Demo: A few years ago, an undergraduate working with me put together a quick demo that allowed Baxter to find and grasp a red cube on a table (he was using a Rubik's cube, red-side-up). It worked reliably, but was sensitive to the height of the table.
- Visual Servoing Baxter Example: This is an example from Active Robots that is quite similar to what we are trying to do. Do not plan on copying this code and having it work.
- Baxter Pick and Learn: This example is from VUB/ULB universities in Belgium. "t uses a shape recognition algorithm to identify similar shapes and place them to a location previously demonstrated by the operator." I have not studied or used this example, but it could be helpful.
- Simulators
- Gazebo: As promised, I have uploaded a copy of Rethink's official Gazebo-based Baxter simulator to Canvas. Feel free to use this, and feel free to share any Gazebo worlds that you create.
- V-REP: If you'd like to investigate using V-REP instead-of or in-addition-to Gazebo, feel free to do so. Check out Jon Rovira's demo for getting started.
- State Machines
- Likely the best ROS way to implement the behaviors that you want are through actions (you certainly don't need actions). ROS provides the SMACH package to quickly build a hierarchical state machine that is implemented via actions. I may spend some time lecturing about this.
- Setup and Calibration
- Likely a big concern you should have is how to quickly guarantee that the environment is configured properly for your code. You could certainly rely on manual calibrations (jigs, tape measures, etc.). Alternatively, you could write some calibrations routines that allow your system to adjust based on the current environment setup. Obviously the automatic solution is more advanced, but a manual procedure will likely work fine. Be sure that you know how to get calibrated quickly so that you don't waste your time during the competition.
- Trajectory Action Server
- The Baxter software provides a Joint Trajectory Action Server that allows Baxter to execute motions through a sequence of joint positions and possibly velocities/accelerations. You may have much better luck achieving high precision motions if you properly use this tool because Baxter can do a much better job of accounting for his own dynamics if the trajectory is known ahead of time.
- Interventions
- The human interventions are only going to be useful if you have a way to easily utilize them. You may want to think about writing some code that provides you with an easy interface for pausing and resuming the execution of your system. I can provide some advice on how to accomplish this.