Testing, Documentation, and Design
Testing
Program testing can be used to show the presence of bugs, never to show their absence!
\(-\) Edsger Dijkstra
Why Testing?
- Provides some level of assurance that your code is correct
- When your code changes, it helps you avoid breaking existing functionality
- Provides built in working examples of how to use your code
- Designing your code to be tested leads to better code
Why Automate Testing?
- Automation is the whole point of robotics!
- Automated tests happen without user intervention which increases the chance that they will be run and can serve their purpose to catch errors
Testing in ROS
- The ROS 2 testing philosophy discusses three types of tests: unit, integration, and systems
- These terms are standard terms, but when applied to ROS packages they take on more concrete meaning
Unit Testing
- Unit Testing usually refers to testing a single function
- Unit tests should ideally be run after every "compile" and can also be run after every push to a git repository using continuous integration
- Thus, unit tests should execute quickly and test only a single isolated item.
Mocking
- Sometimes (especially in robotics) a function takes a long time or is complicated or moves a robot
- Mocking lets you replace functions by just assuming they return a desired value
- Instead of actually doing anything, a mocked function just returns essentially hard-coded values that you specify
- Thus you can test code A, that uses B, by mocking B. This now no-longer tests B but it still lets you test A, under the assumption that B is working
- ROS 2 recommends mocking everything and having no dependencies
- Although mocking creates arguably the purest unit tests there are some problems with this approach:
- Mocking can be a lot of effort (when a mocked dependency changes, so must the mock itself)
- If something is mocked it is not benefiting from extra test coverage that testing another part of the system provides
- We already need the dependencies anyway so isolated the dependencies for testing is not overly helpful
- Mocking is a place to introduce new errors and bugs (e.g., did you correctly mock the object)
- With that said mocking can be helpful in many cases, especially if some code depends on something that takes a long time or moves a robot
- Although mocking creates arguably the purest unit tests there are some problems with this approach:
Integration Testing
- Integration tests are tests that require multiple components to work together to function
- These tests can take longer to run, are typically run less often then unit tests, and encompass a broader range of functionality
- In ROS, the term integration testing typically refers to tests between different components in the same package
Systems Testing
- A systems test is a type of integration test where a whole end-to-end system is testing
- ROS 2 suggests putting such tests in their own packages to maintain decoupling between packages
Test Strategies
- Write tests whenever you encounter a bug
- helps you reproduce the bug
- ensures that you won't encounter the same exact problem again
- Test Driven Development: write tests prior to writing code
- In robotics, testing is generally significantly harder than in most other disciplines
- Testing everything often requires physical hardware to move or sense in the real world
- Try to isolate as much of the "software" part of the robot from the hardware to facilitate testing
Unit Testing in ROS and Python
Frameworks
Pre-made Tests
- Standard pytest tests are created by default with
ros2 pkg
- These tests are mostly used to test stylistic items:
test/test_copyright.py
: - by default this is skipped, enable when you release code to ensure that there is a copyright notice in every filetest/test_flake8.py
: - enforce python styletest/test_pep257
: compliance with PEP 257 docstring conventions
- These tests are mostly used to test stylistic items:
Writing Pytests
- Create a file called
test_<something>.py
or<something>_test.py
in the test directory - In that file, write your tests
- import the code you wish to test
- For each test-case write a function called
test_<testname>
- In that function, call the code and then check results using
assert
- There is much more from there, but that's the basics
Writing Unittests
- import
unitttest
- Write a class that serves as a "Test Case"
- the class inherits from
unittest.TestCase
- each method in the class that starts with the word "test" defines a unit test
- Each unit test calls the code it wants to test and asserts the result using
special
unittest
assertion functions (e.g.,self.assertTrue
,self.assertEqual
)
- the class inherits from
Dependencies
- Anything that you import in a test becomes a
test_depend
in thepackage.xml
Integration Testing in ROS and Python
- This is a poorly documented and likely in-flux part of ROS
- The documentation is brief readmes, some examples, and the source code
- Integration testing is based on launch files
- There are three components to integration testing:
- launch/launch_testing: ROS agnostic testing, based on
unittest
- launch/launch_pytest: ROS agnostic testing, based on
pytest
- launch_testing_ros: ROS-specific testing, based on
unittest
- launch/launch_testing: ROS agnostic testing, based on
- The integration testing appears to be very powerful, letting you
- Directly test the output of processes
- Test how processes start and shut down
- Test messages, services and actions
- We will use launch_testing_ros
- This builds off of launch_testing so looking there for documentation is also useful
- Don't forget about the examples in these repositories
test_depend
onlaunch_testing_ros
- Write a
test_<something>_launch.py
file in thetest
directory Below is some code that highlights the differences between a launch file and a test launch file and explains how it works
import unittest import pytest from launch import LaunchDescription from launch_testing.actions import ReadyToTest # Mark the launch description generation as a rostest # Also, it's called generate_test_description() for a test # But it still returns a LaunchDescription @pytest.mark.rostest def generate_test_description(): # Create a node, as usual. But it's useful to keep it around node = Node(executable=, arguments=)... # return a tuple (LaunchDescription, extra_dict) # extra_dict is a dictionary of parameter names -> values that get passed to # test cases. return ( LaunchDescription([ node, # here is where you put all launch actions you want to run during the test # To test a particular launch file, include it ReadyToTest() # this is the last action. Can be used elsewhere somehow ]), {'myaction': node } # this is a way to pass the node action to the test case ) # The above returns the launch description. Now it's time for the test # The goal is essentially to create a node that can then be used in all tests to # call services and subscribe/publish messages # unlike a regular node, it is often useful to not subclass node but rather # just create it. It is also useful (and necessary) to spin_once() as needed class TestMyTestCaseName(unittest.TestCase): @classmethod def setUpClass(cls): """Runs one time, when the testcase is loaded""" rclpy.init() @classmethod def tearDownClass(cls): """ Runs one time, when testcase is unloaded""" rclpy.shutdown() def setUp(self): """Runs before every test""" # so before every test, we create a new node self.node = rclpy.create_node('test_node') def tearDown(self): """Runs after every test""" # so after every test we destroy the node # Is a new node necessary for each test, or could we # create the nodes when we setupClass? self.node.destroy_node() def test_my_test1(self, launch_service, myaction, proc_output): """In UnitTest, any function starting with "test" is run as a test Args: launch_service - information about the launch myaction - this was passed in when we created the description proc_output - this object streams the output (if any) from the running process """ # Here you can use self.node to publish and subscribe # launch_testing.WaitForTopics waits for something to be published on a topic # spin with rclpy.spin_once() # You can check and verify output with proc_output. launch/launch_testing has more information # You should specify a timeout for rclpy.spin_once() because if no ROS events happen it will # block forever (but if no events happen that is a bug and the test should fail
- This builds off of launch_testing so looking there for documentation is also useful
- Alert! Writing testing nodes can expose various race conditions in your code (which is a good thing) but these can be hard to diagnose. The order of nodes in a launchfile (including testing nodes) is not specified unless you work with events and explicitly do things to order them
- The me495_tf on the
ros2
branch has code that demonstrates some features as do the launch_testing packages themselves
Running Tests
- Before you can run tests, you must
colcon build
the workspace - All tests (unit and integration) can then be run with
colcon test
(without sourcing the workspace, it seems) - To see test results run
colcon test-result
- Often you may wish to see more details (especially when a test fails). Use
colcon test-result --verbose
- Sometimes you want to see the exact output of tests for a specific package. Use
cat log/latest_test/package/stdout_stderr.log
Quality
- ROS 2 has introduced a concept of Quality Levels for each package
- When a package adopts a quality declaration it is promising to meet certain quality standards
- Opinion: few packages seem to have adopted these, but those that have seem to uphold them pretty well.
- Maybe more packages will adopt them as they mature
- The declaration also has lower levels so that if you depend on a package with a low level then you know you should not be.
Documentation
Sphinx
- Sphinx is a tool that converts docstrings in your python source code into well-formatted documentation
- To install:
sudo apt install python3-sphinx
- The ROS 2 method of developing sphinx documents is still under development
- However, for now, put useful meaningful docstrings in your classes and functions
- To install:
Git
- Git is an important part of the documentation process
- Think of git as documentation for your collaborators (including your future-self)
- If you write good log messages, then you can go back and answer "Why was this done?"
- Look at the commit messages in the Linux Kernel Source Code!
- For more information about git, see Git Introduction
Design Guidelines
ROS adds methods for abstraction beyond those provided by a programming language. When thinking about the design of a ROS system, here are some questions that often arise and some guidelines about the trade-offs experienced in these decisions.
- How should the code be divided across nodes?
- Code in a single node is simpler and can take advantage of the abstractions provided by the programming language
- Code that is split across multiple nodes can be re-used and mixed-and-matched with other ROS nodes more easily
- Code that is split across multiple nodes is inherently parallel (since nodes run in parallel).
My (very loose) guideline is that if two nodes are
- Always are either both running or not running
- Subscribed to each other's topics
- Do not need to be running in parallel
then you should keep them in a single node.
- How should the code be divided across packages?
- A single package promotes simplicity
- Multiple packages enable re-use
- I start with a single package. If part of the package seems individually useful, I will spin it off into a separate package.
- How should the code be divided across repositories?
- Single repository is simpler and makes it easier to keep related changes between the packages synchronized.
- Multiple repositories promote decoupling between the packages, potentially making one more generally useful.
- For me, if the packages will be released together and are used for the same project, I keep them in the same repository