UP | HOME

Testing, Documentation, and Design

Testing

Program testing can be used to show the presence of bugs, never to show their absence!

\(-\) Edsger Dijkstra

Why Testing?

  • Provides some level of assurance that your code is correct
  • When your code changes, it helps you avoid breaking existing functionality
  • Provides built in working examples of how to use your code
  • Designing your code to be tested leads to better code

Why Automate Testing?

  • Automation is the whole point of robotics!
  • Automated tests happen without user intervention which increases the chance that they will be run and can serve their purpose to catch errors

Testing in ROS

  • The ROS 2 testing philosophy discusses three types of tests: unit, integration, and systems
  • These terms are standard terms, but when applied to ROS packages they take on more concrete meaning

Unit Testing

  • Unit Testing usually refers to testing a single function
  • Unit tests should ideally be run after every "compile" and can also be run after every push to a git repository using continuous integration
  • Thus, unit tests should execute quickly and test only a single isolated item.

Mocking

  • Sometimes (especially in robotics) a function takes a long time or is complicated or moves a robot
  • Mocking lets you replace functions by just assuming they return a desired value
    • Instead of actually doing anything, a mocked function just returns essentially hard-coded values that you specify
  • Thus you can test code A, that uses B, by mocking B. This now no-longer tests B but it still lets you test A, under the assumption that B is working
  • ROS 2 recommends mocking everything and having no dependencies
    • Although mocking creates arguably the purest unit tests there are some problems with this approach:
      1. Mocking can be a lot of effort (when a mocked dependency changes, so must the mock itself)
      2. If something is mocked it is not benefiting from extra test coverage that testing another part of the system provides
      3. We already need the dependencies anyway so isolated the dependencies for testing is not overly helpful
      4. Mocking is a place to introduce new errors and bugs (e.g., did you correctly mock the object)
    • With that said mocking can be helpful in many cases, especially if some code depends on something that takes a long time or moves a robot

Integration Testing

  • Integration tests are tests that require multiple components to work together to function
    • These tests can take longer to run, are typically run less often then unit tests, and encompass a broader range of functionality
  • In ROS, the term integration testing typically refers to tests between different components in the same package

Systems Testing

  • A systems test is a type of integration test where a whole end-to-end system is testing
  • ROS 2 suggests putting such tests in their own packages to maintain decoupling between packages

Test Strategies

  • Write tests whenever you encounter a bug
    • helps you reproduce the bug
    • ensures that you won't encounter the same exact problem again
  • Test Driven Development: write tests prior to writing code
  • In robotics, testing is generally significantly harder than in most other disciplines
    • Testing everything often requires physical hardware to move or sense in the real world
    • Try to isolate as much of the "software" part of the robot from the hardware to facilitate testing

Unit Testing in ROS and Python

Frameworks

  • Testing ROS python code involves several choices
  • There is the built-in python unit testing framework unittest
  • The pytest framework makes unit-testing in python easy

Pre-made Tests

  • Standard pytest tests are created by default with ros2 pkg
    • These tests are mostly used to test stylistic items: test/test_copyright.py: - by default this is skipped, enable when you release code to ensure that there is a copyright notice in every file test/test_flake8.py: - enforce python style test/test_pep257: compliance with PEP 257 docstring conventions

Writing Pytests

  • Create a file called test_<something>.py or <something>_test.py in the test directory
  • In that file, write your tests
    • import the code you wish to test
    • For each test-case write a function called test_<testname>
    • In that function, call the code and then check results using assert
  • There is much more from there, but that's the basics

Writing Unittests

  • import unitttest
  • Write a class that serves as a "Test Case"
    • the class inherits from unittest.TestCase
    • each method in the class that starts with the word "test" defines a unit test
    • Each unit test calls the code it wants to test and asserts the result using special unittest assertion functions (e.g., self.assertTrue, self.assertEqual)

Dependencies

  • Anything that you import in a test becomes a test_depend in the package.xml

Integration Testing in ROS and Python

  1. This is a poorly documented and likely in-flux part of ROS
    • The documentation is brief readmes, some examples, and the source code
  2. Integration testing is based on launch files
  3. There are three components to integration testing:
  4. The integration testing appears to be very powerful, letting you
    • Directly test the output of processes
    • Test how processes start and shut down
    • Test messages, services and actions
  5. We will use launch_testing_ros
    • This builds off of launch_testing so looking there for documentation is also useful
      • Don't forget about the examples in these repositories
    • test_depend on launch_testing_ros
    • Write a test_<something>_launch.py file in the test directory
    • Below is some code that highlights the differences between a launch file and a test launch file and explains how it works

      import unittest
      import pytest
      from launch import LaunchDescription
      from launch_testing.actions import ReadyToTest
      # Mark the launch description generation as a rostest
      # Also, it's called generate_test_description() for a test
      # But it still returns a LaunchDescription
      @pytest.mark.rostest
      def generate_test_description():
          # Create a node, as usual. But it's useful to keep it around
          node = Node(executable=, arguments=)...
          # return a tuple (LaunchDescription, extra_dict)
          # extra_dict is a dictionary of parameter names -> values that get passed to
          # test cases. 
          return (
              LaunchDescription([
                  node, # here is where you put all launch actions you want to run during the test
                        # To test a particular launch file, include it
                  ReadyToTest() # this is the last action. Can be used elsewhere somehow
                  ]),
                  {'myaction': node } # this is a way to pass the node action to the test case
                  )
      # The above returns the launch description. Now it's time for the test
      # The goal is essentially to create a node that can then be used in all tests to
      # call services and subscribe/publish messages
      # unlike a regular node, it is often useful to not subclass node but rather
      # just create it. It is also useful (and necessary) to spin_once() as needed
      class TestMyTestCaseName(unittest.TestCase):
          @classmethod
          def setUpClass(cls):
              """Runs one time, when the testcase is loaded"""
              rclpy.init()
      
          @classmethod
          def tearDownClass(cls):
              """ Runs one time, when testcase is unloaded"""
             rclpy.shutdown()
      
          def setUp(self):
              """Runs before every test"""
              # so before every test, we create a new node
              self.node = rclpy.create_node('test_node')
      
          def tearDown(self):
              """Runs after every test"""
              # so after every test we destroy the node
              # Is a new node necessary for each test, or could we
              # create the nodes when we setupClass?
              self.node.destroy_node()
      
          def test_my_test1(self, launch_service, myaction, proc_output):
             """In UnitTest, any function starting with "test" is run as a test
      
                Args:
                   launch_service - information about the launch
                   myaction - this was passed in when we created the description
                   proc_output - this object streams the output (if any) from the running process
             """
             # Here you can use self.node to publish and subscribe
             # launch_testing.WaitForTopics waits for something to be published on a topic
             # spin with rclpy.spin_once()
             # You can check and verify output with proc_output. launch/launch_testing has more information
             # You should specify a timeout for rclpy.spin_once() because if no ROS events happen it will
             # block forever (but if no events happen that is a bug and the test should fail
      
  6. Alert! Writing testing nodes can expose various race conditions in your code (which is a good thing) but these can be hard to diagnose. The order of nodes in a launchfile (including testing nodes) is not specified unless you work with events and explicitly do things to order them
  7. The me495_tf on the ros2 branch has code that demonstrates some features as do the launch_testing packages themselves

Running Tests

  1. Before you can run tests, you must colcon build the workspace
  2. All tests (unit and integration) can then be run with colcon test (without sourcing the workspace, it seems)
  3. To see test results run colcon test-result
  4. Often you may wish to see more details (especially when a test fails). Use colcon test-result --verbose
  5. Sometimes you want to see the exact output of tests for a specific package. Use cat log/latest_test/package/stdout_stderr.log

Quality

  • ROS 2 has introduced a concept of Quality Levels for each package
  • When a package adopts a quality declaration it is promising to meet certain quality standards
  • Opinion: few packages seem to have adopted these, but those that have seem to uphold them pretty well.
    • Maybe more packages will adopt them as they mature
    • The declaration also has lower levels so that if you depend on a package with a low level then you know you should not be.

Documentation

Sphinx

  • Sphinx is a tool that converts docstrings in your python source code into well-formatted documentation
    • To install: sudo apt install python3-sphinx
    • The ROS 2 method of developing sphinx documents is still under development
    • However, for now, put useful meaningful docstrings in your classes and functions

Git

  • Git is an important part of the documentation process
  • Think of git as documentation for your collaborators (including your future-self)
  • If you write good log messages, then you can go back and answer "Why was this done?"
  • Look at the commit messages in the Linux Kernel Source Code!
  • For more information about git, see Git Introduction

Design Guidelines

ROS adds methods for abstraction beyond those provided by a programming language. When thinking about the design of a ROS system, here are some questions that often arise and some guidelines about the trade-offs experienced in these decisions.

  1. How should the code be divided across nodes?
    • Code in a single node is simpler and can take advantage of the abstractions provided by the programming language
    • Code that is split across multiple nodes can be re-used and mixed-and-matched with other ROS nodes more easily
    • Code that is split across multiple nodes is inherently parallel (since nodes run in parallel).
    • My (very loose) guideline is that if two nodes are

      1. Always are either both running or not running
      2. Subscribed to each other's topics
      3. Do not need to be running in parallel

      then you should keep them in a single node.

  2. How should the code be divided across packages?
    • A single package promotes simplicity
    • Multiple packages enable re-use
    • I start with a single package. If part of the package seems individually useful, I will spin it off into a separate package.
  3. How should the code be divided across repositories?
    • Single repository is simpler and makes it easier to keep related changes between the packages synchronized.
    • Multiple repositories promote decoupling between the packages, potentially making one more generally useful.
    • For me, if the packages will be released together and are used for the same project, I keep them in the same repository

Author: Matthew Elwin