Introduction to Python
Overview
This document is a guide to python tailored to the MS in Robotics program at Northwestern University. It is meant to be a quick reference and a place to gather external resources rather than a full tutorial. For more information see Other Learning Resources.
Python Basics
Python REPL
- A read-evaluate-print loop (REPL) lets you code interactively. Can run
python3
or (recommended)ipython3
to enter the REPL. - IPython has many nice features: for example better tab completion and (with a little work) auto reloading of changed modules
- Start your REPL!
Best Practices
- These practices are good to think about, even when learning.
- It is okay and even expected that you will not yet understand everything in this section, but you should be aware that practicing programming as an engineering discipline requires thinking about not just the functionality of the code but also its other properties (e.g., maintainability and reliability).
- PEPx - Python Enhancement Proposals are official python statements about the language
- Must Read: PEP20 to understand python's philosophy (its short).
- Read PEP8 to learn how to "properly" format your code.
- The Google python style guide also has some good tips.
- Proper Documentation is important and expected for your code (we can slack a little during the hackathon but not in class or for projects).
- If you create a file, use a docstring to describe what the file is for
- PEP257 discusses docstrings
- When you write a class (and python is Object Oriented so you should be), use a docstring to describe the class
- When you write a method or function, use a docstring to describe the method and parameters
- Docstring should be formatted for use with sphinx, which, using
autodoc
can be setup to automatically generate nice documentation.- Can use google docstring format, scipy format or native sphinx format within the docstrings themselves.
- Code without documentation is incomplete and time-savings from avoiding documentation prove illusory the more a project is used.
- If you create a file, use a docstring to describe what the file is for
- Testing
- In a dynamic language like python, unit testing is especially important.
- For many projects (e.g., research-level as opposed to safety-critical) I follow a practical philosophy when it comes to tests that I attempt to codify here:
- Tests are your friend, bugs are the enemy.
- If you encounter a bug, first write a test to reproduce it, then fix it.
- If writing a test will help you think through some code (e.g., by providing example usage), write the test first.
- If writing some code will help you think through a test, write the code first (but don't forget the test).
- If the goal of the software involves safety, reliability, or maintainability then it also involves proper testing.
- Always remember that tests need to be maintained alongside code. Focus testing efforts where benefits outweigh the costs.
- Test interfaces not implementations.
- If it is easy to write the test, do it.
- If it is hard to test some code consider refactoring, but don't refactor just for the sake of testing.
- Some tests are better than no tests.
- In robotics, testing involves hardware as well as software.
- Python has a built-in unit testing framework called unittest.
- The third-party pytest unit testing framework is commonly used because it enables testing with less boilerplate code
Language Properties
Some basic properties of python
:
- Interpreted: reads line by line and converts source code to machine code (not entirely true, it is more optimized than this crude simplification).
- Object oriented: everything is an object, even basic integers (object is collection of data and methods, called members):
- use
dir()
to see methods of an object. What methods does the interger1
have?
- use
- Strongly, but dynamically, typed:
- Every object has a type.
- Using the wrong type causes a runtime error.
- Use
type
to see the type of an object.
- Duck Typed - "if it quacks like a duck it's a duck".
- If you call methods on an object, as long as the object has those methods defined, it will not be a type error.
- Much of the flexibility in python comes from this property, which enables users to avoid needing rigid class inheritance hierarchies.
Syntax Properties
- Significant Whitespace: tab, newlines, spaces matter.
- As per PEP8, don't use tabs.
- Code blocks are determined by indentation level.
- A good text editor helps you avoid worrying about formatting too much.
- For C and C++ folks: lines do not end in a semicolon.
- Branching statements, function and class definitions end with a colon
:
. - Functions and variables must start with a letter or underscore and are case sensitive.
- Class members are accessed with
.
. Functions are called with()
. - Private class members start with an underscore, but can still technically be accessed.
- Strings can use single
''
or double""
quotes. - Python has a module system. Modules can be imported using
import
statements orfrom
statements.- How a module is imported determines how it changes the default namespace.
- If you
import amodule
you access elements in that module asamodule.item
- If you
from amodule import X
you can access X directly - You can do
import module as m
to access items in module asm.item
- Generally, you should avoid doing
from module import *
which brings all items in module into your namespace, but it can be useful, especially to test something in therepl
quickly
Built-in Types
Some types are built in to the python interpreter (as opposed to types you define yourself).
- int, float, long, complex, boolean (True, False).
- Sequence Types: strings, lists, tuple
- Strings - can use double or single quotes. "Hello" or 'hello'
- list - a collection of objects, can be modified, uses square brackets []. Objects in a list need not be the same type [1, "2434", True].
- tuple - a collection of objects, can't be modified, uses parentheses (a, b)
- dictionary - type of hashtable { key : value}
- Other types are: unicode, bytearray, buffer, xrange.
- Note: xrange is not in python3, all strings are unicode in python3, these are some of the major differences between python2 and python 3
- List indexing - first index is 0. Can also do -1 to index from back to front.
[3:4]
returns part of a list. - Mapping type: dictionary (aka a hashtable). Associates a key with a value ={"key" : value}.
- collections: high performance and advanced data types.
import collections
Exploring Python and getting help
- Official python documentation: https://docs.python.org/3. There is also a tutorial.
If you go through the tutorial, you will be pretty well on your way.
- Even after getting help elsewhere, it is a good idea to go back to the official documentation and fully understand what you are doing.
- use
help()
to bring up information about an object or a type help(dict), help([1,2,3]). type(obj)
gets the type of the objectdir(obj)
gets the methods associated with an object.dir()
gets everything defined in current namespacedir(__builtins__)
to see what builtins are available.
- To see the type of an object, use
type()
Libraries and Pip
There are many libraries available for python
. Many of these libraries are native python code
while others are thin wrappers around a C
or C++
library. Incompatibilities between versions
and weird installation directories can cause a large variety of problems. The advice I give below
should help you avoid many of them.
Pip
- Python has its own package manager, called
pip
. On Ubuntu you call it viapip3
. - Packages for
pip
are hosted on https://www.pypi.org. - Incompatibilities between different package versions from different sources often cause a mess.
- Many
python3
packages are already packaged for Ubuntu and are calledpython3-<packagename>
- These packages are referred to as system packages
- The current version of pip on Ubuntu is configured to disallow you from installing packages with pip unless they are installed into a virtual environment.
Virtual Environments
Virtual Environments enable you to create separate directories with different versions and combinations of python packages installed. Modern python development practices highly encourage the use of virtual environments for everything
To create a virtual environment:
- Run
python3 -m venv path/to/my/venv
(the path can be anything, but is ideally an empty or non-existent directory)- This command creates a virtual environment in a directory called
path/to/my/venv/
- If you need python 2 replace
venv
withvirtualenv
.
- This command creates a virtual environment in a directory called
- Activate the virtual environment using
source path/to/my/venv/bin/activate
- When the virtual environment is active,
pip
will install packages to that environment.- These packages will not affect the rest of the system, and will not be used outside the virtual environment.
- To leave the virtual environment, type
deactivate
- Note: pip packages that depend on external C/C++ libraries can get complicated. You will either have to follow special instructions for how to install them into the virtual environment or rely on system-wide installed packages
Python and ROS 2
- ROS 2 is designed around the use of
system-packages
, not virtual environments- This means that all (most? maybe there are exceptions, but I don't know of any) ROS 2 packages that distributed by ROS
(e.g., you can
apt install
them from ROS's package repository) depend on the system version of that packagethat have python dependencies - If your project is using ROS 2 then it is generally best to rely on system packages
- This means that all (most? maybe there are exceptions, but I don't know of any) ROS 2 packages that distributed by ROS
(e.g., you can
- If you wish to distribute a ROS 2 package in binary form from the ROS 2 package server and it depends on a pip package:
- You need get the python package accepted into Ubuntu.
- In your own development you can use packages installed via pip.
- The best way to do so is as follows:
- Obtain a list of the project's python dependencies (e.g., the
requirements.txt
) - For each dependency, check if it is available via
apt
(usingapt-search
) - If it is available, install that package via
apt
- After all available system dependencies are installed create a virtual environment and pass it the
--system-site-packages
flag- This will cause the virtual environment to use the system-installed packages
- Activate the virtual environment and install the package via pip
- Build ROS package as normal, with the virtual environment active
- If some of the system-packages don't work or are the wrong version, install the correct version into the virtual environment.
- Obtain a list of the project's python dependencies (e.g., the
Python
- Modern python development encourages the use of virtual environments
- Current ros2 packages depend on system-provided packages
- Modern Ubuntu highly discourages the use of
pip
to install python packages, unless in a virtual environment
Pytorch
- Pytorch is Meta's open-source machine learning (ML) library.
- It is most easily installed into a virtual environment. H
- However, sometimes we may w
- python3-astunparse
- python3-expecttest python3-hypothesis python3-filelock python3-networkx python3-fsspec
- Add the
pip
binary directory to your PATH by adding the following line to~/.bashrc
, just before thesource /opt/ros/iron/setup.bash
line:export PATH=$PATH:~/.local/bin
- Install pytorch:
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
. - Pytorch comes with its own version of CUDA, so you do not need to install CUDA separately to use pytorch.
Anaconda
Anaconda is a distribution of python primarily designed to make it easier to install.
- You have no reason to install anaconda; you already have python on your system and are able to use
pip
and virtual environments. - Do NOT, under ANY circumstances, install Anaconda.
- You do not need it and it can easily destroy your ROS installation by overriding python paths and packages and introducing incompatible versions of packages.
- Many professors tell students to install anaconda because this makes it easier to explain to a class where everyone has a different system how to setup python.
- It does not mean that anaconda is actually needed for the class!
- Not installing anaconda may require some extra effort on your part to install the proper python libraries: this is great practice because setting up python environments is an important skill.
- If you have installed anaconda and run into any python-related problem
- I will first tell you to uninstall anaconda (Usually this fixes the problem)
- If uninstalling does not work I will assume that Anaconda (or the steps taken to recover) irreparably messed up your python installation and advise you to reformat your computer.
- Do not install anaconda.
- In the past, students who have ignored this admonition have each spent hours debugging issues that are ultimately solved by uninstalling anaconda.
- If a project only supports installation via anaconda, that does not mean you need to use anaconda: you can install via other means with a little work
- This extra effort is worth it, so you do not mess up your ROS installation!
Important Third-Party Libraries
Here are some commonly used libraries. I typically install these through apt
since I like to have them
available whenever I run python3
casually from the command line and also want the version shipped with Ubuntu to be used for maximum ROS compatibility.
- Numpy - For math operations
- Matplotlib - For plotting
- Scipy - extra scientific programming tools, like numerical integration. Builds on numpy
- pandas - useful for manipulating data tables
- pyserial - for using the serial port
- sympy - symbolic algebra
- pygame - python gaming library. Not all fun and games though, useful for doing animated graphics
- PyQt - used for making guis in python using the QT framework
- Requests - used for making http requests. Python has built-in functionality for this but many find Requests is easier
- sphinx - generate documentation from python packages
Exceptions to using apt
Some scientific libraries (particularly machine learning libraries) are out-of-date in Ubuntu in a way that inhibits their functionality. Using these libraries also puts you behind the curve in a fast moving field. In this case, pip is often the solution.
pytorch
- A machine learning library from Meta. My preferred library because it has an easier to use API than tensorflow and is more widely used in academia (and catching up rapidly in industry).tensorflow
- Google's machine learning library.
Warm-up Exercises
- Write a "guess-the-number" game.
- User enters a number between
a
andb
- The computer tells them if it is too high, or too low
- If too many guesses are taken, then the game is over
- User enters a number between
- Write a program that converts between Roman Numerals and Arabic Numerals
- Implement the mathematical game Nim
- Using
sympy
andmatplotlib
create a program that lets a user enter and plot equations - Using
pygame
, write a simple two-player game (such as snake).- In the game of Snake, the player controls a snake that is \(N\) blocks long
- If the Snake hits itself or the walls, the player loses
- There is food in the arena (which is a single block). When the Snake hits the foot, it grows in length (e.g., by one block).
- The snake moves through the arena by removing the tile that is it's tail and placing that tile at the head, in the direction of travel.
Other Learning Resources
- Official python documentation
- Official Python Tutorial
- If you have never done object oriented programming, you should complete this tutorial.
- Python Crash Course 2nd Edition by Eric Matthes (free Online access through the Northwestern Library)
- Official Python Howtos
- Python Enhancement Proposals