Primer on Python for R Users

Whitespace

Whitespace matters in Python. In R, expressions are grouped into a code block with {}. In Python, that is done by making the expressions share an indentation level. For example, an expression with an R code block might be:

if (TRUE) {
  cat("This is one expression. \n")
  cat("This is another expression. \n")
}
#> This is one expression.
#> This is another expression.

The equivalent in Python:

if True:
  print("This is one expression.")
  print("This is another expression.")
#> This is one expression.
#> This is another expression.

Python accepts tabs or spaces as the indentation spacer, but the rules get tricky when they’re mixed. Most style guides suggest (and IDE’s default to) using spaces only.

Container Types

In R, the list() is a container you can use to organize R objects. R’s list() is feature packed, and there is no single direct equivalent in Python that supports all the same features. Instead there are (at least) 4 different Python container types you need to be aware of: lists, dictionaries, tuples, and sets.

x = [1, 2, 3]
y = x    # `y` and `x` now refer to the same list!
x.append(4)
print("x is", x)
#> x is [1, 2, 3, 4]
print("y is", y)
#> y is [1, 2, 3, 4]

x = [1]
x
#> [1]
x + x
#> [1, 1]
x * 3
#> [1, 1, 1]

x = [1, 2, 3]
x[0]
#> 1
x[1]
#> 2
x[2]
#> 3

try:
  x[3]
except Exception as e:
  print(e)
#> list index out of range

x = [1, 2, 3]
x[-1]
#> 3
x[-2]
#> 2
x[-3]
#> 1

x = [1, 2, 3, 4, 5, 6]
x[0:2] # get items at index positions 0, 1
#> [1, 2]
x[1:]  # get items from index position 1 to the end
#> [2, 3, 4, 5, 6]
x[:-2] # get items from beginning up to the 2nd to last.
#> [1, 2, 3, 4]
x[:]   # get all the items (idiom used to copy the list so as not to modify in place)
#> [1, 2, 3, 4, 5, 6]
x[::2] # get all the items, with a stride of 2
#> [1, 3, 5]
x[1::2] # get all the items from index 1 to the end, with a stride of 2
#> [2, 4, 6]

x = (1, 2) # tuple of length 2
type(x)
#> <class 'tuple'>
len(x)
#> 2
x
#> (1, 2)

x = (1,) # tuple of length 1
type(x)
#> <class 'tuple'>
len(x)
#> 1
x
#> (1,)

x = () # tuple of length 0
print(f"{type(x) = }; {len(x) = }; {x = }")
#> type(x) = <class 'tuple'>; len(x) = 0; x = ()
# example of an interpolated string literals

x = 1, 2 # also a tuple
type(x)
#> <class 'tuple'>
len(x)
#> 2

x = 1, # beware a single trailing comma! This is a tuple!
type(x)
#> <class 'tuple'>
len(x)
#> 1

x = (1, 2, 3)
a, b, c = x
a
#> 1
b
#> 2
c
#> 3

xx = (("a", 1),
      ("b", 2))
for x1, x2 in xx:
  print("x1 = ", x1)
  print("x2 = ", x2)
#> x1 =  a
#> x2 =  1
#> x1 =  b
#> x2 =  2

x = (1, 2, 3)
a, b, c = x # success
a, b = x    # error, x has too many values to unpack
#> ValueError: too many values to unpack (expected 2)
a, b, c, d = x # error, x has not enough values to unpack
#> ValueError: not enough values to unpack (expected 4, got 3)

x = (1, 2, 3)
a, *the_rest = x
a
#> 1
the_rest
#> [2, 3]

x = ((1, 2), (3, 4))
(a, b), (c, d) = x

d = {"key1": 1,
     "key2": 2}
d2 = d
d
#> {'key1': 1, 'key2': 2}
d["key1"]
#> 1
d["key3"] = 3
d2 # modified in place!
#> {'key1': 1, 'key2': 2, 'key3': 3}

d = {"key1": 1, "key2": 2}
d[1] # error
#> KeyError: 1

s = {1, 2, 3}
type(s)
#> <class 'set'>
s
#> {1, 2, 3}

s.add(1)
s
#> {1, 2, 3}

Iteration with for

The for statement in Python can be used to iterate over any kind of container.

for x in [1, 2, 3]:
  print(x)
#> 1
#> 2
#> 3

R has a relatively limited set of objects that can be passed to for. Python by comparison, provides an iterator protocol interface, which means that authors can define custom objects, with custom behavior that is invoked by for. (We’ll have an example for how to define a custom iterable when we get to classes). You may want to use a Python iterable from R using reticulate, so it’s helpful to peel back the syntactic sugar a little to show what the for statement is doing in Python, and how you can step through it manually.

There are two things that happen: first, an iterator is constructed from the supplied object. Then, the new iterator object is repeatedly called with next() until it is exhausted.

l = [1, 2, 3]
it = iter(l) # create an iterator object
it
#> <list_iterator object at 0x1402267a0>

# call `next` on the iterator until it is exhausted:
next(it)
#> 1
next(it)
#> 2
next(it)
#> 3
next(it)
#> StopIteration

In R, you can use reticulate to step through an iterator the same way.

library(reticulate)
l <- r_to_py(list(1, 2, 3))
it <- as_iterator(l)

iter_next(it)
#> 1.0
iter_next(it)
#> 2.0
iter_next(it)
#> 3.0
iter_next(it, completed = "StopIteration")
#> [1] "StopIteration"

Iterating over dictionaries first requires understanding if you are iterating over the keys, values, or both. Dictionaries have methods that allow you to specify which.

d = {"key1": 1, "key2": 2}
for key in d:
  print(key)
#> key1
#> key2

for value in d.values():
  print(value)
#> 1
#> 2

for key, value in d.items():
  print(key, ":", value)
#> key1 : 1
#> key2 : 2

x = [1, 2, 3]

# a list comprehension built from x, where you add 100 to each element
l = [element + 100 for element in x]
l
#> [101, 102, 103]

# a dict comprehension built from x, where the key is a string.
# Python's str() is like R's as.character()
d = {str(element) : element + 100
     for element in x}
d
#> {'1': 101, '2': 102, '3': 103}

Defining Functions with def

Python functions are defined with the def statement. The syntax for specifying function arguments and default values is very similar to R.

def my_function(name = "World"):
  print("Hello", name)

my_function()
#> Hello World
my_function("Friend")
#> Hello Friend

The equivalent R snippet would be

my_function <- function(name = "World") {
  cat("Hello", name, "\n")
}

my_function()
#> Hello World
my_function("Friend")
#> Hello Friend

Unlike R functions, the last value in a function is not automatically returned. Python requires an explicit return statement.

def fn():
  1
print(fn())
#> None

def fn():
  return 1
print(fn())
#> 1

(Note for advanced R users: Python has no equivalent of R’s argument “promises”. Function argument default values are evaluated once, when the function is constructed. This can be surprising if you define a Python function with a mutable object as a default argument value, like a Python list!)

def my_func(x = []):
  x.append("was called")
  print(x)

my_func()
#> ['was called']
my_func()
#> ['was called', 'was called']
my_func()
#> ['was called', 'was called', 'was called']

You can also define Python functions that take a variable number of arguments, similar to ... in R. A notable difference is that R’s ... makes no distinction between named and unnamed arguments, but Python does. In Python, prefixing a single * captures unnamed arguments, and two ** signifies that keyword arguments are captured.

def my_func(*args, **kwargs):
  print("args = ", args) # args is a tuple
  print("kwargs = ", kwargs) # kwargs is a dictionary

my_func(1, 2, 3, a = 4, b = 5, c = 6)
#> args =  (1, 2, 3)
#> kwargs =  {'a': 4, 'b': 5, 'c': 6}

Whereas the * and ** in a function definition signature pack arguments, in a function call they unpack arguments. Unpacking arguments in a function call is equivalent to using do.call() in R.

def my_func(a, b, c):
  print(a, b, c)

args = (1, 2, 3)
my_func(*args)
#> 1 2 3

kwargs = {"a": 1, "b": 2, "c": 3}
my_func(**kwargs)
#> 1 2 3

Defining Classes with class

One could argue that in R, the preeminent unit of composition for code is the function, and in Python, it’s the class. You can be a very productive R user and never use R6, reference classes, or similar R equivalents to the object-oriented style of Python class’s.

In Python, however, understanding the basics of how class objects work is requisite knowledge, because class’s are how you organize and find methods in Python. (In contrast to R’s approach, where methods are found by dispatching from a generic). Fortunately, the basics of class’s are accessible.

Don’t be intimidated if this is your first exposure to object oriented programming. We’ll start by building up a simple Python class for demonstration purposes.

class MyClass:
  pass # `pass` means do nothing.

MyClass
#> <class '__main__.MyClass'>
type(MyClass)
#> <class 'type'>

instance = MyClass()
instance
#> <__main__.MyClass object at 0x14023b260>
type(instance)
#> <class '__main__.MyClass'>

Like the def statement, the class statement binds a new callable symbol, MyClass. First note the strong naming convention, classes are typically CamelCase, and functions are typically snake_case. After defining MyClass, you can interact with it, and see that it has type 'type'. Calling MyClass() creates a new object instance of the class, which has type 'MyClass' (ignore the __main__. prefix for now). The instance prints with its memory address, which is a strong hint that it’s common to be managing many instances of a class, and that the instance is mutable (modified-in-place by default).

In the first example, we defined an empty class, but when we inspect it we see that it already comes with a bunch of attributes (dir() in Python is equivalent to names() in R):

dir(MyClass)
#> ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__']

class MyClass:

  print("MyClass's definition body is being evaluated")

  def __init__(self):
    print(self, "is initializing")
#> MyClass's definition body is being evaluated

print("MyClass is finished being created")
#> MyClass is finished being created

instance = MyClass()
#> <__main__.MyClass object at 0x140266330> is initializing
print(instance)
#> <__main__.MyClass object at 0x140266330>

instance2 = MyClass()
#> <__main__.MyClass object at 0x11e3ad490> is initializing
print(instance2)
#> <__main__.MyClass object at 0x11e3ad490>

class MyClass:
  def a_method(self):
    print("MyClass.a_method() was called with", self)

instance = MyClass()
instance.a_method()
#> MyClass.a_method() was called with <__main__.MyClass object at 0x11e3c7f20>

MyClass.a_method()     # error, missing required argument `self`
#> TypeError: MyClass.a_method() missing 1 required positional argument: 'self'
MyClass.a_method(instance) # identical to instance.a_method()
#> MyClass.a_method() was called with <__main__.MyClass object at 0x11e3c7f20>

class MyRange:
  def __init__(self, start, end):
    self.start = start
    self.end = end

  def __iter__(self):
    # reset our counter.
    self._index = self.start - 1
    return self

  def __next__(self):
    if self._index < self.end:
      self._index += 1 # increment
      return self._index
    else:
      raise StopIteration

for x in MyRange(1, 3):
  print(x)
#> 1
#> 2
#> 3

# doing what `for` does, but manually
r = MyRange(1, 3)
it = iter(r)
next(it)
#> 1
next(it)
#> 2
next(it)
#> 3
next(it)
#> StopIteration

Defining Generators with yield.

Generators are special Python functions that contain one or more yield statements. As soon as yield is included in a code block passed to def, the semantics change substantially. You’re no longer defining a mere function, but a generator constructor! In turn, calling a generator constructor creates a generator object, which is just another type of iterator.

Here is an example:

def my_generator_constructor():
  yield 1
  yield 2
  yield 3

# At first glance it presents like a regular function
my_generator_constructor
#> <function my_generator_constructor at 0x1402579c0>
type(my_generator_constructor)
#> <class 'function'>

# But calling it returns something special, a 'generator object'
my_generator = my_generator_constructor()
my_generator
#> <generator object my_generator_constructor at 0x11e3ff530>
type(my_generator)
#> <class 'generator'>

# The generator object is both an iterable and an iterator
# it's __iter__ method is just a stub that returns `self`
iter(my_generator) == my_generator == my_generator.__iter__()
#> True

# step through it like any other iterator
next(my_generator)
#> 1
my_generator.__next__() # next() is just sugar for calling the dunder
#> 2
next(my_generator)
#> 3
next(my_generator)
#> StopIteration

Encountering yield is like hitting the pause button on a functions execution, it preserves the state of everything in the function body and returns control to whatever is iterating over the generator object. Calling next() on the generator object resumes execution of the function body until the next yield is encountered, or the function finishes.

Iteration closing remarks

Iteration is deeply baked into the Python language, and R users may be surprised by how things in Python are iterable, iterators, or powered by the iterator protocol under the hood. For example, the built-in map() (equivalent to R’s lapply()) yields an iterator, not a list. Similarly, a tuple comprehension like (elem for elem in x) produces an iterator. Most features dealing with files are iterators, and so on.

Any time you find an iterator inconvenient, you can materialize all the elements into a list using the Python built-in list(), or reticulate::iterate() in R. Also, if you like the readability of for, you can utilize similar semantics to Python’s for using coro::loop().

import and Modules

In R, authors can bundle their code into shareable extensions called R packages, and R users can access objects from R packages via library() or ::. In Python, authors bundle code into modules, and users access modules using import. Consider the line:

import numpy

This statement has Python go out to the file system, find an installed Python module named ‘numpy’, load it (commonly meaning: evaluate its __init__.py file and construct a module type), and bind it to the symbol numpy.

The closest equivalent to this in R might be:

dplyr <- loadNamespace("dplyr")

import sys
sys.path
#> ['', '/Users/tomasz/.pyenv/versions/3.12.4/bin', '/Users/tomasz/.pyenv/versions/3.12.4/lib/python312.zip', '/Users/tomasz/.pyenv/versions/3.12.4/lib/python3.12', '/Users/tomasz/.pyenv/versions/3.12.4/lib/python3.12/lib-dynload', '/Users/tomasz/.virtualenvs/r-reticulate/lib/python3.12/site-packages', '/Users/tomasz/github/rstudio/reticulate/inst/python', '/Users/tomasz/.virtualenvs/r-reticulate/lib/python312.zip', '/Users/tomasz/.virtualenvs/r-reticulate/lib/python3.12', '/Users/tomasz/.virtualenvs/r-reticulate/lib/python3.12/lib-dynload']

import os
os.__file__
#> '/Users/tomasz/.virtualenvs/r-reticulate/lib/python3.12/os.py'
numpy.__path__
#> ['/Users/tomasz/.virtualenvs/r-reticulate/lib/python3.12/site-packages/numpy']

numpy.abs(-1)
#> 1

import numpy        # import
import numpy as np  # import and bind to a custom symbol `np`
np is numpy         # test for identicalness, similar to identical(np, numpy)
#> True

from numpy import abs # import only `numpy.abs`, bind it to `abs`
abs is numpy.abs
#> True

from numpy import abs as abs2 # import only `numpy.abs`, bind it to `abs2`
abs2 is numpy.abs
#> True

from numpy import *

Integers and Floats

R users generally don’t need to be aware of the difference between integers and floating point numbers, but that’s not the case in Python. If this is your first exposure to numeric data types, here are the essentials:

• integer types can only represent whole numbers like 1 or 2, not floating point numbers like 1.2.

• floating-point types can represent any number, but with some degree of imprecision.

In R, writing a bare literal number like 12 produces a floating point type, whereas in Python, it produces an integer. You can produce an integer literal in R by appending an L, as in 12L. Many Python functions expect integers, and will error when provided a float.

For example, say we have a Python function that expects an integer:

def a_strict_Python_function(x):
  assert isinstance(x, int), "x is not an int"
  print("Yay! x was an int")

When calling it from R, you must be sure to call it with an integer:

library(reticulate)
py$a_strict_Python_function(3)             # error
#> x is not an int
py$a_strict_Python_function(3L)            # success
#> Yay! x was an int
py$a_strict_Python_function(as.integer(3)) # success
#> Yay! x was an int

What about R vectors?

R is a language designed for numerical computing first. Numeric vector data types are baked deep into the R language, to the point that the language doesn’t even distinguish scalars from vectors. By comparison, numerical computing capabilities in Python are generally provided by third party packages (modules, in Python parlance).

In Python, the numpy module is most commonly used to handle contiguous arrays of data. The closest equivalent to an R numeric vector is a numpy array, or sometimes, a list of scalar numbers (some Pythonistas might argue for array.array() here, but that’s so rarely encountered in actual Python code we don’t mention it further).

Teaching the NumPy interface is beyond the scope of this primer, but it’s worth pointing out some potential tripping hazards for users accustomed to R arrays:

• When indexing into multidimensional numpy arrays, trailing dimensions can be omitted and are implicitly treated as missing. The consequence is that iterating over arrays means iterating over the first dimension. For example, this iterates over the rows of a matrix.

import numpy as np
m = np.arange(12).reshape((3,4))
m
#> array([[ 0,  1,  2,  3],
#>        [ 4,  5,  6,  7],
#>        [ 8,  9, 10, 11]])
m[0, :] # first row
#> array([0, 1, 2, 3])
m[0]    # also first row
#> array([0, 1, 2, 3])

for row in m:
  print(row)
#> [0 1 2 3]
#> [4 5 6 7]
#> [ 8  9 10 11]

• Many numpy operations modify the array in place! This is surprising to R users, who are used to the convenience and safety of R’s copy-on-modify semantics. Unfortunately, there is no simple scheme or naming convention you can rely on to quickly determine if a particular method modifies in-place or creates a new array copy. The only reliable way is to consult the documentation, and conduct small experiments at the reticulate::repl_python().

Decorators

Decorators are just functions that take a function as an argument, and then typically returns another function. Any function can be invoked as a decorator with the @ syntax, which is just sugar for this simple action:

def my_decorator(func):
  func.x = "a decorator modified this function by adding an attribute `x`"
  return func

def my_function(): pass
my_function = my_decorator(my_function)

# @ is just fancy syntax for the above two lines
@my_decorator
def my_function(): pass

One decorator you might encounter frequently is:

• @property, which automatically calls a class method when the attribute is accessed (similar to makeActiveBinding() in R).

with and context management

Any object that defines __enter__ and __exit__ methods implements the “context” protocol, and can be passed to with. For example, here is a custom implementation of a context manager that temporarily changes the current working directory (equivalent to R’s withr::with_dir())

from os import getcwd, chdir

class wd_context:
  def __init__(self, wd):
    self.new_wd = wd

  def __enter__(self):
    self.original_wd = getcwd()
    chdir(self.new_wd)

  def __exit__(self, *args):
    # __exit__ takes some additional argument that are commonly ignored
    chdir(self.original_wd)


getcwd()
#> '/Users/tomasz/github/rstudio/reticulate/vignettes'
with wd_context(".."):
  print("in the context, wd is:", getcwd())
#> in the context, wd is: /Users/tomasz/github/rstudio/reticulate
getcwd()
#> '/Users/tomasz/github/rstudio/reticulate/vignettes'

Learning More

Hopefully, this short primer to Python has provided a good foundation for confidently reading Python documentation and code, and using Python modules from R via reticulate. Of course, there is much, much more to learn about Python. Googling questions about Python reliably brings up pages of results, but not always sorted in order of most useful. Blog posts and tutorials targeting beginners can be valuable, but remember that Python’s official documentation is generally excellent, and it should be your first destination when you have questions.

https://docs.Python.org/3/

https://docs.Python.org/3/library/index.html

To learn Python more fully, the built-in official tutorial is also excellent and comprehensive (but does require a time commitment to get value out of it) https://docs.Python.org/3/tutorial/index.html

Finally, don’t forget to solidify your understanding by conducting small experiments at the reticulate::repl_python().

Thank you for reading!