MicroPython on Microcontrollers¶

MicroPython is designed to be capable of running on microcontrollers. These have hardware limitations which may be unfamiliar to programmers more familiar with conventional computers. In particular the amount of RAM and nonvolatile “disk” (flash memory) storage is limited. This tutorial offers ways to make the most of the limited resources. For instance, Satlink 3 and XLink 500 provide about 1MB of memory to Python programs, and the of 1MB of storage (pre-formatted).

RAM¶

When reducing RAM usage there are two phases to consider: compilation and execution. In addition to memory consumption, there is also an issue known as heap fragmentation. In general terms it is best to minimise the repeated creation and destruction of objects. The reason for this is covered in the section covering the heap.

Compilation Phase¶

When a module is imported, MicroPython compiles the code to bytecode which is then executed by the MicroPython virtual machine (VM). The bytecode is stored in RAM. The compiler itself requires RAM, but this becomes available for use when the compilation has completed.

If a number of modules have already been imported the situation can arise where there is insufficient RAM to run the compiler. In this case the import statement will produce a memory exception.

If a module instantiates global objects on import it will consume RAM at the time of import, which is then unavailable for the compiler to use on subsequent imports. In general it is best to avoid code which runs on import; a better approach is to have initialisation code which is run by the application after all modules have been imported. This maximises the RAM available to the compiler.

Execution Phase¶

There are a number of coding techniques for reducing RAM usage.

Constants

MicroPython provides a const keyword which may be used as follows:

from micropython import const
ROWS = const(33)
_COLS = const(0x10)
a = ROWS
b = _COLS

In both instances where the constant is assigned to a variable the compiler will avoid coding a lookup to the name of the constant by substituting its literal value. This saves bytecode and hence RAM. However the ROWS value will occupy at least two machine words, one each for the key and value in the globals dictionary. The presence in the dictionary is necessary because another module might import or use it. This RAM can be saved by prepending the name with an underscore as in _COLS: this symbol is not visible outside the module so will not occupy RAM.

The argument to const() may be anything which, at compile time, evaluates to an integer e.g. 0x100 or 1 << 8. It can even include other const symbols that have already been defined, e.g. 1 << BIT.

Needless object creation

There are a number of situations where objects may unwittingly be created and destroyed. This can reduce the usability of RAM through fragmentation. The following sections discuss instances of this.

String concatenation

Consider the following code fragments which aim to produce constant strings:

var = "foo" + "bar"
var1 = "foo" "bar"
var2 = """\
foo\
bar"""

Each produces the same outcome, however the first needlessly creates two string objects at runtime, allocates more RAM for concatenation before producing the third. The others perform the concatenation at compile time which is more efficient, reducing fragmentation.

Where strings must be dynamically created before being fed to a stream such as a file it will save RAM if this is done in a piecemeal fashion. Rather than creating a large string object, create a substring and feed it to the stream before dealing with the next.

The best way to create dynamic strings is by means of the string format method:

var = "Temperature {:5.2f} Pressure {:06d}\n".format(temp, press)

Buffers

When accessing devices such as instances of Serial, using pre-allocated buffers avoids the creation of needless objects. Consider these two loops:

while True:
    var = serial.read(100)
    # process data

buf = bytearray(100)
while True:
    serial.readinto(buf)
    # process data in buf

The first creates a buffer on each pass whereas the second re-uses a pre-allocated buffer; this is both faster and more efficient in terms of memory fragmentation.

Bytes are smaller than ints

On most platforms an integer consumes four bytes. Consider the two calls to the function foo():

def foo(bar):
    for x in bar:
        print(x)
foo((1, 2, 0xff))
foo(b'\1\2\xff')

In the first call a tuple of integers is created in RAM. The second efficiently creates a bytes object consuming the minimum amount of RAM. If the module were frozen as bytecode, the bytes object would reside in flash.

Strings Versus Bytes

Python3 introduced Unicode support. This introduced a distinction between a string and an array of bytes. MicroPython ensures that Unicode strings take no additional space so long as all characters in the string are ASCII (i.e. have a value < 126). If values in the full 8-bit range are required bytes and bytearray objects can be used to ensure that no additional space will be required. Note that most string methods (e.g. strip()) apply also to bytes instances so the process of eliminating Unicode can be painless.

s = 'the quick brown fox'  # A string instance
b = b'the quick brown fox'  # a bytes instance

Where it is necessary to convert between strings and bytes the string encode and the bytes decode methods can be used. Note that both strings and bytes are immutable. Any operation which takes as input such an object and produces another implies at least one RAM allocation to produce the result. In the second line below a new bytes object is allocated. This would also occur if foo were a string.

foo = b'   empty whitespace'
foo = foo.lstrip()

Runtime compiler execution

The Python keywords eval and exec invoke the compiler at runtime, which requires significant amounts of RAM. Note that the pickle library employs exec. It may be more RAM efficient to use the json library for object serialisation.

The Heap¶

When a running program instantiates an object the necessary RAM is allocated from a fixed size pool known as the heap. When the object goes out of scope (in other words becomes inaccessible to code) the redundant object is known as “garbage”. A process known as “garbage collection” (GC) reclaims that memory, returning it to the free heap. This process runs automatically, however it can be invoked directly by issuing gc.collect().

The discourse on this is somewhat involved. For a ‘quick fix’ issue the following periodically:

gc.collect()
gc.threshold(gc.mem_free() // 4 + gc.mem_alloc())

Fragmentation¶

Say a program creates an object foo, then an object bar. Subsequently foo goes out of scope but bar remains. The RAM used by foo will be reclaimed by GC. However if bar was allocated to a higher address, the RAM reclaimed from foo will only be of use for objects no bigger than foo. In a complex or long running program the heap can become fragmented: despite there being a substantial amount of RAM available, there is insufficient contiguous space to allocate a particular object, and the program fails with a memory error.

The techniques outlined above aim to minimise this. Where large permanent buffers or other objects are required it is best to instantiate these early in the process of program execution before fragmentation can occur. Further improvements may be made by monitoring the state of the heap and by controlling GC; these are outlined below.

Reporting¶

A number of library functions are available to report on memory allocation and to control GC. These are to be found in the gc and micropython modules. The following example may be pasted at the REPL (ctrl e to enter paste mode, ctrl d to run it).

import gc
import micropython
gc.collect()
micropython.mem_info()
print('-----------------------------')
print('Initial free: {} allocated: {}'.format(gc.mem_free(), gc.mem_alloc()))
def func():
    a = bytearray(10000)
gc.collect()
print('Func definition: {} allocated: {}'.format(gc.mem_free(), gc.mem_alloc()))
func()
print('Func run free: {} allocated: {}'.format(gc.mem_free(), gc.mem_alloc()))
gc.collect()
print('Garbage collect free: {} allocated: {}'.format(gc.mem_free(), gc.mem_alloc()))
print('-----------------------------')
micropython.mem_info(1)

Methods employed above:

gc.collect() Force a garbage collection. See footnote.
micropython.mem_info() Print a summary of RAM utilisation.
gc.mem_free() Return the free heap size in bytes.
gc.mem_alloc() Return the number of bytes currently allocated.
micropython.mem_info(1) Print a table of heap utilisation (detailed below).

The numbers produced are dependent on the platform, but it can be seen that declaring the function uses a small amount of RAM in the form of bytecode emitted by the compiler (the RAM used by the compiler has been reclaimed). Running the function uses over 10KiB, but on return a is garbage because it is out of scope and cannot be referenced. The final gc.collect() recovers that memory.

The final output produced by micropython.mem_info(1) will vary in detail but may be interpreted as follows:

Symbol	Meaning
.	free block
h	head block
=	tail block
m	marked head block
T	tuple
L	list
D	dict
F	float
B	byte code
M	module

Each letter represents a single block of memory, a block being 16 bytes. So each line of the heap dump represents 0x400 bytes or 1KiB of RAM.

Control of Garbage Collection¶

A GC can be demanded at any time by issuing gc.collect(). It is advantageous to do this at intervals, firstly to pre-empt fragmentation and secondly for performance. A GC can take several milliseconds but is quicker when there is little work to do (about 1ms on the Pyboard). An explicit call can minimise that delay while ensuring it occurs at points in the program when it is acceptable.

Automatic GC is provoked under the following circumstances. When an attempt at allocation fails, a GC is performed and the allocation re-tried. Only if this fails is an exception raised. Secondly an automatic GC will be triggered if the amount of free RAM falls below a threshold. This threshold can be adapted as execution progresses:

gc.collect()
gc.threshold(gc.mem_free() // 4 + gc.mem_alloc())

This will provoke a GC when more than 25% of the currently free heap becomes occupied.

In general modules should instantiate data objects at runtime using constructors or other initialisation functions. The reason is that if this occurs on initialisation the compiler may be starved of RAM when subsequent modules are imported. If modules do instantiate data on import then gc.collect() issued after the import will ameliorate the problem.

String Operations¶

MicroPython handles strings in an efficient manner and understanding this can help in designing applications to run on microcontrollers. When a module is compiled, strings which occur multiple times are stored once only, a process known as string interning. In MicroPython an interned string is known as a qstr. In a module imported normally that single instance will be located in RAM, but as described above, in modules frozen as bytecode it will be located in flash.

String comparisons are also performed efficiently using hashing rather than character by character. The penalty for using strings rather than integers may hence be small both in terms of performance and RAM usage - a fact which may come as a surprise to C programmers.

Postscript¶

MicroPython passes, returns and (by default) copies objects by reference. A reference occupies a single machine word so these processes are efficient in RAM usage and speed.

Where variables are required whose size is neither a byte nor a machine word there are standard libraries which can assist in storing these efficiently and in performing conversions. See the array, ustruct and uctypes modules.

Footnote: gc.collect() return value¶

On Unix and Windows platforms the gc.collect() method returns an integer which signifies the number of distinct memory regions that were reclaimed in the collection (more precisely, the number of heads that were turned into frees). For efficiency reasons bare metal ports do not return this value.