User Guide¶

Overview¶

chumicro-msgpack serializes Python objects to compact binary bytes using the MessagePack format and deserializes them back. It covers the subset of msgpack needed on microcontrollers: integers up to 32-bit, 32-bit floats, strings, bytes, booleans, None, lists, tuples, and dicts.

The library exposes four functions: packb and unpackb for bytes-based encoding/decoding, and pack and unpack for stream-based I/O. On CircuitPython boards with the native C msgpack module, all four delegate to the built-in; the pure-Python encoder is never loaded.

Getting started¶

from chumicro_msgpack import packb, unpackb

data = packb({"ssid": "MyNetwork", "configured": True})
print(data)          # compact binary bytes

restored = unpackb(data)
print(restored)      # {'ssid': 'MyNetwork', 'configured': True}

When to use msgpack vs struct¶

Python's struct module and msgpack both produce compact binary data, but they solve different problems:

Use struct when the data layout is fixed and both sides are compiled together (e.g., a sensor reading struct that never changes). Use msgpack when the data is dict-like, may evolve over time, or when you want to inspect the data without knowing the schema.

Stream-based API (preferred on microcontrollers)¶

pack and unpack write to and read from stream objects (anything with .write() or .read()). This matches CircuitPython's native msgpack API.

When writing to a file, socket, or NVM wrapper, pack(obj, stream) reads cleaner than packb + write. On CircuitPython's native encoder it also writes directly to the destination without an intermediate bytes object; on the pure-Python path (MicroPython, CPython) pack encodes to bytes first and then writes, so it is not a RAM saving there.

from io import BytesIO
from chumicro_msgpack import pack, unpack

buffer = BytesIO()
pack({"key": [1, 2, 3]}, buffer)

buffer.seek(0)
result = unpack(buffer)
print(result)  # {'key': [1, 2, 3]}

Bytes-based API¶

packb and unpackb work with bytes objects directly. They are convenient when you need the encoded data in memory, for example to measure its length before writing it with a framing header, or to pass it to an API that expects bytes.

On microcontrollers, be aware that packb allocates a temporary bytearray, grows it during encoding, then copies it to bytes. For small payloads (typical settings dicts) this is fine. On the pure-Python path pack calls packb internally, so it does not reduce this allocation; only the native CircuitPython encoder streams without the intermediate buffer.

from chumicro_msgpack import packb, unpackb

# Encode any supported Python object.
packed = packb([1, "hello", None, True])

# Decode from bytes, bytearray, or memoryview.
original = unpackb(packed)
print(original)  # [1, 'hello', None, True]

unpackb accepts bytes, bytearray, and memoryview, so you can decode directly from a pre-allocated buffer without copying.

Decoding corrupt or untrusted input¶

unpackb is a trusting decoder, not a spec validator. It is safe against malformed framing: truncated, over-length, or trailing-garbage input, and nesting deeper than 8 levels, all raise ValueError instead of returning a silently-wrong value. packb enforces the same depth bound on encode, so anything it accepts round-trips. This matters for flash-backed config and kvstore, where a power-loss-truncated payload must fail loudly rather than decode as a structurally-valid wrong dict.

What it does not do is check that a structurally-valid payload has the type shape you expect: a packed {"port": "80"} decodes fine when your code wants {"port": 80}. Code persisting corruption- or attacker-reachable bytes still owns type-shape validation of what comes back.

Two boundary notes:

• The hardening lives in the pure-Python decoder, which is what runs on CPython, MicroPython, and CircuitPython boards without the native msgpack module. On CircuitPython boards that ship the native C msgpack, decoding goes through the firmware's decoder, whose framing behavior is its own.
• Truncation that lands on a fixed-width primitive or a length prefix raises the same framing message: the pure decoder catches the short struct read and re-raises it as ValueError("malformed msgpack: truncated or over-length framing"), so the failure stays loud and never returns a silent short read.

Integer keys for compact storage¶

When storing settings in NVM or sleep memory, use integer keys instead of strings. Integer keys encode in 1 byte (vs. multiple bytes for quoted strings), saving space on every entry:

from chumicro_msgpack import packb, unpackb
import json

settings = {0: "MyNetwork", 1: "secret123", 2: "lamp", 3: True}

msgpack_size = len(packb(settings))
json_size = len(json.dumps(settings))

print(f"msgpack: {msgpack_size} bytes")
print(f"JSON:    {json_size} bytes")
# msgpack is significantly smaller

Supported types¶

Unsupported types raise TypeError. Integers outside the 32-bit range raise OverflowError.

Memory notes¶

unpackb accepts bytes, bytearray, and memoryview, so you can decode directly from a pre-allocated buffer without copying. Internally the decoder treats the input as a memoryview end-to-end: slices stay as views; only the final bytes / str results are heap allocations.

packb builds a bytearray that grows as encoding proceeds, then copies to bytes once at the end. MicroPython extends a bytearray with exact-fit reallocations (an append adds 8 bytes of slack), so the realloc count scales with the number of tokens written rather than a fixed geometric series.

pack and unpack (stream-based) match the signatures CircuitPython's native C msgpack module exposes and delegate to it on hardware that ships it. On every other runtime (MicroPython, CPython, CircuitPython unix-port), pack calls packb and writes the result to the stream in one shot; it does not stream incrementally. If you need true streaming on those runtimes, write your own loop around packb per element.

Allocation profile (MicroPython 1.26 unix-port, GC disabled)¶

Useful when budgeting heap for a tick. Measured by wrapping each call in gc.collect(); gc.disable(); base = gc.mem_alloc() and reading the delta after 100 iterations.

The numbers shift across firmware versions and payload shapes; they're a sanity baseline, not a contract.

Platform notes¶

The wire format is identical regardless of which implementation is used: data packed on one runtime can be unpacked on any other.

Wire-format compatibility with PyPI msgpack¶

chumicro_msgpack.packb(obj) produces bytes byte-for-byte identical to msgpack.packb(obj, use_single_float=True) for any subset-conforming input. This lets a host-side tool encode with PyPI msgpack while the device decodes with chumicro_msgpack: same wire format, no conversion step.

The subset is:

• Integers in [-2**31, 2**32 - 1].
• Floats encoded as 32-bit (the use_single_float=True flag on the PyPI side).
• Strings, bytes, arrays, and maps with sizes under 65_536.
• No ext types.

The decoder names the offending tag in its ValueError if a stricter encoder produces something outside the subset, for example "float64 (0xcb) not in chumicro msgpack subset; encode with msgpack.packb(obj, use_single_float=True)". Two sharp edges this avoids:

1. PyPI's default float encoding is float64. Without use_single_float=True, msgpack.packb(0.5) emits 0xcb + 8 bytes, which chumicro_msgpack.unpackb rejects.
2. Native delegation is gated to CircuitPython only. On CPython, import msgpack would resolve to the PyPI package (with a different contract: float64, int64, strict_map_key), so chumicro_msgpack only delegates to the native C module when sys.implementation.name == "circuitpython". Other runtimes use the pure-Python encoder bundled in _pure.py.

Examples¶

The examples directory contains complete runnable scripts:

packb_basic.py, packb_size_comparison.py, and stream_roundtrip.py run on CPython, MicroPython, and CircuitPython. circuitpython_nvm_settings.py needs a real CircuitPython board with NVM; save it as code.py on the CIRCUITPY drive.