Train of Thought

Under the hood of your own language – How I customised the programming language Lox for a microcontroller, and you could too!

I’m currently exploring languages on microcontrollers. I’m curious why so many appear to be ‘cut down’ versions of other languages. I believe there is now enough compute in a microcontroller to support an ecosystem of dedicated languages. What would they look like if they targeted microcontrollers as their first use case, and other uses were later add ons?

My long term goal is a language I can prototype in on device; then later extend with specific hardware drivers as I finalise a design and finally ship the evolved code as-is in a production firmware image.

Languages like MicroPython are good for the first goal – prototyping, but I have to dip in to C in various ways to complete the rest. C, of course, is hard work to prototype in! I’d like to be able to use the same language as my project evolves, and never throw away stuff that already works.

What would a starter project for a dedicated language look like? Does it have to be a trivial toy, or could it start from a fully-formed language that’s intended to grow as I explore?

With this in mind, I encountered the book ‘Crafting Interpreters’ a few months ago, and it builds two practical implementations of a language ‘Lox’, each demonstrating features like scanning, parsing, types, garbage collection, a bytecode VM and a REPL to code in. Lox is usably complete, openly available, and sophisticated enough to offer practical utility. It also has plenty of scope for development. Since the implementation technologies involved seemed similar to my understanding of MicroPython’s implementation, it seemed reasonable to imagine it could work well on a microcontroller like the Raspberry Pi Pico.

First signs of life

After compiling a “hello world” C program via a VSCode plugin for Pico development, adding the sources from crafting interpreter’s ‘clox’ C implementation was straightforward. I had to reduce the size of the default stack (294K of RAM on a Pico is a lot, but not much compared to a typical PC) and then the language worked. I could access the REPL over a serial connection and run the tests and samples. I could write my own code ‘on device’.

The author notes that Lox doesn’t provide much practical interaction with the world around it on the system it runs on. print, implemented with C’s printf, is about it. My goal is to add ‘just enough’ to implement the Pico equivalent of hello world – turning on the builtin LED that is wired to GPIO 25¹. I believe this will provide enough infrastructure to take things further in future work.

I want to be able to write something like this Lox:

var PICO_LED = 25;
var GPIO_OUT = true;

gpio_init(PICO_LED);
gpio_set_direction(PICO_LED, GPIO_OUT);

gpio_put(PICO_LED, true);

And see the green LED come on when I run it.

I’m using the C implementation of Lox (there is also a Java one in the book, and many more that others have created). Lox’s author notes that we can use the native function feature to extend the language, and they expect us to do this to influence the world around Lox (eg, for file operations on a PC). My first attempt was to directly implement native functions, and the three functions above (gpio_put, gpio_init and gpio_set_direction) can pretty much be found in examples in the Pico C SDK. However, I’d rather implement these functions in Lox. What ‘native functions’ would those implementations need?

Making the implementation Lox

Reading the datasheets and examples for the Pico and RP2040, setting up a GPIO pin and changing its state is a matter of writing to control registers for the ‘SIO GPIO’ peripheral. These are presented in software as memory addresses, at a separate location to the built in RAM Lox is already using. Many registers are available at several addresses (‘aliases’) and the alias you use governs the exact behaviour you get when accessing it.

Using gpio_put as a simple example, I want the implementation to look something like this:

// Constants from the RP2040 datasheet
var SIO_HW = 0xd0000000;
var GPIO_OUT_OFFSET = 0x10;
var SIO_REGISTER_ALIAS_STRIDE = 4;
var SET_OFFSET = 1 * SIO_REGISTER_ALIAS_STRIDE;
var CLR_OFFSET = 2 * SIO_REGISTER_ALIAS_STRIDE;
var SIO_GPIO_OUT = SIO_HW + GPIO_OUT_OFFSET;

fun gpio_put(pin, value) {
  var alias_offset;

  if (value) {
    alias_offset = SET_OFFSET;
  }
  else {
    alias_offset = CLR_OFFSET;
  }

  poke SIO_GPIO_OUT + alias_offset, 1 << pin;
}

Looking at this code, I observed three features I wanted to add to Lox. Lox lacks a notation for hex constants, and it would be unpleasant not to use those. Lox also lacks the bitwise operations C has that are convenient for manipulating numbers that will get written to registers. Often these are a set of many small sub-fields. Here we can see <<, but &, |, ^, and >> will all be useful. Finally, there is a new statement – poke. This will form part of a pair, poke to write to memory locations that are peripheral registers, and peek to read them.

So I needed three additions: Hex numbers, bitwise operations, and the peek & poke pair to access memory associated with peripheral registers.

Hex Numbers

The full changeset at clox-pico: 007718c

I modified the scanner to spot ‘x’ occurring in a number, with a function ‘isRadix‘, and stash this in a new `TOKEN_HEX_NUMBER` token:

static Token number() {
  int radix = 10;
  if (isRadix(peek())) {
    radix = radixType(peek());
 
    // …

  if (radix == 16) {
    return makeToken(TOKEN_HEX_NUMBER);
  } else {
    return makeToken(TOKEN_NUMBER);
  }
}

Later on the compiler can convert the string from the source character by character into a number. This means that hex numbers live mostly in source code, and are entirely invisible to later code generation parts of the language. I could choose to resolve the hex constants entirely in the scanner, to TOKEN_NUMBER. Choosing the new token gives me slightly simpler code in the current implementation.

Bitwise operations

The full changeset at clox-pico: da1b7c6

I believe that bitwise operations are only meaningful if the underlying bit representation is clear, so the language needs to document that. In Lox, which uses a floating point double for numbers, this uncovers a new requirement: adding unsigned integer support. 

Lox represents numbers with the double C type, which on the Pico is a 64 bit IEE744 binary64 floating point representation. On the Pico target, I believe I can represent all registers with C’s 32bit unsigned integers. Therefore, if I make bitwise operations work well on 32bit unsigned integers, that should be sufficient to be usable. 

I could have chosen to add a new number type to Lox, but to keep things simple, and make progress, I chose to extract a 32bit unsigned integer from a double whenever I needed an operand for a bitwise operation. The double type uses 53bit significands, so 32 bit whole numbers are a subset. This means there will be an error if the number doesn’t fit, and some expense doing the conversion². I plan to remove these costs in future development by adding an integer type to the language alongside the current floating point numbers.  

With that decision, there is a straightforward set of changes to the code: in the scanner creating tokens for these operators; in the code generator to add them to the intermediate language and operations in the VM to finally execute them. They are very similar to the existing binary operations on doubles.

Peek & Poke

The full changeset at clox-pico: 5855762

Peek and Poke are widely used names for operations that access ‘raw memory’, or in this case memory mapped registers.

Peek is easily represented as a function taking one parameter, and returning one, so I used the existing native function extension straightforwardly. Poke is more complicated. We don’t actually want to return a value (registers are often write-only), so I need something closer to the print statement the language already has. I chose to add a new statement ‘poke’ that takes two parameters – an address, and a value to write. Both will be treated as 32bit unsigned integers, and any number not representable by those will cause a runtime error.

I started by adding a TOKEN_POKE keyword, alongside TOKEN_PRINT to the scanner:

static TokenType identifierType() {
  //…
  case 'p':
    if (scanner.current - scanner.start > 1) {
      switch (scanner.start[1]) {
       case 'o': return checkKeyword(2, 2, "ke", TOKEN_POKE);
       case 'r': return checkKeyword(2, 3, "int", TOKEN_PRINT);

I then parse the statement type, and generate code for it, using a new OP_POKE operation.

static void statement() {
  if (match(TOKEN_POKE)) {
    pokeStatement();
  } else if (match(TOKEN_PRINT)) {
    printStatement();
  // …

static void pokeStatement() {
  expression();
  consume(TOKEN_COMMA, "Expect ',' after address.");
  expression();
  consume(TOKEN_SEMICOLON, "Expect ';' after value.");
  emitByte(OP_POKE);
}

Finally I modified the VM to execute this new instruction³:

case OP_POKE: {
  if (!is_uint32(peek(0)) || !is_uint32(peek(1))
  {
    runtimeError("Operands must be numbers that can be uint32.");
    return INTERPRET_RUNTIME_ERROR;
  }
  uint32_t value = as_uint32(pop());
  uint32_t address = as_uint32(pop());
  volatile uint32_t* reg = (volatile uint32_t*) (uintptr_t)address;
  
  *reg = value;
  
  break;
  }

With these changes complete, I can now execute the script above, and implement the remaining gpio_init and gpio_set_direction functions.

Future work

These additions allow me to write code that targets any of the built in peripherals. There are some problems remaining. Some peripherals are looking for data addresses (for example, the DMA unit), so the language will eventually need some way to share addresses of data it creates. This will involve some care when I consider the garbage collection the language includes. Other peripherals will cause interrupts to occur, and these will need some way for Lox to respond, without losing track of the operation in progress at the time. Both of these will be problems I tackle in future work. These will take me quite a long way from the Lox I started with, so a new name will be appropriate too.

Conclusion

With three relatively simple changes, the largest of which is a new statement ‘poke’, I have the start of a dedicated microcontroller language. They are openly licensed, so you can too! Here’s the LED on:

If you want to use this code today, there is quite a range of projects that can be implemented by turning on and off GPIO pins, and the code here will extend to all the GPIO pins exposed on the Pico boards.

My own next steps are to tackle the future work above, which I’ve already started. The header pic is my language ‘Yarg’ generating Conway’s Life on an LED cube. I am just starting the journey towards a language that supports my goals of a tool that can be used from prototype to production, leveraging the power now available in microcontrollers. Star the Yarg repo if you’d like to learn more as I do!

---

I would like to thank the friends who helped review this material before I published it.

1. On the wireless variants, this is a bit more complex, as the LED is connected to the radio chip alongside the RP2040 ↩︎
2. There will also be occasional frustration when a floating point computation produces a number very close to an integer, and then cannot be converted. This does not seem to be a problem I’ve seen in practice yet with the simple calculations my code has needed to manipulate constants for register access. ↩︎
3. The implementation also has an #ifdef code path for compiling Lox on the host PC. In this case we just printf our parameters, so that stdout contains a log of register reads and writes. ↩︎

I’ve got to a place with my Yarg-Lang project that I want to start talking about it with others.

• r/Programming Languages: (Post on: ‘May 2025 what are you working on thread‘
• Threads: What I’ve been up to recently