Testing on Simulators


I’ve been doling out advice on how to use Ceedling to unit test on various simulators for years now… and I’ve gotten Ceedling ported to a dozen different toolchains myself… but this advice often turns into generic “shoulds” and “woulds” instead of concrete examples. I’d like to change that.

In addition to publishing my own custom ports to various toolchains, I’d appreciate anyone who can forward their own project yaml files, support files, and notes for using various targets, simulators, etc. My plan is to collect them into a central repository that can be accessed by anyone… and (of course) I would love to share credit with you if you wish!

If you have any such projects, no matter if they are the start of a port, fully working, or cleanly polished, I’d really appreciate your contribution. You can send them to me directly via email at mvandervoord@gmail.com, or we can arrange something else if you would prefer.

Thanks for the help! You’ll be making future users very happy!

Unity + Jumper = Embedded Software Unit Testing, The Easy Way

This article is reposted with permission of Jonathan Seroussi from Jumper. While Jumper isn't a project of ThrowTheSwitch, it has been designed to work well with Unity and is of interest to our readers. You can find the original article here. ]

Unity and Jumper Virtual Lab.png

Proper unit testing is a must for any software project. To allow unit testing for a software project the R&D team must write a testable, modular code — code that can be divided into self-contained units that can be tested. On top of making the code testable, embedded software developers must make sure their code is portable. On this blog post, we are going to demonstrate an easy way to have unit testing for embedded software. We’ve combined the Jumper Virtual Lab and the popular Unity Testing Framework by Throw The Switch.

The folks at ThrowTheSwitch are on a mission to unleash the powers of C for Embedded Systems without fearing the pitfalls of C. A flexible unit testing framework is a key component in responsible, test driver development.

Unity by ThrowTheSwitch is a superb unit testing framework for C. It’s lightweight, compiles on everything, has a ton of assertion options and connects nicely with CMock for easy full mocking support.

Once you’ve written your embedded code unit tests, there are 3 ways of running them.

  1. On target — run on the designated board or MCU. This works to a certain extent but setting up automation is a bit of a pain. Plus if you do end up automating it, make sure you don’t over-flash your ROM as it’ll get messed up sooner rather than later.
  2. On a PC — maintaining a proper Hardware Abstraction Layer and separating out the application code will make your application testable on an x86 or x86_64 machine. It’s awesome for app code, but you’ll need to mock pretty much everything outside of your application code.
  3. On an emulator — running on something like QEMU or the Jumper Virtual Lab allows full flexibility on what you mock or not, along with the benefit of easily plugging to a test automation framework.

We’re going to demonstrate how to run Unity-based unit tests on the Jumper Virtual Lab. We’re currently supporting the nRF52 by Nordic Semiconductor (STM32 support coming soon). The Virtual Lab is able to execute binaries that were compiled for the nRF52 without having to change anything about your code or how it compiles.


To install Jumper Virtual Lab, go to this link and follow the installation instructions. You’ll need to create an account and download a token file, and then install a python module using pip. And that’s it! At this stage Jumper only works on Ubuntu 16, but Windows and Mac support is coming soon.

Running the sample

Download this sample and unzip it. Inside, you’ll find a pre_compiled folder and a source folder. For convenience, we’ll use the pre compiled option.

In your terminal, go to the pre_compiled folder and enter

jumper run -u -b unity.bin

After a few seconds, the following output will start printing onto your terminal:

Loading virtual device ………… Done

Virtual device is running
test/TestProductionCode.c:32:test_FindFunction_WhichIsBroken_ShouldReturnTheIndexForItemsInList_WhichWillFailBecauseOurFunctionUnderTestIsBroken:FAIL: Expected 1 Was 0
test/TestProductionCode.c:60:test_FunctionWhichReturnsLocalVariable_ShouldReturnCurrentCounter_ButFailsBecauseThisTestIsActuallyFlawed:FAIL: Expected 0x00001234 Was 0x00005A5A
— — — — — — — — — — — 5 Tests 2 Failures 0 Ignored


This test fails on purpose.

Pretty cool, right?

The Code

Let’s drill down to the code. In the sample zip, go to source/main.c. The main.c file includes standard usage of the unity framework. Let’s look at this code:


After calling some functions that use assertions in the sample/source/test/TestProductionCode.c, it’ll call int unity_code = UnityEnd(); to get the number of failed tests.

Here’s a cool feature of the Virtual Lab we designed for these case. The last line: jumper_sudo_exit_with_exit_code(unity_code); is used to stop the program execution and return an exit code to the process that executed the test. This is very useful when running the test through a CI tools like Jenkins and CircleCI.

Using jumper_sudo to terminate the test execution

To do that you’ll need to use Jumper’s exit function:

void jumper_sudo_exit_with_exit_code(uint32_t exit_code)

To add the jumper_sudo code, follow the these instructions:

  1. Get the Jumper Virtual Lab Addons library by cloning this git repo:
  2. Add #include "jumper.h" to your .c file.
  3. Add the jumper.h and jumper.c files from this library to your build.

And you’re set!

This post was based on the following link from the Jumper Virtual Lab documentation.

Trying out Jumper Virtual Lab, Part I: Getting to Blinky

This article is reposted with permission of Luca Ingianni. Luca specializes in bringing DevOps to the embedded world: improving both the lives of engineers and the quality of their products. While not specific to Unit Testing, it discusses a tool which might be useful to members of the ThrowTheSwitch.org community ]

I’m very excited to transfer the principles of DevOps to embedded software and the Internet of Things (gotta be buzzword-compliant), and new tools promise to get us closer to this goal.

A lot is happening in the embedded system tooling space right now; I’m hopeful that the methods used in embedded may finally catch up to the rest of software development.

A lot of startups explore new ways to aid in development, and make the difficult world of embedded development more welcoming.

Introducing Jumper Virtual Lab

Today, I want to introduce you to one such new, interesting tool: Jumper Virtual Lab As you read my discussion of it, keep in mind that Jumper Virtual Lab is still in beta, and various rough edges will certainly be smoothed out as the product is released.

This post will be the first in a small series, exploring Virtual Lab. As is custom in the embedded world, our first attempt will be to get a LED to blink.

Except, with a twist: first we’ll use a physical µC, and after that we’ll execute the exact same binary on Virtual Lab’s simulation, to find out if it actually works like the real thing.

What is it?

Virtual Lab is a hardware virtualisation solution for embedded systems. It allows to simulate microcontrollers to a degree that native code will run on these virtual devices.

This opens a range of possibilities:

  • Create virtual prototypes, and try your software under real-life conditions before physical prototypes have even been delivered
  • Create test scenarios which are otherwise hard to achieve: write to read-only memory, mess with data lines, pause and resume execution of an entire assembly (sensors and µC) without fearing desynchronisation, etc
  • Be unconstrained by number of prototypes: run multiple tests in parallel, afford all developers heaps of virtual devices to try things with
  • Execute at more than real-life speed, or skip in time, to accelerate tests
  • Simplify your CI/CD pipeline, by not having to rely on temperamental HiLs: turn system test into an all-software affair

Getting to Blinky

As promised, we’ll start by using real, proper hardware: the Nordic Semiconductors nRF52 ARM-based microcontroller.

We’ll use Nordic’s nRF52 DK development kit and the SDK’s blinky code example as the basis for our physical tests. Once we get our toolchain running, we’ll satisfy ourselves that Jumper can successfully run the same binaries in its simulation.

Creating the Blinky Binary

Just to make sure you understand my starting point, here’s an outline of the environment I used:

  • nRF5x SDK 14.2.0
  • GNU Am Embedded Toolchain 7-2017-q4-major
  • nRF52 DK development board
  • Jumper 0.0.52

Go to the SDK’s blinky example, build it, and flash it:

$ cd nRF5_SDK_14.2.0_17b948a/examples/peripheral/blinky/pca10040/blank/armgcc/

$ make
mkdir _build
cd _build && mkdir nrf52832_xxaa
Assembling file: gcc_startup_nrf52.S
Compiling file: system_nrf52.c
Compiling file: main.c
Compiling file: boards.c
Compiling file: app_error.c
Compiling file: app_error_weak.c
Compiling file: app_util_platform.c
Compiling file: nrf_assert.c
Compiling file: nrf_strerror.c
Linking target: _build/nrf52832_xxaa.out
   text    data     bss     dec     hex filename
   2088     108      28    2224     8b0 _build/nrf52832_xxaa.out
Preparing: _build/nrf52832_xxaa.hex
Preparing: _build/nrf52832_xxaa.bin
DONE nrf52832_xxaa

$ sudo nrfjprog --family nRF52 -e
Erasing user available code and UICR flash areas.
Applying system reset.

$ sudo ~/projects/spikes/nrf52/nrfjprog/nrfjprog --family nRF52 --program _build/nrf52832_xxaa.hex
Parsing hex file.
Reading flash area to program to guarantee it is erased.
Checking that the area to write is not protected.
Programing device.

$ sudo ~/projects/spikes/nrf52/nrfjprog/nrfjprog --family nRF52 -r
Applying system reset.

And see its glory:

Blinky! (click for the action!)

Blinky! (click for the action!)

installing Jumper Virtual Lab

Conveniently, Jumper comes as a Python 2 module and can be installed via pip.

Follow the instructions here

Once it’s installed, let’s try it out:

$ jumper run --bin _build/nrf52832_xxaa.bin
Loading virtual device

Success! I guess.

Getting to Virtual Blinky

I mean, the previous command was clearly doing… something.

But it’s not a proper blinky yet, is it? I don’t know about you, but I didn’t see anything blinking at all.

Let’s change that!

Let’s build a simple blinky-simulation:


from jumper.vlab import Vlab

# this array holds the four LEDs' visualisation

def on_pin(number, level):
    """This function will be called back for each state change of a pin.
    In this case, we use it to simulate the visible behaviour of the nRF52 DK:
    the blinky example makes the DK's four LEDs bink.
    This function visualise these four leds"""

    # update the state of the simulated LEDs
    if level:
        LEDs[number-17] = "XX"
        LEDs[number-17] = ".."

    # show them in the terminal
    print("   " + LEDs[0] + "   " + LEDs[1])
    print("   " + LEDs[2] + "   " + LEDs[3])

    # go back up three lines, so we overwrite our previous rendering of the LEDs the next time around

# set up the device simulation
v = Vlab(working_directory=dir)

# register the callback which visualises the LEDs

# run the simulation

Once created, you can run it:

$python test.py

You'll see happy blinking XX's. Isn’t it pretty?

Verdict: so far, so cool

So there you have it: a simple device simulation in (generously) ten lines of code, which can be plugged right in place of a physical device.

Compared to where we were only a decade ago (and where most embedded development teams still are), this is pretty amazing.

The next installment of this post series will use Jumper for proper testing, and eventually place it in a complete CI workflow. I’m excited to see how well it’ll work. Are you?


We'd like to start by thanking everyone who participated by voting or commenting. We posted the poll as a bit of an experiment. It was hard to predict what kind of response we would get. We were really excited to see how many people participated in this topic! This definitely motivates us to figure out more ways to plan our roadmap and features together!


The results looked like distribution above. You can see that our opinions are fairly divided, though there are definitely a couple of peaks in sticking with the current behavior (having TEST_ASSERT_EQUAL work as an alias for TEST_ASSERT_EQUAL_INT) and those who want it to be a generic comparison that doesn't really provide details.

There were a number of comments added as well that were also valuable.

  • There was concern that the "Use ==" option would result in cryptic compiler bugs. There are things in C that you just can't compare that way. We all know floats have subtle issues. Even worse, try directly comparing structs and see where that gets you. ;)
  • There are a number of people who voted for integers, but primarily because of momentum / backwards compatibility. Their true preference varied.
  • We did get a couple of people who mentioned MISRA Compliance. This isn't the first time we've heard this (though it's an amusing thing to find on this poll). Please note the contact page. Feel free to send us more detailed information on how we can head in that direction!

Decisions, Decisions.

So where does that leave us on this decision? We try hard not to bloat Unity too badly, but it appears that this is a place where we could easily add some configurability (We know. That was the lowest ranked option... let's discuss).

To keep those upgrading happy (and those who really like Integer as a default option), we'd default to having TEST_ASSERT_EQUAL be an alias for TEST_ASSERT_EQUAL_INT. When people don't love that option, they could use one of these beauties:

  • UNITY_SHORTHAND_AS_INT - Does an integer compare. Since this is the default right now, this does nothing. But by including it, we can get people to start using it and have an excuse to change the default down the road.
  • UNITY_SHORTHAND_AS_MEM - Does a memory compare using sizeof() for len.
  • UNITY_SHORTHAND_AS_RAW - Just uses == to compare. Doesn't give details.
  • UNITY_SHORTHAND_NONE - Will remove this option, replace it with an automatic failure complaining that this isn't a feature anymore.

The nice thing about doing this is that it's really not MUCH more code. The first two are aliases for existing options. The Raw is really just going to use TEST_ASSERT_TRUE under the hood. The None option is just going to use TEST_FAIL_MESSAGE. Bam. Done.

Feel free to comment below if you love / hate this option. We hope to get to this sometime soon, but you have a little time. :)

Please Vote


Unity has had a TEST_ASSERT_EQUAL macro since the beginning of time (or at least as far back as Unity has existed). More than any other macro, this one has caused confusion. So how WOULD you like to see the TEST_ASSERT_EQUAL (and possibly the TEST_ASSERT_NOT_EQUAL) macro to behave?

We held a vote in this post previously. You can find the results here.


Unit Testing Embedded C with Ceedling

If you're interested in getting started with unit testing for embedded applications and you'd like to read an excellent tutorial, you should check out this post by Dmitry Frank. It'll walk you through installing and configuring Ceedling (and therefore Unity and CMock) and through tests for a small example. It's well thought out and a good read.

Dmitry is a talented engineer from Russia. He's worked on a variety of embedded projects, including his own preemptive RTOS which has been used in the wild. You can learn more about him at his website.

We're Famous! (but not really)

One of our lead scientists, Mark, had the awesome opportunity to be interviewed on Embedded.fm and today it went live (follow the link to hear it!). Elecia and Chris do a great job with the show (listen to their other episodes, if you haven't already). Mark learned that it's really hard to explain programming concepts without a whiteboard. Apparently, he'll need to stick to mad science and embedded software and give up his "dream" of being a famous talk show host.

Lab Upgrade!

We've been working miracles out of that old rundown lab for a long time. It's hard to really do mad science well when your lab is full of half-working rundown equipment. Well, we've got a full-blown lab upgrade, now!

(enjoy the new site. If you run into any problems, let us know! Please be patient with us while we migrate your favorite features from the old site to the new format).


- - - The Mad Scientists

It's Alive!

Are you going mad with anticipation!? We have been! 

Our course on Test Driven Development for Embedded Software is finally live! Check out what it's all about. You can get yourself a head start on Unit Testing with Unity. Plus, if the response is good, we'll be getting right back to work on developing part II which will include mocking and interaction-based testing --- and you'll be getting in on that for less than everyone else!

Sign up for just $95. Heck, use code SURLYSCHOOL and take $10 off of that!

When Bad Code Runs Green

Contributed by Matt Wilbur

This is a cautionary tale.

I was recently test-driving a UART module. Nothing earth-shattering, I know. For my current project, I have adopted the "DH" pattern for my driver work, where the Driver provides useful services and interfaces to higher level code and relies on a separate Hardware class (well, in C, a compilation unit), to hide niggly hardware details. I had a simple task for my UART: Receive a character. Pretty basic stuff, eh? That's what I thought.

As I test drove my driver code, I started to add more funky stuff, like allowing my imaginatively-named string receiving function UartDriver_ReceiveString to return before an entire string was read in the case that it received a UART_RX_NOT_READY value from the hardware. That might happen if some clever user asked my driver to always return immediately. Here's the code:

s32 UartDriver_ReceiveString(UartDriver driver, s8* stringToReceive, u32 lengthOfString)
int i;
s8 theChar;

if ( stringToReceive == 0)
return 0;

if ( driver == (UartDriver) 0 )
return 0;

for (i=0; i<lengthOfString; i++)
theChar = getByteFromHardware(driver);
stringToReceive[i] = theChar;
if (theChar == UART_RX_NOT_READY)
return i;

Pretty straightforward, right? The helper function getByteFromHardware called one other function, which I'll replicate here as it was when it was first written.

s8 getByteFromHardware( UartDriver driver )
if( driver->timeout == UART_DRIVER_TIMEOUT_INFINITE )
return getByteFromHardwareWhenItsReady( driver->hardware );
return getByteFromHardwareAndReturnImmediately( driver->hardware );

s8 getByteFromHardwareWhenItsReady( UartHardware hardware )
volatile s8 theCharacter;

theCharacter = getByteFromHardwareAndReturnImmediately( hardware );
} while ( theCharacter == UART_RX_NOT_READY );

s8 getByteFromHardwareAndReturnImmediately( UartHardware hardware )
return UartHardware_ReceiveCharacter( hardware );

"Hey!", you say. "How dare you write code without a test!"

Don't worry. I didn't. Here was my simple test:

void test_UartDriver_ReceiveString_ReceiveGetsNegOneIfTimeoutIsNoWait(void)
s8 stringToReceive[5];

UartHardware_ReceiveCharacter_ExpectAndReturn(mockHardware, UART_RX_NOT_READY);

UartDriver_SetReceiveTimeout(mockDriver, UART_DRIVER_TIMEOUT_NO_WAIT);

TEST_ASSERT_EQUAL(0, UartDriver_ReceiveString(mockDriver, stringToReceive, 5));
TEST_ASSERT_EQUAL(-1, stringToReceive[0]);

"Hey!", you say (stop shouting, please, I can hear you fine). Why so many friggn' calls? We're embedded software designers. We don't like the overhead of such things. True enough. But I will invoke a few items from my defensive bag of tricks.

  1. I'm striving for clarity in the code. As long as I'm not actually running out of ram or failing some sort of timing requirement, I'm quite comfortable with these calls. In this case, I don't have a timing requirement.
  2. One can inline things later. A la C/C++ (with inline), not a la Java with Eclipse.
  3. Don't try to out-optimize your compiler. You won't win.

If you still don't like it, I'm okay with that. TDD has made me confident enough in my code that I'll still sleep at night. I'll also refer you to Test Driven Development For Embedded C, 12.5, page. 267. Don't have the book? Go buy it.

As any sane person would, I use Ceedling to do my TDD. I felt very clever when all tests ran green.

Test 'TestUartDriver.c'
Generating dependencies for UartDriver.c...
Compiling UartDriver.c...
Linking TestUartDriver.out...
Running TestUartDriver.out...

Awesome. So, I took this code, made a library and linked it with my application. Badness. Bad badness. A simple, simple CLI (linenoise), was not receiving characters. How dare it! I tested this code! It's good! I figured I misunderstood the hardware (mocking only buys you so much). I spent several hours scratching my head during DOH (debugging on hardware). Finally, I realized what happened. My helper getByteFromHardwareWhenItsReady was not helping. It was hindering. It kept the received character for itself and did not share it. Very rude.

So how did this slip through my safety net? Bad luck. In assembly, here is how UartDriver_ReceiveString breaks down around the call to getByteFromHardware.

0x804acd5 <UartDriver_ReceiveString+47> call 0x804ad03 <getByteFromHardware>
0x804acda <UartDriver_ReceiveString+52> mov%al,-0x9(%ebp)

Okay, so we know getByteFromHardware puts the received char in al. So, what does getByteFromHardware look like?

0x804ad1c <getByteFromHardware+25>call 0x804ad32 <getByteFromHardwareWhenItsReady>
0x804ad21 <getByteFromHardware+30>jmp0x804ad30 <getByteFromHardware+45>
0x804ad23 <getByteFromHardware+32>mov0x8(%ebp,%eax
0x804ad26 <getByteFromHardware+35>mov(%eax),%eax
0x804ad28 <getByteFromHardware+37>mov%eax,(%esp)
0x804ad2b <getByteFromHardware+40>call 0x804ad50 <getByteFromHardwareAndReturnImmediately>
0x804ad30 <getByteFromHardware+45>leave

What a clever compiler. It simply calls getByteFromHardwareWhenItsReady or getByteFromHardwareAndReturnImmediately and then leaves eax in a pristine state so that getByteFromHardware can use it. But this is where it ought to break, no? I am not actually returning the character from getByteFromHardwareWhenItsReady like I should. In C, I am not. But that clever compiler was too clever for me.

0x804ad3e <getByteFromHardwareWhenItsReady+12>call 0x804ad50 <getByteFromHardwareAndReturnImmediately> 
0x804ad43 <getByteFromHardwareWhenItsReady+17>mov%al,-0x9(%ebp)
0x804ad46 <getByteFromHardwareWhenItsReady+20>movzbl -0x9(%ebp),%eax
0x804ad4a <getByteFromHardwareWhenItsReady+24>cmp$0xff,%al
0x804ad4c <getByteFromHardwareWhenItsReady+26>je 0x804ad38 <getByteFromHardwareWhenItsReady+6>
0x804ad4e <getByteFromHardwareWhenItsReady+28>leave
0x804ad4f <getByteFromHardwareWhenItsReady+29>ret

See what's going on? You don't? Come on! It's staring you in the face. Nothing happens to eax between the call to getByteFromHardwareAndReturnImmediately and the ret. So, the return value from getByteFromHardwareAndReturnImmediately (theCharacter) is being returned up to UartDriver_ReceiveString, as far as my x86 computer is concerned. Suppose I force another call to a function before returning so eax is modified.

s32 foo() { return -1; }

s8 getByteFromHardware( UartDriver driver )
if( driver->timeout == UART_DRIVER_TIMEOUT_INFINITE )
return getByteFromHardwareWhenItsReady( driver->hardware );
return getByteFromHardwareAndReturnImmediately( driver->hardware );

s32 bar = foo();

Sure enough. Failures galore. If we return theCharacter after that call, all is well again.

As an epilogue, it was (rightly) pointed out to me that I should really have some dedicated tests fore these helper functions. That is, in addition to test_UartDriver_ReceiveString_ReceiveGetsNegativeOneIfTimeoutIsNoWait, I should have a test like this:

void test_UartDriver_Helper_getByteFromHardwareAndReturnImmediatelyReturnsCorrectByte(void)
UartHardware_ReceiveCharacter_ExpectAndReturn( mockHardware, UART_RX_NOT_READY );
TEST_ASSERT_EQUAL(-1, getByteFromHardwareAndReturnImmediately(mockDriver));

I did add that. Same problem, but I feel better having that test in place.

And that, my friends, is how your compiler can screw you and why sometimes tests can't always save your ass from dumb mistakes.

As postscript (which differs from an epilogue, I hope) I realized a fundamental flaw in my zeal to get a working UART driver. In fact, I'd be surprised if Kent Beck weren't getting out a ruler at some point during this note so he could smack me on the knuckles (the Kent-Beck-in-my-head that is. I'm not delusional enough to think Kent will read this). In case you're as dim as I was, I'll confess my sins in the hope that it will save my knuckles. Typing verbatim from my Test Driven Development by Example, by (the real person) Kent Beck, here is the TDD micro-cycle:

  1. Quickly add a test.
  2. Run all tests and see the new one fail.
  3. Make a little change.
  4. Run all the tests and see them all succeed.
  5. Refactor to remove duplication.

Okay, maybe there was more than one flaw (pobody's nefect), but the biggest one that could have avoided this whole ordeal is number 2. I didn't see the test fail. I was so happy with myself and my ace coding skills that the test passed the first time, I didn't test the test. Which, to me, is one of the functions of step 2. Bad me, but lesson learned. Hopefully, you've learned it too.