An Overview of Unit Testing Tools for LabVIEW

Posted on by Brett Percy

It’s great that there are so many choices for unit testing in LabVIEW, but how do you choose which framework is right for your project? If you are unsure where to start with unit testing or which tool might best suit your needs, read on

I’ll start with an overview of the current unit testing options for LabVIEW, and my take on their strengths and weaknesses. It does assume some familiarity with the terminology of unit testing, so it doesn’t attempt to explain every term. I’ve also broken out a list of features that should help you decide which tool is most appropriate for your needs.

JKI VI Tester

This is JKI‘s first Unit Testing framework, originally released in 2009 and later published as an open source project in 2017. JKI also developed Caraya which is described below. Both frameworks have quite different takes on how to write and run unit tests

VI Tester is implemented using LabVIEW classes, and requires subclassing of the provided TestCase class to implement tests. If you or your team are not comfortable with LVOOP1 then VI Tester may not be for you

This is a tried and tested framework, and probably the most widely used of all of the tools I’ve looked at. It’s not the latest cool thing, but I’d argue that’s a feature not a bug when you’re talking about a framework you rely on to ensure that your own code is bug-free. Having said that, the project is not as actively maintained as it used to be. At the time of writing, the last release was in February 2019, and there are a number of open pull-requests from more than 12 months ago

VI Tester is based on the xUnit software test architecture, which is an industry standard approach used by unit testing frameworks for many other languages, like jUnit for Java and xUnit.net for C# and friends. It provides a couple of base classes that you subclass to create your tests, and automatically extracts the unit tests from your project so they can be run

Because of its community support, there are many additional tools available for VI Tester and tutorials on how to use it, including how to integrate it with CI2 tools like Jenkins

JKI VI Tester is available to download in VI Package Manager, and the source code is on Github: https://github.com/JKISoftware/JKI-VI-Tester

NI Unit Testing Framework

NI’s Unit Testing Framework is the 1st party option for unit testing in LabVIEW. NI has an ISO 9001 certification, so this framework is a good choice if you are working in a regulated environment

UTF can be used with classes, but requires writing wrappers around method VIs so it’s not the best option if you use LVOOP

Unit Tests are configured using a dialog box to edit the .lvtest file that actually specifies the test. I find hopping in and out of various parts of the dialog to be a bit cumbersome compared to writing test code directly. The .lvtest file is just a tab-delimited text file though, so it’s possible to manually edit the test values, or create dedicated tools to modify the file3

One interesting feature of UTF is support for test vectors. This allows multiple tests to use the same test VI but with different sets of test data and expected results. As with the .lvtest file, these vectors are stored as text files so it’s possible to build up tooling to create them automatically from a requirements document or spreadsheet

NI Unit Testing Framework is available in the NI Package Manager

Caraya

JKI’s newer unit testing framework, Caraya, launched in late 2015. Like VI Tester, this is an open source project with an active community of users

It might seem odd that one company has released two different unit testing frameworks, but the tools work sufficiently differently to justify both existing

Caraya’s approach is to let you to take the existing test harness code you have cobbled together to test something in your application, and turn it into a unit test with a minimum of fuss

This is a light-weight framework, in contrast with VI Tester, which means that you need to provide your own structure for your tests. It doesn’t provide, or force you to use, xUnit style set-up and tear-down methods. Whether you find this a benefit or not is a personal choice – I actually quite like the more opinionated style of VI Tester

Caraya also provides assertions, which fit in nicely with its ‘the VI is the test’ approach. Assertions4 are included as part of your project’s built executable, unlike unit tests which are typically only run during development or before building the executable. They are just a VI that generates an error when the expected inputs are not met, so they can be used to check values at runtime in order to log or otherwise handle problems. Caraya’s unit tests are built on these assertions, with some additional VIs to set up the name of the test

Caraya is available to download in VI Package Manager, and the source code is on Github: https://github.com/JKISoftware/Caraya

JKI have a forum for Caraya at https://forums.jki.net/forum/65-caraya-unit-tester/

InstaCoverage

InstaCoverage is the newest of the tools I’ve looked at, being released in 2017. It’s a commercial unit testing framework that also has a free Core version available with some reduction in features compared to the Pro variant

Like NI, IncQuery Labs have an ISO 9001 certification, making InstaCoverage suitable for use in regulated industries

Speed is a core feature of InstaCoverage, as well as providing code coverage reporting, which is lacking in most of the other tools. It’s also the only unit testing tool that currently offers support for LabVIEW NXG – but only in the free version. InstaCoverage also supports testing on RT targets, but without code coverage reporting

InstaCoverage focuses on testing individual VIs – right-clicking on a VI in the project allows you to create the test harness for it, which includes xUnit-style setup and teardown VIs. You can add additional test cases in the test editor, but you then need to manually add the new test case to case structures in the setup and teardown VIs. The absence of scripted support for adding these test cases is an unfortunate missing feature in an otherwise very powerful unit testing tool.

The free Core version is available in NI Package Manager, and both Core and Pro versions can be downloaded directly from https://instacoverage.io/download/

AST Unit Tester

This is another unit testing tool that’s intended to be fast to execute and simple to use. It’s a good way to dip your toe into unit testing as it doesn’t require AST Unit Tester to be installed in order to open a project that uses its tests. This means you can explore adding unit tests without inflicting any dependencies on the rest of your team

AST Unit Tester doesn’t require LVOOP, and test VIs just use a simple #AST_Unit_Test bookmark and a connector pane convention. Apart from that there’s no enforced structure and you are free to write your tests however you like – so long as they return True for a pass and False for a fail. As with Caraya this is both good and bad as you are responsible for enforcing any coding conventions rather than relying on the tool to lead you in the right direction

There’s no hierarchy to the tests, you can either run one test or all of them, so this is not really suitable for large projects. There’s also a GUI-less runner VI included that allows the tests to be run by a CI tool, but its output appears to be restricted to plain text

AST Unit Tester is available in VI Package Manager

So which tool meets my needs?

If you have any must-have requirements for your unit testing tool, I’ve catagorised the tools here to make it easier to decide which one matches your needs

I need a unit testing solution for use in a regulated environment

  • NI Unit Testing Framework
  • InstaCoverage

I would prefer an Open Source tool

  • JKI Caraya – BSD3 License
  • JKI VI Tester – BSD License

Free as in beer

  • JKI Caraya
  • JKI VI Tester
  • AST Unit Tester
  • InstaCoverage Core5
  • NI Unit Testing Framework (if you already have a LabVIEW Pro license)

I need reporting of test coverage as well as results

  • InstaCoverage Pro
  • NI Unit Testing Framework

CI Integration (xUnit compatible report generation)

  • JKI VI Tester
  • JKI Caraya
  • InstaCoverage Pro
  • NI Unit Testing Framework

Ongoing Commercial Support is important

  • NI Unit Testing Framework
  • InstaCoverage

I’m using (or planning to use) LabVIEW NXG

  • InstaCoverage Core6

I require Real-Time Target support

  • InstaCoverage7
  • NI Unit Testing Framework8
  • Caraya9

I don’t want to use LVOOP

  • InstaCoverage
  • NI Unit Testing Framework
  • JKI Caraya
  • AST Unit Tester

Built-in support for Test Vectors

  • NI Unit Testing Framework

If you have more questions about using any of these tools, or about LabVIEW unit testing in general I suggest heading over to the Unit Testing Group in NI’s forums – just follow this link: https://forums.ni.com/t5/Unit-Testing-Group/gp-p/5182?profile.language=en

Subscribe to our newsletter

Did you find this post helpful? Sign up to the Lonely Ant newsletter to get more useful tips delivered directly to your mailbox

Unsubscribe at any time. I will not share your information with third parties.

* indicates required

  1. LabVIEW Object Oriented Programming – try these if you’re starting out https://devs.wiresmithtech.com/blog/learn-teach-labview-oop/ or https://labvolution.com/labview-getting-started-with-lvoop/
  2. Continuous Integration
  3. I’ve seen this approach used in TestStand to good effect
  4. Assertions are a way of capturing what the expected value of something is, like ‘this boolean should be true’ or ‘this value should be less than five’. An assertion is raised if value doesn’t match what is expected
  5. InstaCoverage Core doesn’t include code coverage or support for Real Time targets, but is provided free of charge
  6. the Pro version is not compatible with NXG according to https://instacoverage.io/details/
  7. Pro version only
  8. UTF doesn’t do code coverage calculation for real-time targets, but tests will run
  9. You will need to set the test suite to run non-interactively as the UI won’t run on RT