Every October we release a new version of our reporting component for developers named List & Label. You can imagine what it's like the weeks before the release: time pressure and frustration one minute, relief and joy the next. So far, we've always made it so we could rightfully celebrate our little release party afterwards. Good thing, because a non-release party isn't that funny 😋.

Anyway, I'd like to share some insights into what happens as we shift near Release to Manufacturing.

First Stage – Alpha Version

The first stage of our release cycle is the release of our Alpha version used by some key partners and customers to adapt their additions to List & Label. The Alpha provides a first metric for the stability of the coming version. This milestone is usually achieved by the end of July. Based on the Alpha feedback we are able to fine-tune some of the features and make it to the public beta program in early September.

By that time, we already had fixed some 500 tickets in the Redmine ticketing system which we're using as bug management tool. BTW – if you're also using Redmine make sure to check out the free reporting solution for Redmine we've donated to the community a couple of years ago.

This is what our progress looks like for the last 60 days before the release of version 26 in October 2020:

As you can see, my will to keep the number of open tickets at or below 100 is a bit pathological 😉. This is a fact that's known by the whole team and everyone acts accordingly. We're now starting to burn down rapidly to reach the zero-bug-count in time with the end of the beta phase.

Beta Phase – Code Freeze

In the second phase, the beta version is released to all our Enterprise Edition subscribers. This offers ample opportunity to see if our latest bits still work nicely in your own application. We're getting loads of useful feedback through this channel and even get an opportunity to shape the final touches of newly added features.

During the beta phase we've hit the code freeze for UI resources. Translation takes some time and we're releasing List & Label in a number of languages (English, German, French, Italian, Spanish and Chinese). These "core" languages are being translated by a professional translation service. Other languages are provided by selected partners and customers and may become available shortly after release.

The actual code freeze is just after the end of the beta phase. We only fix killer bugs after this milestone, and only on a peer-peer-review base.

Of course we run an intense and thorough in-house beta test in addition to our external beta program. This internal process is repeated for each sprint in our agile planning process to enable incremental releases at any stage.

Automated Builds and Tests

Manual testing is complemented by automated tests. Both UI and reporting engine are subject to automated testing for most of our nightly builds. The builds and tests are fully automated by our CI server – we use TeamCity for that purpose. We use commercial tools as well as validation tools we've created just for this purpose. The main reporting engine tests consist of several hundred reports which are run by the engine and validated against references. Each deviation from the reference files is automatically reported and raises yet another ticket in Redmine. The QA process also includes performance tests to make sure there are no latencies induced by accident. This way we have been able to provide high quality releases with the most recent versions.

This is a view of our TeamCity project:

As you can see, the last List & Label 26 Core build failed. The reason was a failed regression test that can easily be verified by looking at the build's artifacts:

The file itself is a "before/after" comparison of the last build's output vs. the stored reference. To get an impression of the number of tests, here's a scroll over the results of a typical build:

Wrap-Up

So this is how we've headed towards our 26th version. Despite the pandemic craze and everyone still working from home, we've been able to stick to our usual processes.

We've come a long way since the initial release of List & Label back in 1992 when the tool was born out of the need for a professional reporting component to use with combit's CRM applications (click only if you understand German 😊, it's only made for this market). Our great team has grown through the years to offer professional services you've come to expect from us. Thanks to everyone involved in-house as well as all of our faithful customers. It's been a pleasure to work with you through the last 26 versions, and I'm looking forward to yet another great release very soon.

If you are interested in the features we've developed for our reporting tool, you can check out the development history (sorry for the German learners, this page is in English again).