Download Full PDF version

We created this blog to provide an essential, independent guide for Testing using Atlassian. Atlassian’s Marketplace is vast and rapidly changing and organizations have a multitude of choices for many solution areas – testing is especially rich. With the accepted demise of the old HP ALM platform, now owned by Micro Focus, many organizations have realized that they already have Jira, an excellent platform to support all of these testing functions.

We are not going to focus on feature matrices – these are easily manufactured and are anyway a moving target based on each app vendor’s own development efforts.  Instead, our analysis dives inside these apps to understand how they address the secret sauce of testing – how teams together identify problems and fix them quickly.  Atlassian’s mantra and moniker is Teams – and it’s enabling shared vision, insights and actions that’s at the crux of Blended Perspectives’ mission as a consulting firm. Blended Perspectives will enable you to “Test Together”; equally to “Plan Together” and “Service Together” – for a tool is only transformative when impactful in scope and artful in execution.

Apps is used throughout this paper (Atlassian’s old terms were “plugins” and “add-on”).  This paper provides a context for assessing testing apps, identifying the relevant apps for discussion and analyzing their strengths and weaknesses.  Since the details are a moving target our objective is less to understand whether an app has this or that feature – rather we provide a logical framework to help you select the right app for you.  A few readers will be unfamiliar with Atlassian and Jira so there is a short primer on the tools below.  Experienced testers should easily grasp the points being made, even without a Jira background.


There exists two categories of testing apps:

  1. Jira-based apps – built for Atlassian and fully integrated with Jira (for Server or Cloud)
  2. Standalone apps designed to integrate with Jira (e.g. Zephyr Enterprise or qTest)

This paper focuses on the most common scenario – Jira-based apps for Server. Further we’re leaving Cloud vs Server and the second category for another white paper. Data Center – highly scaled and resilient – is considered alongside Server. While maturing, today Cloud apps tends to run less feature rich versions than apps for Server. So we assume that going Cloud simply amplifies or constricts the pros and cons of each app discussed in this paper.

The standalone testing apps in the market such as Zephyr Enterprise have a much richer set of capabilities but unlike the old HP ALM application these apps are designed to seamlessly integrate with Jira.

Applications Reviewed

The apps we review are as follows:

  1. Test Management for Jira (TM4J)
  2. Xray
  3. Zephyr for Jira

All of these are excellent apps – but each comes at the testing space from different vectors. Blended Perspectives has experience implementing all three. We have practical “real world” observations of their strengths and weaknesses. It’s this we share.

The Atlassian Stack and Context for Testing

In the diagram below the very basic model of how Jira works is explained. Jira calls all its record items “issues”. Essentially types of issues are created that really can be anything – not only epics and stories, but also risks and other project-control records, and potentially tests.  The key is that once issues are created they can be managed in the various facilities that Jira provides, such as Filters, Projects and Boards.  When considering third party apps, an important question is: how does the application work with other Jira facilities, rather than as a standalone product? Conversely, what does the new application need that is not natively available in Jira.  A further element is that reporting apps such as eazyBI can generate excellent reports if the items or issues in question conform to the basic Jira architecture displayed below:

The next aspect to understand is how the various core Atlassian applications work together to support the Software Development Lifecycle (SDLC).  Testing is best thought of in the context of the SDLC main functions.  As an example, when code is built using a build engine such as Bamboo – it will automatically detect and expose any breaks – this in itself is a form of initial basic testing.  Moreover, build engines can trigger further automated testing which will create defects that need triage in the development backlog.

Best practices for using Atlassian for SDLC are generally as follows:

  • Document requirements more completely in Confluence with appropriate flow charts and context of how epics and stories are mapped against the application.  (Later this will form an essential part of the application documentation).
  • Create and link the relevant planning issues in Jira (rather than carry all the requirement detail around in Jira).  As stories are worked on in Jira, code (branches) is created that can be linked all the way back to Confluence.
  • As each branch is finished it is committed, Bamboo notices and will automatically rebuild.

Thus traceability is ensured throughout the entire lifecycle, documentation gets centralized in a documentation platform and relationships are clear.

Notionally, Testing could be supported by creating an issue type called “test case” with more functional information carried in Confluence.  These test cases could be excluded from the relevant scrum boards.  The problem with this is that a test case can be run over and over and in different environments.  Thus test in the traditional sense needs more attention.

Testing Solution Architecture Options and Background

So if we add testing into this mix – we essentially need some additional entities or objects.  Pure agile calls for a fully integrated testing approach but in reality testing applications will typically support the following functional model described below. Fundamentally, defects certainly need to end up back in the Jira backlog for teams to work on.  They need to be a part of the release management process also. When reviewing testing apps, how they deal with this matter is a major consideration.  Testing specialists will look at the elements below and consider how well supported they are and how manageable the various aspects of the overall process.

Given the needs of Test Plan, Test Cases and Test Execution, the question is how various apps tie these elements together.  To answer, we have to incorporate the above with the Atlassian overall model. This helps us truly appreciate how a testing app will work with the Atlassian stack.

In the model below we have three stakeholder perspectives:

1) The Developer/Scrum team functions.

2) The specialist testing view.

3) DevOps which lives in the middle (serves both of these previous functions).

As can be seen below manual testing is going to generate defects, but so obviously will DevOps initiated automated testing.

We appreciate that with agile best practices there should not really be a divide between the core Agile development process and testing but also it’s important to accept that for platforms and other major systems there are often many pre-existing test cases and automated testing scripts, especially for regression testing.

Any solution that enables visibility and reporting across this spectrum will be very advantageous because it will enable teams to determine where there are bottlenecks, full traceability, deployment speed and other key performance metrics.

It’s important to point out that the end result of any test is either a pass, or the creation of a Jira issue such as defect or bug.  This Jira issue creation forms the work items in the backlog of the development team.  Thus testing apps ultimately trigger the creation of defect issues that involve the larger whole of the project and team.  Because of the highly “intermingled” nature of development and testing we believe that any world class app needs to be “intermeshed” as much as possible with core Jira functionality.  This is in addition to the straight race to “feature richness”, as well as integration with build engines and other related tools.

Downloads & Review

There is one more additional dimension to our analysis and that’s perceived market share.  At the time of this review we are showing the relative number of downloads each product has below.  In our experience this is indicative of the popularity of the app.   We always encourage clients to inspect the marketplace download statistics – just to ensure that they appreciate the feedback and popularity of apps.  It’s our belief that Zephyr has been traditionally the market leader and so has dominated the space.  This doesn’t mean it’s the richest testing app in the market. Rather it’s the first mover and current market leader by volume.  There is so much feedback on these apps that it’s difficult to intuit anything from this data set. The results though are clear below – Zephyr is larger by 4* than Xray and Xray is 50% larger than TM4J.  We know that downloads are not sales, but in this case we’re assuming that Xray and TM4J convert similarly, while Zephyr has a relatively poorer close rate (nevertheless leaving them clearly the market leader).

Overall Results

The testing space is a complicated arena because there are three, very well qualified solutions, competing for your business.  Yet as will be seen they come at the challenge from varying directions which means that it’s possible to distill the true differences between them.  We feel that it’s important to reflect a number of key considerations in how you view these offerings:

1) Mesh with Jira – why? because we believe that testing is not an isolated activity and that teams should test together.  Jira has been designed to enable this and is richer for it. Isolated solutions may work in some contexts (complex reporting for instance) but not testing.

2) Feature richness – this is hard to argue with and is always a major consideration in this kind of report.

3) Market share.  Of course any first mover will achieve such an advantage but it’s important not to overlook what features or even lack of complexity might be driving this. Success cannot be overlooked.

As an FYI we reviewed pricing and did not consider it a material consideration.

Below is our Jira Testing app excellence matrix:

What does this mean to you the buyer?

We think that it’s important to consider your own context.

  1. A) If you are a smaller organization with say less than 100-200 IT staff and want a simpler application for testing – Zephyr could be a great choice.
  2. B) If you are a larger organization and testing is fairly compartmentalized and all the features of the app, being more “meshed” with Jira are not important, then look at TM4J.
  3. C) If you are a larger enterprise and looking for a full strength, highly configurable solution for testing – go for Xray.

It doesn’t surprise us that all three products are finding success, likely because of some of the factors identified above.

­Want more? We have more.

This is a short version of a longer research paper. To get the research version please use below opt-in to download.

Thank you for reading this blog. To accompany it we have some research pieces that substantiate our findings. This outlines each app, the way it interacts with Jira’s model, how it integrates and some pros and cons.

We’d be delighted to share with our clients.