In our “Category Report” series of blog posts we break down individual categories and look into them in a bit more depth. Today we will be looking at Test Management. In MARS we categorize apps that manage, automatize, and generally aid in the testing process as Test Management.
Test Management is a very significant Atlassian Marketplace category Figure 1 shows that Testing is the 3rd fastest growing category in percentage terms with a remarkable growth of 11% this year.
Figure 1: Marketplace Overview
With Atlassian focusing on cloud and performance at scale on Data Center, how users leverage 3rd party apps is going to make a huge difference in terms of your Atlassian ROI. We’re here to help you achieve that ROI, today and tomorrow.
With the Atlassian Marketplace booming (instances have grown 99% from 2018 to 2021) customers are increasingly recognizing the incredible value that is available from using 3rd party apps. However, with nearly 4,000 available apps and 1.6M instances, navigating it can be difficult.
At Blended Perspectives our Marketplace Analytic Research Service (MARS) means that we are uniquely situated to help you with this. MARS is our database containing over 2.5 years’ worth of in-depth data on the Atlassian Marketplace. We collect data covering all facets of the Marketplace including instances, growth, product, platform, license, price, reviews, and many more. Perhaps most interestingly, we have also carefully categorized all apps with over 500 instances (the 500 Club) which represent 83% of all Marketplace instances.
Test Management’s growth is largely being driven by the two dominant players in the Testing category–Zephyr (made by SmartBear) and Xray (made by Xblend). Although there are a number of decent free Test Management apps, as shown by Figure 2 (in particular QAlity which has seen astonishing growth of nearly 80% this year), for a truly all encompassing, feature heavy testing app customers have to look at the paid options and in particular Zephyr and Xray.
Figure 2: Top Free Test Management Apps
We can see from Figure 3 that most customers do opt for a paid Test Management app with 91% of all Testing instances being on paid apps. This is much higher than for the marketplace as a whole where 68% of instances take place on free apps.
Figure 3: Paid vs Free Test Management Breakdown
In terms of the offerings from Zephyr and Xray you have their two signature apps of Zephyr Squad and Xray Test Management as well as Zephyr Scale (formerly known as TM4J). As Figure 4 shows, Zephyr Squad is a more popular app than Xray with 20,000 instances vs just under 15,000, however, Xray is growing much faster with growth of 16% this year vs 11% for Zephyr. Over the longer term, Xray has also grown significantly faster than Zephyr: 270% vs 47% since 2018.
Zephyr’s takeover of TM4J in March 2020 was definitely the biggest news in testing last year. The move was an interesting one and could be good for both products as synergies arise, in particular for TM4J (Zephyr Scale) as it joins a team experienced at innovating within the testing space. However, while customers will be sure to benefit from product improvements, TM4J customers may feel more pressure towards SmartBear’s more expensive enterprise platform.
Figure 4: Top Paid Test Management Apps
In terms of the usability of these three apps, we believe that Xray is your best bet. We believe that Xray’s faster growth is justified by excellent integration capabilities, advanced email notifications, a very strong test case repository, and the fact that it is the most feature rich Jira based app for testing. On top of this, Xray is significantly cheaper than both Zephyr offerings. On Data Center Xray is significantly cheaper at every price point while on Cloud it is moderately cheaper at every price point.
This is not to say that the Zephyr apps are not quality applications. Zephyr Squad has a native importer, is the fastest app to set up and configure, has the best most intuitive interface, and it is easy to create new tests or defects from the UI with the correct issue links in one step. Meanwhile, Zephyr Scale has a rich feature offering, solid reporting & a variety of gadgets, while self containment can simplify administration.
However, Zephyr Squad is relatively weak on preconditions for test cases, is enabled globally rather than at the project level, is not strong on email notifications, and does not appear to be a fully featured test case repository as per Xray. Meanwhile, Zephyr Scale suffers from segregation of testing from other functions, the fact that other third party apps enabled in the Jira installation can’t be used without specific integrations as there are no issue types, and a lack of native code and build integrations. This is all on top of Xray being a more cost effective option over these two apps.
Zephyr Squad and Zephyr Scale definitely have their place. For smaller organizations that want a more simple Test Management app then Zephyr Squad is a great choice. Likewise, for larger organizations where testing is fairly “compartmentalized” and you want less Jira administration and don’t have a significant need for integrations/automated testing then Zephyr Scale is also a great choice. However, as an overall Testing application we believe that Xray is the strongest, most configurable solution available to Atlassian users today.
If you want to hear more from us on Test Management then check out our webinar and if you have any questions regarding MARS then get in touch with us at hello@blendedperspectives.com.