Research Director, Agile ALM, Quality & Portfolio Strategies
Leaders
Micro FocusFeatured Vendor
IBM
SmartBear
Perforce
Tricentis
BMC
Broadcom
HCL
Major Players
Microsoft
Veracode
Keysight
Synopsys
Parasoft
Sauce Labs
GitLab
Worksoft
digital.ai
UiPath
Contenders
Checkmarx
NTT Security AppSec Solutions
SonarSource
Contrast Security
Applitools
Participants
HeadSpin
High-quality, adaptive, resilient, and secure software delivered with velocity enabled our dynamic shift to digitization, hybrid work, and business innovation and execution over the past years during the COVID-19 pandemic, and now moving into 2022–2023 and beyond. Automation played and continues to play a key role — developers cite lack of automation as the primary inhibiter to productivity in IDC’s PaaSView and the Developer survey research (of around 1,000 participants worldwide who responded to that question). Continuous testing processes with sufficient functionality across core areas of increasingly complex deployments enable DevOps and faster releases — no business is able to afford the costs of releasing irrelevant, broken, poorly performing, and insecure code more quickly. Compressed time frames demand test automation and rapid transitions to continuous testing.
For that reason, this enterprise ASQ IDC MarketScape encompasses high-end portfolios with deep and broad ASQ capabilities. We also see a shortage of skilled, professional developers who benefit increasingly from low-code and no-code development and the emergence of “citizen developers” (who also have visibility into core business needs, but little or no experience with testing). Automation in this context fuels digital transformation. We see organizations trending toward machine learning (ML) and artificial intelligence (AI) as well as seeing intelligent process automation (IPA) and robotic process automation (RPA) expanding beyond business areas (such as HR) into IT and software quality initiatives into testing and software quality as well as a longer-term commitment to intelligent analytics with ML and AI.
So the requirement to deploy secure applications across mixed environments — from web to mobile to cloud to embedded at agile speeds with complex global sourcing and demand for collaborative hybrid work for business and big data analytics — necessitate a variegated approach to quality and to software deployment. For this reason, IDC’s evaluation criteria for enterprise automated software quality incorporate breadth and depth across areas for combined ASQ capabilities with a continuous, “end to end” application life-cycle approach through deployment for DevOps. For enterprises, it is neither enough nor effective to leverage testing phases, capabilities, and delivery platforms in isolation. This study — and three related IDC MarketScape documents for ASQ assessments completed in coordination with one another — examines ASQ adoption patterns and trends and how they are impacting business success and ASQ solution availability from high-end providers and from smaller, innovative testing and ASQ vendors. Specifically, key strategy measures for enterprise ASQ success include the following:
Relevant, high-quality, dynamically performing, adaptive, and secure software drives business innovation and success now more than ever. Application quality remains challenging for companies as they continue to deal with an unpredictable and challenging global economy with constrained and erratically available resources at the same time, dramatically increased software deployment, and technology complexity with faster iterative release time frames. According to our most recent DevOps research (1Q22), nearly all (90.5%) organizations surveyed are releasing features with a lead time of a month or less, an increase of 26 percentage points from 2020 (see U.S. Accelerated Application Delivery Survey, IDC #US47924622, January 2022). In addition, organizations delivering features in 1-2 weeks doubled from 2020 to 2021. This significantly compresses time frames for quality and testing, demanding a commensurate shift to agile and adaptive testing processes. Also in that context, organizations must incorporate application security testing and code analytics, coordinating design and quality strategies to encompass software resilience in the face of increasingly virulent attacks and risks (with high-speed deployments).
Effective strategies, decision making, and software quality execution mean the difference between corporate flexibility, evolution, and success — or in other cases, software failure (including negative consequences to transactions, corporate reputation and, potentially, entire businesses). There is little leeway for poor software quality as companies deal with a volatile economic, business, and political climate that remains erratic with ongoing consequences from the pandemic, along with increasingly complex sourcing and deployments (from legacy to mobile to Internet of Things [IoT] platforms) with less internal development and quality assurance (QA) resources than were in place 18–24 months ago.
With volatile international economic dynamics and other factors (including corporate responsibility), risk, trust, and compliance initiatives are growing in scope and impact and must be met by effective quality approaches in conjunction with requirements and change management/auditing. Modern development is engaging organizations in microservices and codeless development while still often requiring coordination with system of record on legacy platforms (even as those with knowledge about how to access and maintain those platforms are retiring in increasing numbers).
Effective strategies and decision making for quality assurance, value, risk, and quality control remain a matter of survival in the current dynamic technology, international economy, and regulatory climate (as emerging technologies and adaptive software drive global businesses).
In addition, IDC has seen ongoing investments in complex sourcing for software development projects. This continues the existing trend for combining internal IT resources with contractors, both onshore and offshore providers, as well as significant increases in use of open source. With declining internal resources, as companies have been engaged with service providers, the demand for effective software project collaboration and coordination has increased geometrically.
Software quality automation — using tools with appropriate capabilities and process support, combined with effective organizational strategies — is more important than ever for successful software implementations. It is in part due to this increase in multisource software projects that IDC has chosen to focus on combined ASQ portfolio suites for enterprise ASQ, incorporating emerging technologies and trends (mobile and digital innovation, cloud, and quality and continuous testing related to DevOps and DevSecOps), digital transformation/digital quality, quality/security, and cloud testing. ASQ solutions in this context can give a basis for successful collaboration and metrics, where communication between far-flung, hybrid resources and communities across multiple corporate and global cultures and use of open source is a necessity (and where long-term deployment and service management assessments and costs are a core factor of the end-to-end software DevOps life cycle). We also see increasing use of open source software (OSS), which brings both benefits and risks that bring demand focus, management and associated opportunities for both reuse and remediation.
Yet implementation of these products remains challenging from a behavioral and organizational perspective and costly for enterprise adoption. Human beings are wired more for consistency than for change. For companies to succeed with ASQ, user buy-in and consistent adoption are key, which demand process and behavioral changes. Dated, inadequate, or nonexistent project quality information and poor testing data torpedo successful software quality execution.
We see companies approaching this in a variety of ways, based on their needs and maturity. Lightweight ASQ solutions can be more quickly adopted for small and medium-sized businesses (SMBs). Targeted ASQ capabilities enable focused adoption for core areas (e.g., static and/or dynamic analysis, memory leak detection, and agile testing). Global enterprise organizations tend to require high-end, functionally broad, and rich ASQ products. In both contexts, we see many users evaluating and opting for flexible delivery models for adaptive infrastructure access for testing in the cloud and to enable faster adoption (with on-demand hosted and ASQ SaaS). Moreover, we see enterprise solutions seeking to “shift left” to become more agile and to enable modern development.
Our most recent DevOps survey research also shows that a majority of DevOps teams (at 72.5%) have increased engagement with advanced software testing and quality strategies such as use of AI and ML, continuous testing as part of continuous deployments, and leveraging communities of practice, although the remainder continue to evolve and/or deal with a mix of manual testing and sporadic use of test automation (see Figure 2).
Q. What best describes your organization’s approach to software quality?
n = 200
Source: IDC’s Accelerated App Delivery Survey, August 2021
The purpose of this ASQ study and the other three companion IDC MarketScape documents is to enable context for purchase decisions in those areas of the ASQ market where we see most activity and a need for guidance. Adoption must be accompanied with organizational and process changes, and we consistently heard as we spoke with the 70 or so references as part of this study that continuous testing, agile approaches, and continuous integration to enable quality software as part of an end-to-end DevOps strategy were top of mind as part of automation decisions.
This section explains IDC’s key observations resulting in a vendor’s position in the IDC MarketScape. While every vendor is evaluated against each of the criteria outlined in the Appendix, the description here provides a summary of each vendor’s strengths and challenges.
Micro Focus is positioned in the Leaders category in the 2022 IDC MarketScape for worldwide enterprise ASQ and continuous testing for digital execution.
Micro Focus products considered as part of this vendor assessment in this research initiative include ALM Octane, ALM/Quality Center, LoadRunner family (Professional, Enterprise, Cloud, and Developer), UFT family (One, Developer, and Mobile), Enterprise Test Server, Service Virtualization, Fortify Static Code Analyzer, Fortify WebInspect, Fortify on Demand, and Enterprise Developer.
Micro Focus was founded in 1976 and is based in Newbury, England. Currently, Micro Focus employs over 12,000 people across 48 countries, serving more than 15,000 customers across its ASQ and other products. The company has grown primarily by acquisition, with one of the most impactful for its ASQ portfolio being the merger with HPE’s software business segment in September 2017. This merger brought Micro Focus several of its current primary ASQ (and DevOps) products, including ALM Octane and Quality Center, UFT, LoadRunner, and Service Virtualization. Since 2018, Micro Focus has worked to streamline its ASQ product ecosystem, reshaping its portfolio from 35 distinct products to 8 marketed products/families in 2021. Micro Focus is well positioned to enable enterprise ASQ due to its products’ broad and deep testing capabilities supported by tools for PPM, ALM, mainframe modernization, and value stream management. This research initiative also coordinates software quality analysis with application security testing. We increasingly observe the need to bring these teams together (along with those architecting applications to design for greater resilience). Most recently, in 1Q22, Micro Focus’ security business unit CyberRes acquired the cloud-native SCA provider Debricked to augment its existing security capabilities and enable users to better assess, leverage, and secure open source software (OSS) components.
Micro Focus has two main product groups for quality testing: the LoadRunner family and the UFT family. LoadRunner provides the ability to create, define, debug, and execute load tests at scale. LoadRunner supports testing of over 50 technologies and application environments, can scale to as many as 5 million users, and integrates with IDEs to shift testing left via LoadRunner Developer. The product is available in several versions: Professional for colocated teams, Enterprise for globally distributed teams, and Cloud for running performance tests without deploying and managing infrastructure. The UFT products offer functional testing and automation for more than 200 GUI and API technologies and centralize the testing architecture layers, including presentation, business logic, database, components, services, and APIs. UFT One provides functional testing with one tool across web, mobile, API, and enterprise applications, as well as AI-powered test automation. UFT Developer is designed for continuous testing and continuous integration by delivering GUI testing capabilities within the IDE of choice. UFT Mobile supports mobile device testing by providing a lab and management via on-premises devices, hosted remotely or emulated. Finally, UFT Sprinter’s goal is to enable efficient manual testing through capabilities such as data injection, automated defect scanners, and autogenerated documentation. Micro Focus’ Service Virtualization product supports all these testing capabilities, providing a single solution for creating virtual services across unit, functional, and performance testing.
In addition to the LoadRunner family, Micro Focus also provides functional testing for IBM mainframe applications in Enterprise Test Server. The product leverages virtualized hardware on premises or in the cloud to reduce costs and to help scale test capacity. Enterprise Test Server can be aligned with other Micro Focus solutions to provide pipeline execution, extending the ability to automate testing processes for mainframe applications.
Micro Focus’ Fortify family of products offers security testing including SAST, DAST, IAST, and SCA. Fortify Static Code Analyzer analyzes source code, bytecode, or binaries and offers broad language coverage across 27 languages. Fortify WebInspect provides DAST via crawling and auditing along with guided scans, templates, and reports. The product is supported by Fortify’s Software Security Research (SSR) team for quarterly security check/rulepack content. WebInspect currently covers over 3,400 checks, including new updates for XPath Injection and Server-Side Template Injection. IAST is offered via Fortify on Demand, Micro Focus’ application security as a service. An active, rather than passive, IAST, the solution leverages WebInspect’s crawling and auditing capabilities to provide both server-side and client-side vulnerability detection. In addition, Fortify on Demand offers discovery scans for identifying web applications and APIs and the option to have test results reviewed by Micro Focus’ security experts. SCA is also offered via partnership through Fortify on Demand, a capability powered by an OEM relationship with Sonatype. This SCA capability is enabled by a command line–based build tool that combines both Fortify and Sonatype scanning steps into one command that can be integrated with any build pipeline for automation purposes. Finally, Fortify provides on-premises, public cloud, and SaaS offerings, with testing services layered on as an option.
Micro Focus synchronizes both quality and security testing capabilities with agile and DevOps processes via ALM Octane and ALM/Quality Center. The products act as a central hub, unifying and calibrating testing efforts, providing dashboards for analytics and insights, as well as release governance capabilities.
Micro Focus is differentiated by its broad and deep support, with testing capabilities from mainframes to containers, and across OSs, browsers, devices, protocols, and GUI frameworks. For example, the UFT family of products allows for the automated testing of around 200 GUI and API technologies across mobile, web, desktop, and mainframe. Another differentiator for Micro Focus is the ability to scale implementations from small teams to distributed, global organizations. Micro Focus’ licensing and consumption model is flexible and includes SaaS, public cloud, or on premises and is supported by enterprise credential support and flexible implementation configuration. For example, LoadRunner Cloud scales leveraging Kubernetes and Docker containers, and many components of Micro Focus’ ASQ portfolio are multitenant. Finally, Micro Focus is built on a composable and open service architecture that enables integration between Micro Focus products, open source, and third-party software. Further, Micro Focus enables traceability and analyzability with a data lake that collates data across the integrated tools.
References with whom IDC spoke experienced an increase in velocity using Micro Focus ASQ products. One customer cited application release frequency went from 5 weeks to 12 days. Another reference noted that manual testing of one of the company’s forms would take five minutes manually, and with Micro Focus products, its team could process 20–30 forms in the same amount of time. Customers echoed that Micro Focus products freed up resources to allow testers to focus on more complex scenarios and exploratory testing.
Micro Focus’ strengths lie in the formidable technical breadth and depth of the company’s portfolio. Micro Focus’ ASQ offerings provide a comprehensive solution that spans the software development and testing life cycle from ideation to production and includes synergistic capabilities such as portfolio management; application governance and analytics; functional, performance, and security testing; and release management. The company’s strengths also lie in a user base, which Micro Focus can target with its modernizing product portfolio, looking to the organization’s products to move forward into agile DevOps and modern development. These combined factors contribute to Micro Focus’ position in the market.
As a long-term, established ASQ provider with a complex portfolio and established customer base, one of Micro Focus’ challenges is how to innovate and adapt to dynamically changing technology, business, organizational, and process needs while bringing along both its product line and people. While users change behavior with reluctance, smaller, innovative, and emerging ASQ companies target Micro Focus, resulting in some customer attrition. As a result, Micro Focus must evolve and optimize product capabilities, improve user interface design, target modern development, and continue to innovate on flexible pricing and adaptive sales approaches. In addition, the company has seen leadership turn over recently, which can hamper execution. At the same time, Micro Focus has made significant strides in honing its portfolio through a combination of pragmatic and astute consolidation, product retirement, and capability merging with simplified naming and packaging approaches. But the challenge remains to effectively articulate the range and scale of the company’s product capabilities. And despite the breadth of its portfolio, there are emerging capabilities that Micro Focus would benefit from adding (e.g., IaC security in Fortify).
u003cp class=u0022p1u0022u003eIDC structured its approach to inclusion of vendors in the enterprise ASQ category based on the strength of their products’ ASQ capabilities and strategy, revenue share in part (as indicators of adoption and staying power), and/or differentiated position and capabilities in rapidly emerging markets of concern. IDC evaluated 24 vendors for inclusion in the 2022 IDC MarketScape for enterprise ASQ (up from 19 vendors in our previous assessment, several of which were acquired).u003c/pu003e
u003cp class=u0022p1u0022u003eCriteria for inclusion include the following:u003c/pu003e
u003cul class=u0022ul1u0022u003e
tu003cli class=u0022li2u0022u003eSufficient automated software quality capabilities available in key areas of concern for IDC clients and/or related innovative positioning (e.g., test management, test data management, performance and functional testing, code analytics, service virtualization, agile and continuous testing, mobile testing, digital quality, application security testing [AST] and software composition analysis [SCA], and/or mainframe and business application testing)u003c/liu003e
tu003cli class=u0022li2u0022u003eAppeared in IDC discussions with end-user clients as part of RFP and other inquiry for ASQ during 2021–2022u003c/liu003e
tu003cli class=u0022li3u0022u003eA minimum overall revenue of $20 million for CY21, with at least $10 million of that revenue from ASQu003c/liu003e
u003c/ulu003e
u003cp class=u0022p1u0022u003eSmaller, targeted vendors with engaging functionality and focus were also included in this study to provide context for emerging areas of importance (even if they do not have a full portfolio of enterprise capabilities). Vendors evaluated in this IDC MarketScape are Applitools, BMC, Broadcom, Checkmarx, Contrast Security, digital.ai, GitLab, HCL, HeadSpin, IBM, Keysight (formerly Eggplant), Micro Focus, Microsoft, NTT Security AppSec Solutions (formerly WhiteHat Security), Parasoft, Perforce, Sauce Labs, SmartBear, SonarSource, Synopsys, Tricentis, UiPath, Veracode, and Worksoft.u003c/pu003e
u003cp class=u0022p1u0022u003eThe focus for customers is on ASQ breadth and depth; scalability, coordination with end-to-end DevOps life-cycle management, and strong data analytics as well as process support for systemic adoption and engagement have been key drivers for leadership. At the same time, with the rapid increase of low-code and no-code development and microservices, key emerging trends for enterprise ASQ are increased demand for intuitive and modern testing approaches, the ability to test APIs (as the underlying integration layer), emerging support of improved analytics with ML and AI, and early leverage of robotic process automation as part of testing. Also essential is the role of application security testing as part of DevSecOps. So combined approaches in this context for enterprise ASQ are key. Emerging support for digital quality and u0022edgeu0022 platforms as well as availability and support of the cloud and cloud applications are also a part of our assessment. So this IDC MarketScape for worldwide enterprise ASQ and continuous testing for digital execution includes all vendors with capabilities across the areas for which IDC provides in-depth analysis in the other three vendor assessments.u003c/pu003e
u003cp class=u0022p1u0022u003eFor the purposes of this analysis, IDC divided potential key measures for success into two primary categories: capabilities and strategies.u003cspan class=u0022Apple-converted-spaceu0022u003e u003c/spanu003eu003c/pu003e
u003cp class=u0022p1u0022u003ePositioning on the y-axis reflects the vendor’s current capabilities and menu of services and how well aligned the vendor is to customer needs. The capabilities category focuses on the capabilities of the company and product today, here and now. Under this category, IDC analysts will look at how well a vendor is building/delivering capabilities that enable it to execute its chosen strategy in the market.u003c/pu003e
u003cp class=u0022p1u0022u003ePositioning on the x-axis, or strategies axis, indicates how well the vendor’s future strategy aligns with what customers will require in three to five years. The strategies category focuses on high-level decisions and underlying assumptions about offerings, customer segments, and business and go-to-market plans for the next three to five years.u003c/pu003e
u003cp class=u0022p1u0022u003eThe size of the individual vendor markers in the IDC MarketScape represents the market share of each individual vendor within the specific market segment being assessed.u003cspan class=u0022Apple-converted-spaceu0022u003e u003c/spanu003eu003c/pu003e
u003cp class=u0022p1u0022u003eKey trends driving user adoption include the urgent need to u0022shift leftu0022 with continuous testing by applying agile approaches to quality practices to avert the expense and reputational damage of finding code problems in production. We also increasingly see coordination between quality and security testing — the opportunity to design and architect up front for resilience and the ability to leverage emerging AI and ML capabilities to evolve better quality hygiene, processes, and workflows to improve code quality at a time of urgent demand for fast deployments. The emergence of multimodal platforms beyond mobile, including mixed reality and IoT, will continue to drive opportunities for creativity on the part of test automation vendors and broader engagement. Along with the democratization of development with no-code/low-code approaches, there is a commensurate demand to incorporate testing into these environments so that nonprofessional developers can create better quality code. Leverage of machine learning and artificial intelligence in that context becomes increasingly important. The rise of APIs as the enabling technology for services development in these environments and of RPA brings additional opportunities to automation of mundane repetitive tasks and to help ensure the quality of these enabling technologies. For these reasons, in part, we are bullish about the ASQ adoption and coordination with key organizational and process change for DevOps and DevSecOps moving forward.u003cspan class=u0022Apple-converted-spaceu0022u003e u003c/spanu003eu003c/pu003e
u003cp class=u0022p1u0022u003eIDC MarketScape criteria selection, weightings, and vendor scores represent well-researched IDC judgment about the market and specific vendors. IDC analysts tailor the range of standard characteristics by which vendors are measured through structured discussions, surveys, and interviews with market leaders, participants, and end users. Market weightings are based on user interviews, buyer surveys, and the input of IDC experts in each market. IDC analysts base individual vendor scores, and ultimately vendor positions on the IDC MarketScape, on detailed surveys and interviews with the vendors, publicly available information, and end-user experiences in an effort to provide an accurate and consistent assessment of each vendor’s characteristics, behavior, and capability.u003c/pu003e
u003cp class=u0022p1u0022u003eAutomated software quality (ASQ) tools support software unit testing, system testing, user integration testing, and software quality assurance. Functions such as test specification, generation, execution, results analysis, u0022bug tracking,u0022 test data and QA management, functional/regression, and stress and load testing are included in this category. ASQ SaaS and testing in the cloud and of cloud applications (private, public, and hybrid), virtual test lab management, and service virtualization as well as software quality analysis and measurement are included in this category. Emerging platform and software testing support for mobile, video, crowdsourcing, end-user experience, embedded software quality, and other areas (e.g., enterprise resource planning and mainframe) are also included in this category. The software quality analysis and measurement aspect of ASQ consists of software tools that enable organizations to observe, measure, and evaluate software complexity, size, productivity, and risk. Examples of capabilities provided by these software analysis tools include architectural assessment of design consequences (on software performance, stability, adaptability, and maintainability), static analysis, dynamic analysis, and quality metrics for complexity, size, risk, and productivity to establish baselines and to help judge project progress and resource capabilities.u003c/pu003e
u003cul class=u0022ul1u0022u003e
tu003cli class=u0022li1u0022u003eIDC MarketScape: Worldwide Application Security Testing, Code Analytics, and Software Composition Analysis 2022 Vendor Assessment – Coordinating Security and Quality for Resilience and DevSecOps (IDC #US47097521, forthcoming)u003c/liu003e
tu003cli class=u0022li1u0022u003eIDC MarketScape: Worldwide Cloud Testing 2022 Vendor Assessment u003cspan class=u0022s2u0022u003eu003ciu003e—u003c/iu003eu003c/spanu003e Empowering Velocity (IDC #US47097221, March 2022)u003c/liu003e
tu003cli class=u0022li1u0022u003eIDC MarketScape: Worldwide Mobile Testing and Digital Quality 2022 Vendor Assessment u003cspan class=u0022s2u0022u003eu003ciu003e—u003c/iu003eu003c/spanu003e Enabling Multimodal Dynamism for Digital Innovation (IDC #US47097321, forthcoming)u003c/liu003e
tu003cli class=u0022li1u0022u003eAddressing Quality for Customer Communications Infrastructure and Engagement: Strategies to Manage Complexity and Optimize the Business (IDC #US48467821, December 2021)u003c/liu003e
tu003cli class=u0022li1u0022u003eMarket Analysis Perspective: Worldwide Agile ALM, Quality, and Portfolio Strategies, 2021 – Dynamic Coordination for Quality DevOps Delivery (IDC #US47148721, September 2021)u003c/liu003e
tu003cli class=u0022li1u0022u003eIDC PeerScape: Practices for Enabling Innovation with Agile Quality, Continuous Testing and Modern Development (IDC #US47917321, June 2021)u003c/liu003e
tu003cli class=u0022li1u0022u003eWorldwide Automated Software Quality Forecast, 2021-2025: Digital Transformation Demands Quality and Drives Growth (IDC #US47916121, June 2021)u003c/liu003e
tu003cli class=u0022li1u0022u003eWorldwide Automated Software Quality Market Shares, 2020: Rapid Digitization Impels Ongoing Quality Adoption (IDC #US47916021, June 2021)u003c/liu003e
tu003cli class=u0022li1u0022u003eIDC’s Worldwide Software Taxonomy, 2021 (IDC #US47588620, April 2021)u003c/liu003e
u003c/ulu003e
