The whopping increase in internet usage, mobile devices, and their availability with varied OS (Operating Systems) versions, are compelling organizations to build applications that are compatible with both web and mobile platforms. Thus, it becomes imperative to perform cross-browser and multi-platform testing for applications and have them tested across multiple web browsers, browser versions, OS, and mobile versions.

Thanks to test automation, a good chunk of this type of testing are now automated, thereby reducing the test execution cycle, and improving the product release frequency.

How is cross-browser/platform test automation done?

By using open source/commercial test automation tools along with necessary hardware around, test automation execution can be managed in the following ways:

1. Using local physical hardware: The desktops and mobile devices that are available locally at work location can be connected across a LAN to simulate a lab setup. Deploying test automation tools such as Selenium grid or master/slave concept of a few CI/CD tools can leverage this infrastructure setup to distribute the automated test execution.  Some of the pros and cons of using this method are:

Pros

1. 1. Better control of infrastructure
   2. Any unused desktops/mobile devices can be leveraged

Cons

1. 1. Lab maintenance overhead
   2. Need additional manpower
   3. Poor scalability

For most products during the initial stages, developers do entire testing, limiting the deployed environments to minimal. Testing could be done on a Dev environment and with distributed/outsourced test team, it becomes important to have a dedicated environment for testing. Now, depending on the merging model, introduce the test environments to conduct feature testing, manual, and automated regression testing.

2. Using cloud-based virtual machines: Tools like Selenium grid or any CI/CD test execution agents can be set up easily using virtual machines via a cloud service provider like AWS, Google Cloud, or Azure. These virtual machines can be configured to get the desired browser (version), simulators, and emulator combinations. These can also be integrated with the CI/CD tools and easily scaled as and when the need arises.

Pros

1. 1. Relatively lesser cost, when compared to purchasing and maintaining physical hardware

Cons

1. 1. Cost of cloud instances
   2. The additional overhead of managing the hardware requires additional technical expertise

Using cloud-based cross-browser testing/test automation tools as an alternative

While the above-mentioned two options are relatively advantageous over others, the maintenance overhead is a key problem. To address this limitation, several CBT (Cross-Browser Testing) vendors provide off-the-shelf solutions and additional capabilities to test automation developers and managers. Some of these include:

1. 1. Support Live, Automation, and Visual testing: The CBT tools allow users to interactively test on the live browsers and devices, support automation testing of varied frameworks, and provide visual testing.
   2. No Device lab: The CBT tools provide and maintain a wide range of OS with browsers and mobile simulators/emulators. These include the latest OS versions and patches, browser versions, and mobile models, thereby completely removing device-lab maintenance overhead.
   3. Real-device Testing: Since few of the CBT tools provide live devices on the cloud, users can perform real device testing at a minimal cost.
   4. Less-maintenance: The CBT tools are browser-based; hence the usage complexity is highly reduced. They also have a zero-set-up effort and zero maintenance.
   5. Dashboard: Most of the CBT tools provide dashboards containing test analytics, testing logs that allow users to view the test execution history, detailed logs, and screenshots.
   6. Configurations: These tools also provide support for local testing, parallel testing, debugging options, etc. through simple config parameter updates in the scripts.
   7. Framework support: The CBT tools also support a wide range of automation frameworks and languages.

Comparison analysis of different cross-browser/platform testing infrastructure vendors

Some of the tool vendors in the market provide similar capabilities around cross-browser/platform testing infrastructure. However, picking the right tool for testing can be a bit tricky. This is where we have come up with a quick comparison analysis for some of the prominent tools available in the market:

 

Criteria	Browserstack	LambdaTest	CrossBrowserTesting	SauceLabs
Speed of test execution	Relatively Fast	Relatively Slow	Relatively Slow	Fastest
Availability of required browser types, their versions, and mobile devices	Provides 2000+ browsers across mobile and desktops including physical devices support. [Learn more]	Provides 2000+ browsers across mobile and desktops. Limited access to physical devices through ‘Tiers’. [Learn more]	Provides 2050+ browsers across mobile and desktops including a wide range of physical devices. [Learn more]	Provides 2000+ browsers across mobile and desktops including physical devices support. [Learn more]
Headless browser execution support	No support	No support	Yes	Yes
Tool integrations	Moderate, provides support for integrating 44 tools/plugins. [Learn more]	Best, provides support for integrating with a maximum number ~86 of tools/plugins. [Learn more]	Moderate, supports integrations with ~21 tools/plugins. [Learn more]	Moderate, supports integrations with the most popular CI/CD tools/plugins. [Learn more]
Testing privately or locally hosted pages or run tests in developer environment**	Can be accomplished using “Browserstacklocal” binary. [Learn more]	Can be accomplished using Lambda Tunnel app. [Learn more]	Can be accomplished using an encrypted tunnel created between your machine and Cross Browser Testing. [Learn more]	Can be accomplished using Sauce Connect Proxy. [Learn more]
Reporting/Test result sharing	No Public URL is provided.	Test results can be shared by using: • Public URL [Learn more]	Test results can be shared by using: • Public link • Pre-configured email • Slack Channel [Learn more]	Test results can be shared by using: • Public link where access can be customized. [Learn more]
Monthly cost for automated testing for desktop and mobile (2 parallel tests and 10 users)	$499	$238	$312	$499
Geo-Location Testing	Yes – inbuilt support	Yes – inbuilt support	Limited – Ability to test localized content is limited to the availability of open HTTP Proxy	No inbuilt support. However, this can be achieved through WonderProxy [Learn more]
Technical Support	Fast – Provides a Live Chat option.	Fast – Provides a Live Chat option.	Slow – Takes a day or two to respond to queries via email.	Slow – Takes a day or two to respond to queries via email.

** Testing Privately or Locally Hosted Pages – Testing remote browsers and devices can access websites hosted on your internal networks which are otherwise not accessible over the public Internet.

Disclaimer: The views and opinions expressed in this article are personal and belong solely to the authors and do not necessarily to the author’s employer, organization, committee, or individual group. This analysis is performed purely based on the author’s personal usage experience with the tool vendors in a span of 3 years.