Case Study
About Selenium
Selenium is a suite of open source tools for automating web browsers, primarily for testing web applications across different browsers and operating systems. At its core is a large polyglot repository, with over 200k lines of code written Java, C#, JavaScript, Rust, Ruby and Python, as well as hundreds of tests.
10x
Long CI runs waste developer time
As Selenium grew in usefulness and popularity, more and more scenarios needed to be tested across multiple web browsers (Chrome, Firefox…), browser versions (dev, beta…) and operating systems. This caused CI times in GitHub to quickly spiral out of control – it took just a few months to go from 45 minutes to over 5 hours. This led to developers waiting incredibly long times before finding out if their Pull Requests had passed all the tests, or before discovering new bugs.
Bazel and EngFlow to the rescue
This CI runtime crisis motivated the team to expand its Bazel usage and adopt EngFlow’s Remote Build Execution platform. Doing both of these things empowered Selenium with clean, transparent, functional and hermetic builds, where build steps and testing steps could be interwoven thanks to Bazel’s efficient build graph, as well as heavily cached and parallelized across many machines thanks to EngFlow’s enterprise-grade remote cache and execution service.
Initial surprises and quick wins
Switching the first tests to run on EngFlow was completely seamless: All it took was EngFlow providing a Remote Execution endpoint, and Selenium making a small configuration change for the first tests to be running remotely. The browser-based tests took a bit more effort, even with Bazel already managing browser and driver versions, because they required the ability to choose the Docker image the build actions were executed on – but with EngFlow's help this was also straightened out.
When running the entire Chrome test suite on EngFlow for the first time, many tests were immediately green out-of-the-box and completed 5x faster, which was a good place to be starting from. Running all the linters, which could take over 1 hour in GitHub, took just 5 minutes on EngFlow – that’s a 12x speed-up!
{ Faster, Simpler } — Choose two
EngFlow doesn’t just make builds and tests faster, it also lets you simplify them.
You can see this for yourself in EngFlow’s Selenium Build and Test UI:
For failed tests, developers get a nicely rendered view of which test cases failed and why, as well as everything they need to dig in. For example, in the figure above, the terminal output on the left prints out that a target failed and includes the path to a log file containing more details. In EngFlow’s UI, the failure reason is shown inline, as well as a summary of which of the 20 test cases failed.
EngFlow also helps you simplify complex, expensive and error-prone platform-specific setups. Builds and tests specify how each individual action was configured and run, which can be analyzed to fix problems and improve their performance.
Simon is the creator of Selenium WebDriver. For a long time, he led Google’s Browser Automation team, and he has also been the tech lead of Buck at Facebook. He is currently working as a software engineer for a large tech company.