|Friday, February 22|
|PS2 Poster Session 2 (with refreshments)||
Fri, Feb 22, 4:45 PM - 6:15 PM
Performance Testing of the Firefox Web Browser*Christina Choi, Mozilla
Joseph Kelly, Harvard University
Keywords: browser, internet, software, testing, analysis, design, multiple comparisons, hypothesis testing
At Mozilla, developers drive open web innovations by introducing new features that will enhance users’ experience of Firefox. Due to the large volume of code changes pushed to the code tree everyday, developers rely on automated in-house tests to identify whether a specific change in the code has introduced a bug; a regression. These tests are intended to measure key aspects of the browser such as its functionality, responsiveness, and performance; however, many of these tests were unable to detect significant regressions due to a lack of explicit test design. To improve this process, a group of statisticians have been working with engineers to integrate more systematic experimental design and analyses in Mozilla’s testing environment. This includes blocking on certain factors to reduce irrelevant variation, the implementation of principled statistical analysis to control for multiple hypothesis testing via a false discovery rate control method and the introduction of smoothing to remove the systematic variation present in the testing environment. These changes have increased the ability to detect regressions in the codebase and opened doors for more scientific testing schema at Mozilla.