Speeding Up Automated Testing With GUI Rendering Inference

Abstract

   Increasingly complex mobile applications require better and more accurate methods of automatedvGUI testing. Traditional testing frameworks relying on fixed delays or pixel-based image comparisonvmethods, such as SSIM, have a lot of limitations. Most of the time, these methods misclassifyvGUI rendering states, which leads to false positives and inefficient testing workflows. This thesis,therefore, proposes a new approach to these issues by the inference of the rendering state of GUIs using deep learning-based models. Drawing on large-scale pre-trained image classification models like Vision Mamba, Vision Transformers, and ResNet, it enables the accurate classification between fully rendered and partially rendered GUIs. It does this through a fine-tuning process of a large-model on a subset of RICO, a very large repository of GUI images from truly diverse mobile applications. It also involves more sophisticated

model-training techniques to ensure that the best is gotten: transfer learning and other effective optimization techniques. Instead, this system architecture’s semantic examination of GUI elements goes deep into more meaningful matches of context and visual information well beyond anything at the level of pixels. The efficiency and accuracy of GUI testing will increase significantly with the proposed approach. That is different from fixed throttles that introduce unnecessary delays: it guarantees automated tests execute on fully rendered GUIs, which, in turn, reduces false positives and test re-runs. With this enhancement, development teams will save precious time and computational resources; this turns out to speed up the process at lower costs. Deep learning-based classification introduces a dynamic intelligent system that changes according to various GUI rendering scenarios; therefore, it is also more flexible and robust than what is already available. The contributions of this thesis are three-fold: it proposes a new deep learning-based approach for inferring GUI rendering states, develops a high-quality dataset to support fine-tuning models, and performs an in-depth comparative analysis of large image classification models for GUI testing.Ultimately, this study provides a very solid foundation for further research into the rendering state classification in GUI testing.