This alert has been successfully added and will be sent to: You will be notified whenever a record that you have chosen has been cited.
To manage your alert preferences, click on the button below. Manage my AlertsGraphical User Interface (GUI) is unavoidable in modern software apps. It facilitates the interactions between the users and the apps. As shown on the Google play store, some apps with higher downloads often have higher-quality, well-designed and tested GUI. GUI testing has become a necessary step in the app development process, and related research become a hot spot in recent years. However, there isn’t a review about GUI testing of mobile apps, which brings obstacles to new researchers. In this paper, we systematically review publications between 2010 and 2020, to gain an insight into GUI testing for mobile apps. Even though the earliest research was published around 1997 but we believe the considered years are likely to include the advances in the field. Specifically, the paper aims to identify (i) the main objectives of GUI testing, (ii) the approaches applied (iii) the evaluation metrics (iv) the challenges and future research directions. To cover all relevant literature, following a predefined systematic literature review procedure, involving both the automatic and manual search strategies, we found 75 primary studies. Four research questions are proposed to analyze them. We found that functionality is the main objective of GUI testing. Model-based testing is the most common approach. Metrics such as error detection, execution time, and code coverage are often used to evaluate the performance of GUI testing techniques. Finally, we outline some key challenges as well as possible research directions. We believe our work would provide a clue for new researchers as well as more research in GUI testing.
David Adamo, Md Khorrom Khan, Sreedevi Koppula, and Renée Bryce. 2018. Reinforcement learning for Android GUI testing. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. ACM, 2–8.
Emil Alégroth, Robert Feldt, and Pirjo Kolström. 2016. Maintenance of automated test suites in industry: An empirical study on Visual GUI Testing. Information and Software Technology 73 (2016), 66–80.
Amira Ali, Huda Amin Maghawry, and Nagwa Badr. 2018. Automated parallel GUI testing as a service for mobile applications. Journal of Software: Evolution and Process 30, 10 (2018), e1963.
Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Salvatore De Carmine, and Atif M Memon. 2012. Using GUI ripping for automated testing of Android applications. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. ACM, 258–261.
Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Bryan Dzung Ta, and Atif M Memon. 2014. MobiGUITAR: Automated model-based testing of mobile apps. IEEE software 32, 5 (2014), 53–59.
Saswat Anand, Mayur Naik, Mary Jean Harrold, and Hongseok Yang. 2012. Automated concolic testing of smartphone apps. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. ACM, 59.
Luca Ardito, Riccardo Coppola, Marco Torchiano, and Emil Alégroth. 2018. Towards automated translation between generations of GUI-based tests for mobile devices. In Companion Proceedings for the ISSTA/ECOOP 2018 Workshops. ACM, 46–53.
Yauhen Leanidavich Arnatovich and Lipo Wang. 2018. A systematic literature review of automated techniques for functional gui testing of mobile applications. arXiv preprint arXiv:1812.11470(2018).
Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of android apps. In Acm Sigplan Notices, Vol. 48. ACM, 641–660.
Gigon Bae, Gregg Rothermel, and Doo-Hwan Bae. 2014. Comparing model-based and dynamic event-extraction based GUI testing techniques: An empirical study. Journal of Systems and Software 97 (2014), 15–46.
Young-Min Baek and Doo-Hwan Bae. 2016. Automated model-based Android GUI testing using multi-level GUI comparison criteria. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. ACM, 238–249.
Carlos Bernal-Cárdenas, Kevin Moran, Michele Tufano, Zichang Liu, Linyong Nan, Zhehan Shi, and Denys Poshyvanyk. 2019. Guigle: a GUI search engine for Android apps. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 71–74.
Nataniel P Borges, Maria Gómez, and Andreas Zeller. 2018. Guiding app testing with mined interaction models. In 2018 IEEE/ACM 5th International Conference on Mobile Software Engineering and Systems (MOBILESoft). IEEE, 133–143.
Nataniel P Borges, Jenny Hotzkow, and Andreas Zeller. 2018. DroidMate-2: a platform for Android test generation. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 916–919.
Nataniel P Borges, Jenny Rau, and Andreas Zeller. 2020. Speeding up GUI Testing by On-Device Test Generation. In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1340–1343.
Pearl Brereton, Barbara A Kitchenham, David Budgen, Mark Turner, and Mohamed Khalil. 2007. Lessons from applying the systematic literature review process within the software engineering domain. Journal of systems and software 80, 4 (2007), 571–583.
Alexandre Canny, Philippe Palanque, and David Navarre. 2020. Model-Based Testing of GUI Applications Featuring Dynamic Instanciation of Widgets. In 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 95–104.
Yuzhong Cao, Guoquan Wu, Wei Chen, and Jun Wei. 2018. CrawlDroid: Effective Model-based GUI Testing of Android Apps. In Proceedings of the Tenth Asia-Pacific Symposium on Internetware. ACM, 19.
Neha Chaudhary and O Sangwan. 2016. Metrics for event driven software. PhD. Scholar of Gautama Buddha University, India (IJACSA) International Journal of Advanced Computer Science and Applications 7, 1(2016).
Lin Cheng, Zijiang Yang, and Chao Wang. 2017. Systematic reduction of GUI test sequences. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 849–860.
Wontae Choi, George Necula, and Koushik Sen. 2013. Guided gui testing of android apps with minimal restart and approximate learning. In Acm Sigplan Notices, Vol. 48. ACM, 623–640.
Wontae Choi, Koushik Sen, George Necula, and Wenyu Wang. 2018. DetReduce: minimizing Android GUI test suites for regression testing. In Proceedings of the 40th International Conference on Software Engineering. ACM, 445–455.
Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated test input generation for android: Are we there yet?(e). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 429–440.
Edward T-H Chu and Jun-Yan Lin. 2018. Automated GUI Testing for Android News Applications. In 2018 International Symposium on Computer, Consumer and Control (IS3C). IEEE, 14–17.
Lazaro Clapp, Osbert Bastani, Saswat Anand, and Alex Aiken. 2016. Minimizing GUI event traces. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 422–434.
Riccardo Coppola, Emanuele Raffero, and Marco Torchiano. 2016. Automated mobile UI test fragility: an exploratory assessment study on Android. In Proceedings of the 2nd International Workshop on User Interface Test Automation. ACM, 11–20.
Zhen Dong, Marcel Böhme, Lucia Cojocaru, and Abhik Roychoudhury. 2020. Time-travel testing of Android apps. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 481–492.
Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In Proceedings of the 41st International Conference on Software Engineering. IEEE Press, 269–280.
Shuai Hao, Bin Liu, Suman Nath, William G J Halfond, and Ramesh Govindan. 2014. PUMA: programmable UI-automation for large-scale dynamic analysis of mobile apps. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services. ACM, 204–217.
L V Haoyin. 2017. Automatic android application GUI testing—A random walk approach. In 2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET). IEEE, 72–76.
Kristian Fjeld Hasselknippe and Jingyue Li. 2017. A Novel Tool for Automatic GUI Layout Testing. In 2017 24th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 695–700.
Cuixiong Hu and Iulian Neamtiu. 2011. Automating GUI testing for Android applications. In Proceedings of the 6th International Workshop on Automation of Software Test. ACM, 77–83.
Gang Hu, Xinhao Yuan, Yang Tang, and Junfeng Yang. 2014. Efficiently, effectively detecting mobile app bugs with appdoctor. In Proceedings of the Ninth European Conference on Computer Systems. ACM, 18.
Gennaro Imparato. 2015. A combined technique of GUI ripping and input perturbation testing for Android apps. In Proceedings of the 37th International Conference on Software Engineering-Volume 2. IEEE Press, 760–762.
Marek Janicki, Mika Katara, and Tuula Pääkkönen. 2012. Obstacles and opportunities in deploying model‐based GUI testing of mobile software: a survey. Software Testing, Verification and Reliability 22, 5(2012), 313–341.
Casper S Jensen, Mukul R Prasad, and Anders Møller. 2013. Automated testing with targeted event sequence generation. In Proceedings of the 2013 International Symposium on Software Testing and Analysis. ACM, 67–77.
Meichen Ji. 2018. UIChecker: An Automatic Detection Platform for Android GUI Errors. In 2018 IEEE 9th International Conference on Software Engineering and Service Science (ICSESS). IEEE, 957–961.
He Jiang, Liming Nie, Zeyi Sun, Zhilei Ren, Weiqiang Kong, Tao Zhang, and Xiapu Luo. 2016. Rosf: Leveraging information retrieval and supervised learning for recommending code snippets. IEEE Transactions on Services Computing 12, 1 (2016), 34–46.
Jouko Kaasila, Denzil Ferreira, Vassilis Kostakos, and Timo Ojala. 2012. Testdroid: automated remote UI testing on Android. In Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia. ACM, 28.
Taeyeon Ki, Alexander Simeonov, Chang Min Park, Karthik Dantu, Steven Y Ko, and Lukasz Ziarek. 2017. Fully automated ui testing system for large-scale android apps using multiple devices. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 185.
Barbara Kitchenham. 2004. Procedures for performing systematic reviews. Keele, UK, Keele University 33, 2004 (2004), 1–26.
Pingfan Kong, Li Li, Jun Gao, Kui Liu, Tegawendé F Bissyandé, and Jacques Klein. 2018. Automated testing of android apps: A systematic literature review. IEEE Transactions on Reliability 68, 1 (2018), 45–66.
Ang Li, Zishan Qin, Mingsong Chen, and Jing Liu. 2014. ADAutomation: An activity diagram based automated GUI testing framework for smartphone applications. In 2014 Eighth International Conference on Software Security and Reliability (SERE). IEEE, 68–77.
Xiao Li, Nana Chang, Yan Wang, Haohua Huang, Yu Pei, Linzhang Wang, and Xuandong Li. 2017. ATOM: Automatic maintenance of GUI test scripts for evolving mobile applications. In 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, 161–171.
Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019. Humanoid: A deep learning-based approach to automated black-box android app testing. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1070–1073.
Chia-Hui Lin, Cheng-Zen Yang, Peng Lu, Tzu-Heng Lin, and Zhi-Jun You. 2018. Exploration Scheduling for Replay Events in GUI Testing on Android Apps. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol. 1. IEEE, 90–99.
Ying-Dar Lin, Jose F Rojas, Edward T-H Chu, and Yuan-Cheng Lai. 2014. On the accuracy, efficiency, and reusability of automated test oracles for android devices. IEEE Transactions on Software Engineering 40, 10 (2014), 957–970.
Mario Linares-Vásquez, Martin White, Carlos Bernal-Cárdenas, Kevin Moran, and Denys Poshyvanyk. 2015. Mining android app usages for generating actionable gui-based execution scenarios. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 111–122.
Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An input generation system for android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 224–234.
Riyadh Mahmood, Nariman Mirzaei, and Sam Malek. 2014. Evodroid: Segmented evolutionary testing of android apps. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 599–609.
Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective automated testing for Android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, 94–105.
Atif M Memon, Martha E Pollack, and Mary Lou Soffa. 2000. A planning-based approach to GUI testing. Proceedings of The 13th International Software/Internet Quality Week (2000).
Atif M Memon, Martha E Pollack, and Mary Lou Soffa. 2001. Hierarchical GUI test case generation using automated planning. IEEE transactions on software engineering 27, 2 (2001), 144–155.
Nariman Mirzaei, Hamid Bagheri, Riyadh Mahmood, and Sam Malek. 2015. Sig-droid: Automated system input generation for android applications. In 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 461–471.
Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016. Reducing combinatorics in GUI testing of android applications. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, 559–570.
Kevin Moran, Mario Linares-Vásquez, Carlos Bernal-Cárdenas, Christopher Vendome, and Denys Poshyvanyk. 2016. Automatically discovering, reporting and reproducing android application crashes. In 2016 IEEE international conference on software testing, verification and validation (icst). IEEE, 33–44.
Inês Coimbra Morgado and Ana C R Paiva. 2015. Testing approach for mobile applications through reverse engineering of UI patterns. In 2015 30th IEEE/ACM International Conference on Automated Software Engineering Workshop (ASEW). IEEE, 42–49.
Inês Coimbra Morgado and Ana C R Paiva. 2018. Mobile GUI testing. Software Quality Journal 26, 4 (2018), 1553–1570.
Liming Nie, He Jiang, Zhilei Ren, Zeyi Sun, and Xiaochen Li. 2016. Query expansion based on crowd knowledge for code search. IEEE Transactions on Services Computing 9, 5 (2016), 771–783.
Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of Android applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 153–164.
Minxue Pan, Tongtong Xu, Yu Pei, Zhong Li, Tian Zhang, and Xuandong Li. 2019. GUI-guided repair of mobile test scripts. In Proceedings of the 41st International Conference on Software Engineering: Companion Proceedings. IEEE Press, 326–327.
Chao Peng and Ajitha Rajan. 2020. CAT: Change-focused Android GUI Testing. arXiv preprint arXiv:2011.11766(2020).
Xue Qin, Hao Zhong, and Xiaoyin Wang. 2019. TestMig: migrating GUI test cases from iOS to Android. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 284–295.
Imran Ali Qureshi and Aamer Nadeem. 2013. GUI testing techniques: a survey. International Journal of Future computer and communication 2, 2(2013), 142.
Alireza Sadeghi, Reyhaneh Jabbarvand, and Sam Malek. 2017. Patdroid: permission-aware gui testing of android. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 220–232.
Ibrahim Anka Salihu and Rosziati Ibrahim. 2016. Systematic Exploration of Android Apps’ Events for Automated Testing. In Proceedings of the 14th International Conference on Advances in Mobile Computing and Multi Media. ACM, 50–54.
Richard K Shehady and Daniel P Siewiorek. 1997. A method to automate user interface testing using variable finite state machines. In Proceedings of IEEE 27th International Symposium on Fault Tolerant Computing. IEEE, 80–88.
Wei Song, Xiangxing Qian, and Jeff Huang. 2017. EHBDroid: Beyond GUI testing for Android applications. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 27–37.
Ting Su. 2016. FSMdroid: guided GUI testing of android apps. In 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C). IEEE, 689–691.
Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, stochastic model-based GUI testing of Android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 245–256.
Sathyanarayanan Subramanian, Thomas Singleton, and Omar El Ariss. 2016. Class coverage GUI testing for Android applications. In 2016 International Conference on System Reliability and Science (ICSRS). IEEE, 84–89.
Tommi Takala, Mika Katara, and Julian Harty. 2011. Experiences of system-level model-based GUI testing of an Android application. In 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation. IEEE, 377–386.
Tuyet Vuong and Shingo Takada. 2019. Semantic analysis for deep Q-network in android GUI testing. In 31st International Conference on Software Engineering and Knowledge Engineering, SEKE 2019. Knowledge Systems Institute Graduate School, 123–128.
Jue Wang, Yanyan Jiang, Chang Xu, Chun Cao, Xiaoxing Ma, and Jian Lu. 2020. ComboDroid: generating high-quality test inputs for Android apps via use case combinations. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 469–480.
Jue Wang, Yanyan Jiang, Chang Xu, Xiaoxing Ma, and Jian Lu. 2019. Automatic test-input generation for Android applications. SCIENTIA SINICA Informationis 49, 10 (2019), 1234–1266.
Peng Wang, Bin Liang, Wei You, Jingzhe Li, and Wenchang Shi. 2014. Automatic Android GUI traversal with high coverage. In 2014 Fourth International Conference on Communication Systems and Network Technologies. IEEE, 1161–1166.
Hsiang-Lin Wen, Chia-Hui Lin, Tzong-Han Hsieh, and Cheng-Zen Yang. 2015. Pats: A parallel gui testing framework for android applications. In 2015 IEEE 39Th annual computer software and applications conference, Vol. 2. IEEE, 210–215.
Thomas Wetzlmaier, Rudolf Ramler, and Werner Putschögl. 2016. A framework for monkey GUI testing. In 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, 416–423.
Martin White, Mario Linares-Vásquez, Peter Johnson, Carlos Bernal-Cárdenas, and Denys Poshyvanyk. 2015. Generating reproducible and replayable bug reports from android application crashes. In 2015 IEEE 23rd International Conference on Program Comprehension. IEEE, 48–59.
Haowei Wu, Yan Wang, and Atanas Rountev. 2018. Sentinel: Generating GUI tests for Android sensor leaks. In 2018 IEEE/ACM 13th International Workshop on Automation of Software Test (AST). IEEE, 27–33.
Qing Xie. 2006. Developing cost-effective model-based techniques for GUI testing. In Proceedings of the 28th international conference on Software engineering. 997–1000.
Feng Xue. 2020. Automated mobile apps testing from visual perspective. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 577–581.
Masato Yamamoto, Evgeny Pyshkin, and Maxim Mozgovoy. 2018. Reducing False Positives in Automated OpenCV-based Non-Native GUI Software Testing. In Proceedings of the 3rd International Conference on Applications in Information Technology. ACM, 41–45.
Jiwei Yan, Hao Liu, Linjie Pan, Jun Yan, Jian Zhang, and Bin Liang. 2020. Multiple-entry testing of android applications by constructing activity launching contexts. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 457–468.
Jiwei Yan, Hao Zhou, Xi Deng, Ping Wang, Rongjie Yan, Jun Yan, and Jian Zhang. [n.d.]. Efficient testing of GUI applications by event sequence reduction. Science of Computer Programming 201 ([n. d.]), 102522.
Cheng-Zen Yang, Yuan-Fu Luo, Yu-Jen Chien, and Hsiang-Lin Wen. 2016. Learning to prioritize GUI test cases for Android laboratory programs. In Proceedings of the International Conference on Artificial Intelligence and Robotics and the International Conference on Automation, Control and Robotics Engineering. ACM, 12.
Shengqian Yang, Haowei Wu, Hailong Zhang, Yan Wang, Chandrasekar Swaminathan, Dacong Yan, and Atanas Rountev. 2018. Static window transition graphs for Android. Automated Software Engineering 25, 4 (2018), 833–873.
Wei Yang, Mukul R Prasad, and Tao Xie. 2013. A grey-box approach for automated GUI-model generation of mobile applications. In International Conference on Fundamental Approaches to Software Engineering. Springer, 250–265.
Chao-Chun Yeh, Shih-Kun Huang, and Sung-Yen Chang. 2013. A black-box based android GUI testing system. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services. ACM, 529–530.
Razieh Nokhbeh Zaeem, Mukul R Prasad, and Sarfraz Khurshid. 2014. Automated generation of oracles for testing user-interaction features of mobile apps. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. IEEE, 183–192.
Xia Zeng, Dengfeng Li, Wujie Zheng, Fan Xia, Yuetang Deng, Wing Lam, Wei Yang, and Tao Xie. 2016. Automated test input generation for android: Are we really there yet in an industrial case?. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 987–992.
Chucheng Zhang, Haoliang Cheng, Enyi Tang, Xin Chen, Lei Bu, and Xuandong Li. 2017. Sketch-guided GUI test generation for mobile applications. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 38–43.