research-article
Free access
Authors: Chang-Ai Sun, Jiayu Xing, Xiaobei Li, Xiaoyi Zhang, An Fu
MET 2024: Proceedings of the 9th ACM International Workshop on Metamorphic Testing
Pages 26 - 33
Published: 13 September 2024 Publication History
Metrics
Total Citations0Total Downloads0Last 12 Months0
Last 6 weeks0
New Citation Alert added!
This alert has been successfully added and will be sent to:
You will be notified whenever a record that you have chosen has been cited.
To manage your alert preferences, click on the button below.
Manage my Alerts
New Citation Alert!
Please log in to your account
PDFeReader
- View Options
- References
- Media
- Tables
- Share
Abstract
Metamorphic testing (MT) is widely adopted for testing image processing applications. Although a variety of metamorphic relations (MRs) have been proposed, using all of them for testing will cost a large amount of computational resources. In addition, complex transformation operations are not well supported when generating follow-up test images based on MRs. To overcome these limitations, this study proposes a general MT framework for image processing applications, which employs CycleGAN to generate images that are very close to the realistic scenarios and leverages MRs for various categories of image processing applications. Two optimization strategies called EquivalentMR and SSampling are further proposed to reduce MRs and test images, respectively. A prototype tool called MT4I was developed. The experimental results showed that the proposed framework was capable of effectively testing various categories of image processing applications, while optimization strategies can reduce the amounts of MRs and test images without significantly jeopardizing the fault detection effectiveness.
References
[1]
2021. MutPy 0.6.1. https://pypi.org/project/MutPy.
[2]
Chang ai Sun, An Fu, Zuoyi Wang, Qin Wen, Peng Wu, and T.Y. Chen. 2020. Iterative Metamorphic Testing for Web Services: Technique and Case Studies. International Journal of Web and Grid Services, 16, 4 (2020), 364–392.
Digital Library
[3]
S. S. Banerjee, S. Jha, J. Cyriac, Z. T. Kalbarczyk, and R. K. Iyer. 2018. Hands off the wheel in autonomous vehicles?: A systems perspective on over a million miles of field data. In Proceedings of the 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks. 586–597.
[4]
H. B. Braiek and F. Khomh. 2019. DeepEvolution: A search-based testing approach for deep neural networks. In Proceedings of the International Conference on Software Maintenance and Evolution (ICSME). 454–458.
[5]
T. Y. Chen. 2010. Metamorphic testing: a simple approach to alleviate the oracle problem. In Proceedings the 5th IEEE International Symposium on Service Oriented System Engineering. 1–2.
Digital Library
[6]
T. Y. Chen, F. C. Kuo, H. Liu, P. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou. 2018. Metamorphic testing: a review of challenges and opportunities. Comput. Surveys, 51, 1 (2018), 4:1–4:27.
[7]
T. Y. Chen, T. H. Tse, and Z. Q. Zhou. 2003. Fault-based testing without the need of oracles. Information and Software Technology, 45, 1 (2003), 1–9.
Digital Library
[8]
Chu Chu, Andrey Zhmoginov, and Mark Sandler. 2017. Cyclegan, a master of steganography. arXiv preprint arXiv:1712.02950.
[9]
Y. Deng, G. N. Lou, X. Zheng, T. Y. Zhang, M. Kim, H. Liu, C. Wang, and T. Y. Chen. 2021. BMT: behavior driven development-based metamorphic testing for autonomous driving models. In Proceedings of the 6th International Workshop on Metamorphic Testing (MET). 32–36.
[10]
G. W. Dong, C. H. Nie, and B. W. Xu. 2009. Effectively metamorphic testing based on program path analysis. Chinese Journal of Computers, 32, 5 (2009), 1002–1013.
[11]
M. N. Du, N. H. Liu, and X. Hu. 2019. Techniques for interpretable machine learning. Communications of the ACM (CACM), 63, 1 (2019), 68–77.
Digital Library
[12]
A. Dwarakanath, M. Ahuja, S. Sikand, R. M. Rao, R. P. J. C. Bose, N. Dubash, and S. Podder. 2018. Identifying implementation bugs in machine learning based image classifiers using metamorphic testing. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. 118–128.
[13]
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. 2014. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems. 2672–2680.
[14]
T. Jameel, M. X. Lin, and C. Lin. 2016. Metamorphic relations based test oracles for image processing applications. International Journal of Software Innovation (IJSI), 4, 1 (2016), 16–30.
Digital Library
[15]
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Proceedings of the 13th European Conference on Computer Vision. 740–755.
[16]
J. Mayer and R. Guderlei. 2006. On random testing of image processing applications. In Proceedings of the 6th International Conference on Quality Software. 85–92.
[17]
P. Naidu, H. Gudaparthi, and N. Niu. 2021. Metamorphic testing for convolutional neural networks: relations over image classification. In Proceedings of the 22nd International Conference on Information Reuse and Integration for Data Science (IRI). 99–106.
[18]
S. H. N. Santos, B. N. C. Silveira, S. A. Andrade, M. Delamaro, and S. R. S. Souza. 2020. An experimental study on applying metamorphic testing in machine learning applications. In Proceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing (SAST 20). 98–106.
Digital Library
[19]
S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-Cortés. 2016. A survey on metamorphic testing. IEEE Transaction on Software Engineering, 42, 9 (2016), 805–824.
[20]
James Z. Wang, Jia Li, and Gio Wiederhold. 2001. SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture Libraries. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23, 9 (2001), 947–963.
Digital Library
[21]
S. Wang and Z. D. Su. 2020. Metamorphic object insertion for testing object detection systems. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1053–1065.
[22]
X. Y. Xie, J. Ho, C. Murphy, G. E. Kaiser, B. W. Xu, and T. Y. Chen. 2009. Application of metamorphic testing to supervised classifiers. In Proceedings of the 9th International Conference on Quality Software. 135–144.
[23]
Hehui Zhou, Wenbo Li, Ziming Kong, Lei Huang, Ting Liu, and Yang Liu. 2020. Deepbillboard: Systematic physical-world testing of autonomous driving systems. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 347–358.
Digital Library
Index Terms
Metamorphic Testing of Image Processing Applications: A General Framework and Optimization Strategies
Software and its engineering
Software creation and management
Software verification and validation
Software defect analysis
Software testing and debugging
Recommendations
- Self-Checked Metamorphic Testing of an Image Processing Program
SSIRI '10: Proceedings of the 2010 Fourth International Conference on Secure Software Integration and Reliability Improvement
Metamorphic testing is an effective technique for testing systems that do not have test oracles, for which it is practically impossible to know the correct output of an arbitrary test input. In metamorphic testing, instead of checking the correctness of ...
Read More
- Fault detection effectiveness of source test case generation strategies for metamorphic testing
MET '18: Proceedings of the 3rd International Workshop on Metamorphic Testing
Metamorphic testing is a well known approach to tackle the oracle problem in software testing. This technique requires the use of source test cases that serve as seeds for the generation of follow-up test cases. Systematic design of test cases is ...
Read More
- Metamorphic Testing: A Simple Approach to Alleviate the Oracle Problem
SOSE '10: Proceedings of the 2010 Fifth IEEE International Symposium on Service Oriented System Engineering
The oracle problem is very common in the testing of service-oriented systems. Metamorphic testing has been proposed to alleviate the oracle problem in software testing. This talk aims at presenting the state of the art in metamorphic testing. It ...
Read More
Comments
Information & Contributors
Information
Published In
MET 2024: Proceedings of the 9th ACM International Workshop on Metamorphic Testing
September 2024
40 pages
ISBN:9798400711176
DOI:10.1145/3679006
- General Chairs:
- Huai Liu
Swinburne University of Technology, Australia
, - Aldeida Aleti
Monash University, Australia
, - Aitor Arrieta
Mondragon University, Spain
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [emailprotected].
Sponsors
- SIGSOFT: ACM Special Interest Group on Software Engineering
- AITO
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 September 2024
Permissions
Request permissions for this article.
Check for updates
Author Tags
- Fault Detection Effectiveness
- Image Processing Applications
- Metamorphic Testing
- Software Testing
Qualifiers
- Research-article
Funding Sources
- National Natural Science Foundation of China
Conference
MET '24
Sponsor:
- SIGSOFT
Contributors
Other Metrics
View Article Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
Total Citations
Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 07 Sep 2024
Other Metrics
View Author Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in
Full Access
Get this Publication
Media
Figures
Other
Tables
