There is a growing concern on algorithm fairness, according to wider adoption of machine learning techniques in our daily life. Testing of individual fairness is an approach to algorithm fairness concern. Verification Based Testing (VBT) is a state-of-the-art testing technique for individual fairness, that leverages verification techniques using constraint solving. In this paper, we develop a black-box individual fairness testing technique Vbt-X, which applies hash-based sampling techniques to the test case generation part of Vbt, aiming to improve its testing ability. Our evaluation by experiments confirms that Vbt-X improves the testing ability of Vbt by 2.92 times in average.
@inproceedings{10.1007/978-3-031-21251-2_3,author={Zhao, Zhenjiang and Toda, Takahisa and Kitamura, Takashi},editor={Papadakis, Mike and Vergilio, Silvia Regina},title={Efficient Fairness Testing Through Hash-Based Sampling},booktitle={Proceedings of Search-Based Software Engineering},year={2022},publisher={Springer International Publishing},address={Cham},pages={35--50},isbn={978-3-031-21251-2},}
SSBSE
Applying Combinatorial Testing to Verification-Based Fairness Testing
Takashi Kitamura, Zhenjiang Zhao, and Takahisa Toda
In Proceedings of Search-Based Software Engineering 2022
Fairness testing, given a machine learning classifier, detects discriminatory data contained in it via executing test cases. In this paper, we propose a new approach to fairness testing named Vbt-Ct, which applies combinatorial t-way testing (CT) to Verification Based Testing (Vbt). Vbt is a state-of-the-art fairness testing method, which represents a given classifier under test in logical constraints and searches for test cases by solving such constraints. CT is a coverage-based sampling technique, with an ability to sample diverse test data from a search space specified by logical constraints. We implement a proof-of-concept of Vbt-Ct, and see its feasibility by experiments. We also discuss its advantages, current limitations, and further research directions.
@inproceedings{10.1007/978-3-031-21251-2_7,author={Kitamura, Takashi and Zhao, Zhenjiang and Toda, Takahisa},editor={Papadakis, Mike and Vergilio, Silvia Regina},title={Applying Combinatorial Testing to Verification-Based Fairness Testing},booktitle={Proceedings of Search-Based Software Engineering},year={2022},publisher={Springer International Publishing},address={Cham},pages={101--107},isbn={978-3-031-21251-2},}
JSAI
Note on CDCL Inference with Similar Learnt Clauses (in Japanese)
The conflict-driven clause learning (CDCL) is a standard algorithmic framework on which almost state-of-the-art SAT solvers are based. During the solving process of the CDCL solver, many learnt clauses are generated, and those turned out to be useless in terms of some criteria are removed. Since the decision commonly relies on heuristics, the same clauses can appear multiple times. Hence, the evaluation of the learnt clause utility has a significant impact on the solver’s performance. The recently proposed DL heuristic determines the utility in terms of the number of times clauses are generated. To improve this, we introduce the similarity of learnt clauses and propose a similarity-based clause management method. In experiments we compared our method with the DL, both implemented on top of CaDiCal, and we confirmed that our method outperforms the DL as well as the intact CaDiCal in both PAR-2 scores and the number of solved instances.
@article{Zhao+JSAI22,title={Note on CDCL Inference with Similar Learnt Clauses (in Japanese)},author={Zhao, Zhenjiang and Toda, Takahisa},journal={Proceedings of the Annual Conference of JSAI},volume={JSAI2022},pages={4K1GS105-4K1GS105},year={2022},doi={10.11517/pjsai.JSAI2022.0_4K1GS105},}
Poster Presentations
2023
MLSE
Consideration of Fairness Testing Method Based on a Complete Search for Paths in Decision Tree
Zhenjiang Zhao, Takahisa Toda, and Takashi Kitamura
In Special Interest Group on Machine Learning Systems Engineering, Jun 2023