研究発表

This paper proposes a general-purpose robustness testing tool for machine learning models, particularly focusing on handling heterogeneous tabular data with feature constraints and diverse model architectures. The method utilizes Targeted Property-Based Testing (TPBT) from software engineering, implemented using the Hypothesis library. To address the limitations of simple robustness testing, we introduce two utility functions for TPBT: one based on the rank change of probability scores (tpbtb) and another on the difference from the top probability score (tpbtm). Furthermore, a penalty term is incorporated into the utility function to suppress the generation of invalid adversarial examples that violate feature constraints, which is crucial for tabular data. Preliminary experiments in unconstrained setting on the Bank dataset using GBDT and MLP models demonstrate that the tpbtm utility function outperforms basic PBT and tpbtb in terms of execution time and success rate when searching for adversarial examples given maximum perterbation thresholds.

This paper addresses the black-box fairness testing of machine learning models, specifically focusing on improving the approximation performance of surrogate models. Traditional fairness testing primarily emphasized efficiency (the number of discriminatory instances found per unit time), often overlooking coverage, which can lead to findings clustered in a narrow area. To address this, the study introduces the metrics of Precision (equivalent to the conventional success rate) and Recall (coverage of discriminatory data) from information retrieval into the fairness testing field. A new, meaningful definition of Recall is proposed using Monte Carlo methods and spatial "quantization". Building on the state-of-the-art AFT (Approximation-Guided Fairness Testing) method, the paper proposes and evaluates two new approaches that prioritize Recall: one that resets the oracle data and discards failure data (Method A), and another that resets the oracle data but adds failure data (Method B). Experimental results show that Method A, which emphasizes resetting the oracle data and discarding failure data, improves Recall compared to other methods, though at the expense of lower Precision and efficiency.

We develop methods to quantitatively verify fairness properties in tree ensemble models. Rather than simply identifying whether bias exists, our approach estimates the ratio of all counterexamples and characterizes the regions where they occur. We adapt verification frameworks previously used for neural networks, leveraging the discrete structure of tree ensembles to provide both upper and lower bounds on fairness violations. Experimental results across five datasets demonstrate our technique outperforms existing fairness testing methods in both effectiveness and efficiency.

As machine learning (ML) systems are increasingly used in various fields, including tasks with high social impact, concerns about their fairness are growing. To address these concerns, individual fairness testing (IFT) has been introduced to identify individual discriminatory instances (IDIs) that indicate the violation of individual fairness in a given ML classifier. In this paper, we propose a black-box testing algorithm for IFT, named Aft (short for Approximation-guided Fairness Testing). Aft constructs approximate models based on decision trees, and generates test cases by sampling paths of the decision trees. Our evaluation by experiments confirms that Aft outperforms the state-of-the-art black-box IFT algorithm ExpGA both in efficiency (by 3.42 times) and diversity of IDIs identified by algorithms (by 1.16 times).

There are growing concerns regarding the fairness of Machine Learning (ML) algorithms. Individual fairness testing is introduced to address the fairness concerns, and it aims to detect discriminatory instances which exhibit unfairness in a given classifier from its input space. XGBoost is one of the most prominent ML algorithms in recent years. In this study, we propose an individual fairness testing method for XGBoost classifier, leveraging the formal verification technique. To evaluate our method, we build XGBoost classifiers on three real-world datasets, and conduct individual fairness testing against them. Through the evaluation, we observe that our method can correctly detect discriminatory instances in XGBoost classifiers within an acceptable running time. Among all testing tasks, the longest running time for detecting 100 discriminatory instances is 2656.4 seconds.

Context: There are growing concerns about algorithmic fairness, as some machine learning (ML)-based algorithms have been found to exhibit biases against protected attributes such as gender, race, age and so on. Individual fairness requires an ML classifier to produce similar outputs for similar individuals. Verification Based Testing (Vbt) is a state-of-the-art black-box testing algorithm for individual fairness that leverages constraint solving to generate test cases. Objective: Generating diverse test cases is expected to facilitate efficient detection of diverse discriminatory data instances (i.e., cases that violate individual fairness). Hashing-based sampling techniques draw a sample approximately uniformly at random from the set of solutions of given Boolean constraints. We propose Vbt-X, which improves Vbt with hashing-based sampling, aiming to improve its testing performance. Method: We realize hashing-based sampling for Vbt. The challenge is that the off-the-shelf hashing-based sampling techniques cannot be integrated in a straightforward manner because the constraints in Vbt are generally not Boolean. Moreover, we propose several enhancement techniques to make Vbt-X more efficient. Results: To evaluate our method, we conduct experiments, where Vbt-X is compared to Vbt, Sg and ExpGA (other well-known fairness testing algorithms) over a set of configurations consisting of several datasets, protected attributes, and ML classifiers. The results show that, with each configuration, Vbt-X detects more discriminatory data instances with higher diversity than Vbt and Sg. Vbt-X detects discriminatory data instances with higher diversity than ExpGA, though the number of discriminatory data instances detected by Vbt-X is lesser than ExpGA. Conclusion: Our proposed method performs better than other state-of-the-art black-box fairness testing algorithms, particularly in terms of diversity. Our method can serve to efficiently identify flaws in ML classifiers with respect to individual fairness for subsequent improvements of an ML classifier. On the other hand, although our method is specific to individual fairness, it could work for testing other aspects of a software system such as security and counterfactual explanations with some technical adaptations, which remains for future work.

There is a growing concern on algorithm fairness, according to wider adoption of machine learning techniques in our daily life. Testing of individual fairness is an approach to algorithm fairness concern. Verification Based Testing (VBT) is a state-of-the-art testing technique for individual fairness, that leverages verification techniques using constraint solving. In this paper, we develop a black-box individual fairness testing technique Vbt-X, which applies hash-based sampling techniques to the test case generation part of Vbt, aiming to improve its testing ability. Our evaluation by experiments confirms that Vbt-X improves the testing ability of Vbt by 2.92 times in average.

Fairness testing, given a machine learning classifier, detects discriminatory data contained in it via executing test cases. In this paper, we propose a new approach to fairness testing named Vbt-Ct, which applies combinatorial t-way testing (CT) to Verification Based Testing (Vbt). Vbt is a state-of-the-art fairness testing method, which represents a given classifier under test in logical constraints and searches for test cases by solving such constraints. CT is a coverage-based sampling technique, with an ability to sample diverse test data from a search space specified by logical constraints. We implement a proof-of-concept of Vbt-Ct, and see its feasibility by experiments. We also discuss its advantages, current limitations, and further research directions.

CDCLソルバーでは多くの学習節を計算してしまう可能性もあるため，積極的に不要な学習節を削除するためのヒューリスティックが活用されている．このため，CDCLソルバーの実行中には，同じ節を複数回学習し，削除する場合がある．近年の研究では，重複学習節を削除しないことでCDCLソルバーの効率が上がる結果が報告されている．本論文では，この手法の改善として，類似した学習節を削除しないようにするヒューリスティックを提案する．このヒューリスティックの実験評価をSAT Competitions 2020と2021のベンチマークで行い，先行手法との性能比較を行う．

公平性テストは，機械学習モデルにおける不公平を検出する．近似誘導型公平性テストは，代理モデルを用いてテストケースを生成するが，NP困難な制約求解に依存するため効率の課題がある．本研究では，制約求解を用いず，差別的空間解析に基づく新たな手法を提案する．

機械学習モデルの活用が広がる一方で、人種や性別などの属性に基づく機械学習モデルの差別的な判断が社会問題として認識されている。一般的な検証手法は、モデルが公平性を満たしていることを証明するか、公平性違反を示す一つの反例を提示するかのいずれかにとどまる。これに対して、定量的検証は、入力空間全体における反例の割合や、反例が存在する領域といった情報も提供する。これまでの定量的検証手法は DNN を対象としてきたが、決定木アンサンブルに対応する手法はまだ確立されていない。本発表では、既存の定量的検証手法をレビューし、決定木アンサンブルに対する厳密な定量的検証手法を考案する。本手法は、決定木アンサンブルの特性を活用することで、すべての入力の組合せの確認を回避しつつ、反例が存在する領域を網羅的に抽出できる。

機械学習モデルの公平性テストは，そのモデルにおける公平性を違反するケースを検出する技術である．検証ベース手法は，一連の公平性テスト技術であり，代理モデルとして決定木を使用することで，より効果的なテストケースを生成する．しかし，従来の研究では，代理モデルの近似性能や，近似性能と検証ベース手法のテスト性能の関連性，さらに決定木を代理モデルとして用いる妥当性について十分な検討が行われていない．本研究では，検証ベース手法における代理モデルの近似性能を考察し，その近似性能向上の可能性についても検討する．

機械学習モデルの公平性テストは，与えられたモデルにおける公平性の違反を検出する．検証ベース手法は，一連の公平性テスト技術である．従来の研究では，検証ベース手法でテストケースを生成する際に制約求解を用いてきたが，制約求解はNP困難であり，効率の課題がある．本発表では，制約求解を使用せずにパスサンプリングを採用することで，検証ベース手法のテストケース生成部分を改良する新たな手法を紹介する．

公平性テストは，機械学習モデルにおける公平性に違反する差別データを検出する．これまでの研究はテスト効率に着目するが，多様性が考慮されていないため，類似した差別データが検出される恐れがある．本研究では，多様性に配慮した手法を提案し，その性能を評価する．

機械学習モデルは広く活用されるようになってきた一方，機械学習モデルにより人種や性別などの要因で個人が不公平な扱いを受ける懸念が持たれている．機械学習モデルの公平性テスト手法は，機械学習モデルにおける公平性に違反する差別データを検出する．Verification Based Testing (VBT) はSharmaらによって開発されたテスト手法である．その考えは，与えられた機械学習モデルを決定木で近似することにより，得られた決定木の差別データは機械学習モデルの差別データである可能性が高い．本発表では，VBTのテスト能力を向上することを目指し，決定木における差別データを検出するためのSMTソルバーに基づく手法と，決定木のパスの全探索に基づく手法を考察する．

近年の技術発展により，機械学習モデルが実社会の様々な場面（裁判判決や融資審査、入学審査、等）で使用されるようになっている．一方で，そうした機械学習モデルが差別的な判断をすることが社会問題として認識されつつある．本発表では機械学習モデル公平性のテストのコンセプトを説明し，我々が開発する機械学習モデル公平性のテストVBT-Xを紹介する．

本発表では，VBTにハッシュサンプリングと呼ばれる技術を適用して，多様な差別データをサンプリングできるようにする手法(VBT-X)について紹介する．その他，更なる改良の可能性，差別データの多様性，公平性以外への拡張などのVBT-Xに関連する課題についても議論する．

機械学習の普及に伴い，機械学習モデルの公平性についての懸念が高まっている．機械学習モデルの公平性テストとは，与えられたモデルに含まれる公平性に違反する差別データを検出する技術である．Verification Based Testing (VBT)はSharmaらによって開発された，形式検証技術を基にしたテスト手法である．本研究では，VBTのテストケース生成部分にハッシュベースのサンプリング技術を適用し，テスト能力の向上を目指したVBT-Xと呼ばれる公平性テスト技術を提案した．VBT-XはVBTより差別データの検出能力が 2.92 倍に向上したことを計算機実験により確認した．