-
-
-
-
- 
- Usman Gohar
- Iowa State University
-
-
- Towards Understanding Fairness and its Composition in Ensemble Machine Learning
- Machine Learning (ML) software has been widely adopted in modern society, with reported fairness implications for minority groups based on race, sex, age, etc.
- Many recent works have proposed methods to measure and mitigate algorithmic bias in ML models. The existing approaches focus on single classifier-based ML models.
- However, real-world ML models are often composed of multiple independent or dependent learners in an ensemble (e.g., Random Forest), where the fairness composes in a non-trivial way.
- How does fairness compose in ensembles? What are the fairness impacts of the learners on the ultimate fairness of the ensemble? Can fair learners result in an unfair ensemble?
- Furthermore, studies have shown that hyperparameters influence the fairness of ML models. Ensemble hyperparameters are more complex since they affect how learners are combined in
- different categories of ensembles. In this paper, we comprehensively study popular real-world ensembles: Bagging, Boosting, Stacking, and Voting. We have developed a benchmark
- of 168 ensemble models collected from Kaggle on four popular fairness datasets. We use existing fairness metrics to understand the composition of fairness. Our results show that
- ensembles can be designed to be fairer without using mitigation techniques. We also identify the interplay between fairness composition and data characteristics to guide fair
- ensemble design.
-
- Presentation Date: Monday, November 20, 2023 at 4:00 PM CET
-
-
-
-
-
-
+
@@ -244,6 +218,41 @@ Past Seminars
+
+
+ -
+
+ Towards Understanding Fairness and its Composition in Ensemble ML, Monday, November 20, 2023, by Usman Gohar from ISU
+
+
+
+
+
+
+
+
Usman Gohar
+ Iowa State University
+
+
+ Towards Understanding Fairness and its Composition in Ensemble Machine Learning
+ Machine Learning (ML) software has been widely adopted in modern society, with reported fairness implications for minority groups based on race, sex, age, etc.
+ Many recent works have proposed methods to measure and mitigate algorithmic bias in ML models. The existing approaches focus on single classifier-based ML models.
+ However, real-world ML models are often composed of multiple independent or dependent learners in an ensemble (e.g., Random Forest), where the fairness composes in a non-trivial way.
+ How does fairness compose in ensembles? What are the fairness impacts of the learners on the ultimate fairness of the ensemble? Can fair learners result in an unfair ensemble?
+ Furthermore, studies have shown that hyperparameters influence the fairness of ML models. Ensemble hyperparameters are more complex since they affect how learners are combined in
+ different categories of ensembles. In this paper, we comprehensively study popular real-world ensembles: Bagging, Boosting, Stacking, and Voting. We have developed a benchmark
+ of 168 ensemble models collected from Kaggle on four popular fairness datasets. We use existing fairness metrics to understand the composition of fairness. Our results show that
+ ensembles can be designed to be fairer without using mitigation techniques. We also identify the interplay between fairness composition and data characteristics to guide fair
+ ensemble design.
+
+ Presentation Date: Monday, November 20, 2023 at 4:00 PM CET
+
+
+
+
+
+
+
-
ITER: Iterative Neural Repair for Multi-Location Patches, Monday, November 6, 2023, by He Ye from CMU
@@ -259,7 +268,14 @@
He Ye
ITER: Iterative Neural Repair for Multi-Location Patches
- Automated program repair (APR) has achieved promising results, especially using neural networks. Yet, the overwhelming majority of patches produced by APR tools are confined to one single location. When looking at the patches produced with neural repair, most of them fail to compile, while a few uncompilable ones go in the right direction. In both cases, the fundamental problem is to ignore the potential of partial patches. In this paper, we propose an iterative program repair paradigm called ITER founded on the concept of improving partial patches until they become plausible and correct. First, ITER iteratively improves partial single-location patches by fixing compilation errors and further refining the previously generated code. Second, ITER iteratively improves partial patches to construct multi-location patches, with fault localization re-execution. ITER is implemented for Java based on battle-proven deep neural networks and code representation. ITER is evaluated on 476 bugs from 10 open-source projects in Defects4J 2.0. ITER succeeds in repairing 76 of them, including 15 multi-location bugs which is a new frontier in the field.
+ Automated program repair (APR) has achieved promising results, especially using neural networks. Yet, the overwhelming majority of patches produced
+ by APR tools are confined to one single location. When looking at the patches produced with neural repair, most of them fail to compile, while a few
+ uncompilable ones go in the right direction. In both cases, the fundamental problem is to ignore the potential of partial patches. In this paper, we
+ propose an iterative program repair paradigm called ITER founded on the concept of improving partial patches until they become plausible and correct.
+ First, ITER iteratively improves partial single-location patches by fixing compilation errors and further refining the previously generated code.
+ Second, ITER iteratively improves partial patches to construct multi-location patches, with fault localization re-execution. ITER is implemented for
+ Java based on battle-proven deep neural networks and code representation. ITER is evaluated on 476 bugs from 10 open-source projects in Defects4J 2.0.
+ ITER succeeds in repairing 76 of them, including 15 multi-location bugs which is a new frontier in the field.
Presentation Date: Monday, November 6, 2023 at 3:00 PM CET
@@ -283,8 +299,16 @@ Yi-Hsien Chen
Guided Malware Sample Analysis Based on Graph Neural Networks
- Malicious binaries have caused data and monetary loss to people, and these binaries keep evolving rapidly nowadays. While manual analysis is slow and ineffective, automated malware report generation is a long-term goal for malware analysts and researchers. This study moves one step toward the goal by identifying essential functions in malicious binaries to accelerate and even automate the analyzing process. We design and implement an expert system based on our proposed graph neural network called MalwareExpert. The system pinpoints the essential functions of an analyzed sample and visualizes the relationships between involved parts. The evaluation results show that our approach has a competitive detection performance (97.3% accuracy and 96.5% recall rate) compared to existing malware detection models. Moreover, it gives an intuitive and
-easy-to-understand explanation of the model predictions by visualizing and correlating essential functions. We compare the identified essential functions reported by our system against several expert-made malware analysis reports from multiple sources. Our qualitative and quantitative analyses show that the pinpointed functions indicate accurate directions. In the best case, the top 2% of functions reported from the system can cover all expert-annotated functions in three steps. We believe that the MalwareExpert system has shed light on automated program behavior analysis.
+ Malicious binaries have caused data and monetary loss to people, and these binaries keep evolving rapidly nowadays. While manual analysis is slow
+ and ineffective, automated malware report generation is a long-term goal for malware analysts and researchers. This study moves one step toward the goal
+ by identifying essential functions in malicious binaries to accelerate and even automate the analyzing process. We design and implement an expert system
+ based on our proposed graph neural network called MalwareExpert. The system pinpoints the essential functions of an analyzed sample and visualizes the
+ relationships between involved parts. The evaluation results show that our approach has a competitive detection performance (97.3% accuracy and 96.5% recall
+ rate) compared to existing malware detection models. Moreover, it gives an intuitive and easy-to-understand explanation of the model predictions by visualizing
+ and correlating essential functions. We compare the identified essential functions reported by our system against several expert-made malware analysis reports
+ from multiple sources. Our qualitative and quantitative analyses show that the pinpointed functions indicate accurate directions. In the best case, the top 2%
+ of functions reported from the system can cover all expert-annotated functions in three steps. We believe that the MalwareExpert system has shed light on
+ automated program behavior analysis.
Presentation Date: Monday, October 30, 2023 at 10:30 AM CET
@@ -308,9 +332,14 @@ Nikhil Parasaram
Rete: Learning Namespace Representation for Program Repair
- A key challenge of automated program repair is finding correct patches in the vast search space of candidate patches. Real-world programs define large namespaces of variables that considerably contributes to the search space explosion. Existing program repair approaches neglect information about the program namespace, which makes them inefficient and increases the chance of test-overfitting.
- We propose Rete, a new program repair technique, that learns project-independent information about program namespace and uses it to navigate the search space of patches. Rete uses a neural network to extract project-independent information about variable CDU chains, def-use chains augmented with control flow. Then, it ranks patches by jointly ranking variables and the patch templates into which the variables are inserted.
- We evaluated Rete on 142 bugs extracted from two datasets, ManyBugs and BugsInPy. Our experiments demonstrate that Rete generates six new correct patches that fix bugs that previous tools did not repair, an improvement of 31% and 59% over the existing state of the art.
+ A key challenge of automated program repair is finding correct patches in the vast search space of candidate patches. Real-world programs define
+ large namespaces of variables that considerably contributes to the search space explosion. Existing program repair approaches neglect information about
+ the program namespace, which makes them inefficient and increases the chance of test-overfitting. We propose Rete, a new program repair technique, that
+ learns project-independent information about program namespace and uses it to navigate the search space of patches. Rete uses a neural network to extract
+ project-independent information about variable CDU chains, def-use chains augmented with control flow. Then, it ranks patches by jointly ranking variables
+ and the patch templates into which the variables are inserted. We evaluated Rete on 142 bugs extracted from two datasets, ManyBugs and BugsInPy. Our
+ experiments demonstrate that Rete generates six new correct patches that fix bugs that previous tools did not repair, an improvement of 31% and 59% over
+ the existing state of the art.
Presentation Date: Monday, October 9, 2023 at 10:30 AM CEST
- Usman Gohar
-Iowa State University
-Towards Understanding Fairness and its Composition in Ensemble Machine Learning
-Machine Learning (ML) software has been widely adopted in modern society, with reported fairness implications for minority groups based on race, sex, age, etc. - Many recent works have proposed methods to measure and mitigate algorithmic bias in ML models. The existing approaches focus on single classifier-based ML models. - However, real-world ML models are often composed of multiple independent or dependent learners in an ensemble (e.g., Random Forest), where the fairness composes in a non-trivial way. - How does fairness compose in ensembles? What are the fairness impacts of the learners on the ultimate fairness of the ensemble? Can fair learners result in an unfair ensemble? - Furthermore, studies have shown that hyperparameters influence the fairness of ML models. Ensemble hyperparameters are more complex since they affect how learners are combined in - different categories of ensembles. In this paper, we comprehensively study popular real-world ensembles: Bagging, Boosting, Stacking, and Voting. We have developed a benchmark - of 168 ensemble models collected from Kaggle on four popular fairness datasets. We use existing fairness metrics to understand the composition of fairness. Our results show that - ensembles can be designed to be fairer without using mitigation techniques. We also identify the interplay between fairness composition and data characteristics to guide fair - ensemble design.
- -Presentation Date: Monday, November 20, 2023 at 4:00 PM CET
- -Past Seminars
+
+
+ -
+
+ Towards Understanding Fairness and its Composition in Ensemble ML, Monday, November 20, 2023, by Usman Gohar from ISU
+
+
+
+
+
+
+
+
Usman Gohar
+ Iowa State University
+
+
+ Towards Understanding Fairness and its Composition in Ensemble Machine Learning
+ Machine Learning (ML) software has been widely adopted in modern society, with reported fairness implications for minority groups based on race, sex, age, etc.
+ Many recent works have proposed methods to measure and mitigate algorithmic bias in ML models. The existing approaches focus on single classifier-based ML models.
+ However, real-world ML models are often composed of multiple independent or dependent learners in an ensemble (e.g., Random Forest), where the fairness composes in a non-trivial way.
+ How does fairness compose in ensembles? What are the fairness impacts of the learners on the ultimate fairness of the ensemble? Can fair learners result in an unfair ensemble?
+ Furthermore, studies have shown that hyperparameters influence the fairness of ML models. Ensemble hyperparameters are more complex since they affect how learners are combined in
+ different categories of ensembles. In this paper, we comprehensively study popular real-world ensembles: Bagging, Boosting, Stacking, and Voting. We have developed a benchmark
+ of 168 ensemble models collected from Kaggle on four popular fairness datasets. We use existing fairness metrics to understand the composition of fairness. Our results show that
+ ensembles can be designed to be fairer without using mitigation techniques. We also identify the interplay between fairness composition and data characteristics to guide fair
+ ensemble design.
+
+ Presentation Date: Monday, November 20, 2023 at 4:00 PM CET
+
+
+
+
+
+
+
-
ITER: Iterative Neural Repair for Multi-Location Patches, Monday, November 6, 2023, by He Ye from CMU
@@ -259,7 +268,14 @@
He Ye
ITER: Iterative Neural Repair for Multi-Location Patches
- Automated program repair (APR) has achieved promising results, especially using neural networks. Yet, the overwhelming majority of patches produced by APR tools are confined to one single location. When looking at the patches produced with neural repair, most of them fail to compile, while a few uncompilable ones go in the right direction. In both cases, the fundamental problem is to ignore the potential of partial patches. In this paper, we propose an iterative program repair paradigm called ITER founded on the concept of improving partial patches until they become plausible and correct. First, ITER iteratively improves partial single-location patches by fixing compilation errors and further refining the previously generated code. Second, ITER iteratively improves partial patches to construct multi-location patches, with fault localization re-execution. ITER is implemented for Java based on battle-proven deep neural networks and code representation. ITER is evaluated on 476 bugs from 10 open-source projects in Defects4J 2.0. ITER succeeds in repairing 76 of them, including 15 multi-location bugs which is a new frontier in the field.
+ Automated program repair (APR) has achieved promising results, especially using neural networks. Yet, the overwhelming majority of patches produced
+ by APR tools are confined to one single location. When looking at the patches produced with neural repair, most of them fail to compile, while a few
+ uncompilable ones go in the right direction. In both cases, the fundamental problem is to ignore the potential of partial patches. In this paper, we
+ propose an iterative program repair paradigm called ITER founded on the concept of improving partial patches until they become plausible and correct.
+ First, ITER iteratively improves partial single-location patches by fixing compilation errors and further refining the previously generated code.
+ Second, ITER iteratively improves partial patches to construct multi-location patches, with fault localization re-execution. ITER is implemented for
+ Java based on battle-proven deep neural networks and code representation. ITER is evaluated on 476 bugs from 10 open-source projects in Defects4J 2.0.
+ ITER succeeds in repairing 76 of them, including 15 multi-location bugs which is a new frontier in the field.
Presentation Date: Monday, November 6, 2023 at 3:00 PM CET
@@ -283,8 +299,16 @@ Yi-Hsien Chen
-
+
+
+
-
+ + Towards Understanding Fairness and its Composition in Ensemble ML, Monday, November 20, 2023, by Usman Gohar from ISU + ++ +++++++
+
Usman Gohar
+Iowa State University
+++Towards Understanding Fairness and its Composition in Ensemble Machine Learning
+Machine Learning (ML) software has been widely adopted in modern society, with reported fairness implications for minority groups based on race, sex, age, etc. + Many recent works have proposed methods to measure and mitigate algorithmic bias in ML models. The existing approaches focus on single classifier-based ML models. + However, real-world ML models are often composed of multiple independent or dependent learners in an ensemble (e.g., Random Forest), where the fairness composes in a non-trivial way. + How does fairness compose in ensembles? What are the fairness impacts of the learners on the ultimate fairness of the ensemble? Can fair learners result in an unfair ensemble? + Furthermore, studies have shown that hyperparameters influence the fairness of ML models. Ensemble hyperparameters are more complex since they affect how learners are combined in + different categories of ensembles. In this paper, we comprehensively study popular real-world ensembles: Bagging, Boosting, Stacking, and Voting. We have developed a benchmark + of 168 ensemble models collected from Kaggle on four popular fairness datasets. We use existing fairness metrics to understand the composition of fairness. Our results show that + ensembles can be designed to be fairer without using mitigation techniques. We also identify the interplay between fairness composition and data characteristics to guide fair + ensemble design.
+ +Presentation Date: Monday, November 20, 2023 at 4:00 PM CET
+ +
+
+
-
ITER: Iterative Neural Repair for Multi-Location Patches, Monday, November 6, 2023, by He Ye from CMU @@ -259,7 +268,14 @@
He Ye
ITER: Iterative Neural Repair for Multi-Location Patches
-Automated program repair (APR) has achieved promising results, especially using neural networks. Yet, the overwhelming majority of patches produced by APR tools are confined to one single location. When looking at the patches produced with neural repair, most of them fail to compile, while a few uncompilable ones go in the right direction. In both cases, the fundamental problem is to ignore the potential of partial patches. In this paper, we propose an iterative program repair paradigm called ITER founded on the concept of improving partial patches until they become plausible and correct. First, ITER iteratively improves partial single-location patches by fixing compilation errors and further refining the previously generated code. Second, ITER iteratively improves partial patches to construct multi-location patches, with fault localization re-execution. ITER is implemented for Java based on battle-proven deep neural networks and code representation. ITER is evaluated on 476 bugs from 10 open-source projects in Defects4J 2.0. ITER succeeds in repairing 76 of them, including 15 multi-location bugs which is a new frontier in the field.
+Automated program repair (APR) has achieved promising results, especially using neural networks. Yet, the overwhelming majority of patches produced + by APR tools are confined to one single location. When looking at the patches produced with neural repair, most of them fail to compile, while a few + uncompilable ones go in the right direction. In both cases, the fundamental problem is to ignore the potential of partial patches. In this paper, we + propose an iterative program repair paradigm called ITER founded on the concept of improving partial patches until they become plausible and correct. + First, ITER iteratively improves partial single-location patches by fixing compilation errors and further refining the previously generated code. + Second, ITER iteratively improves partial patches to construct multi-location patches, with fault localization re-execution. ITER is implemented for + Java based on battle-proven deep neural networks and code representation. ITER is evaluated on 476 bugs from 10 open-source projects in Defects4J 2.0. + ITER succeeds in repairing 76 of them, including 15 multi-location bugs which is a new frontier in the field.
Presentation Date: Monday, November 6, 2023 at 3:00 PM CET
Yi-Hsien Chen
Guided Malware Sample Analysis Based on Graph Neural Networks
-Malicious binaries have caused data and monetary loss to people, and these binaries keep evolving rapidly nowadays. While manual analysis is slow and ineffective, automated malware report generation is a long-term goal for malware analysts and researchers. This study moves one step toward the goal by identifying essential functions in malicious binaries to accelerate and even automate the analyzing process. We design and implement an expert system based on our proposed graph neural network called MalwareExpert. The system pinpoints the essential functions of an analyzed sample and visualizes the relationships between involved parts. The evaluation results show that our approach has a competitive detection performance (97.3% accuracy and 96.5% recall rate) compared to existing malware detection models. Moreover, it gives an intuitive and -easy-to-understand explanation of the model predictions by visualizing and correlating essential functions. We compare the identified essential functions reported by our system against several expert-made malware analysis reports from multiple sources. Our qualitative and quantitative analyses show that the pinpointed functions indicate accurate directions. In the best case, the top 2% of functions reported from the system can cover all expert-annotated functions in three steps. We believe that the MalwareExpert system has shed light on automated program behavior analysis.
+Malicious binaries have caused data and monetary loss to people, and these binaries keep evolving rapidly nowadays. While manual analysis is slow + and ineffective, automated malware report generation is a long-term goal for malware analysts and researchers. This study moves one step toward the goal + by identifying essential functions in malicious binaries to accelerate and even automate the analyzing process. We design and implement an expert system + based on our proposed graph neural network called MalwareExpert. The system pinpoints the essential functions of an analyzed sample and visualizes the + relationships between involved parts. The evaluation results show that our approach has a competitive detection performance (97.3% accuracy and 96.5% recall + rate) compared to existing malware detection models. Moreover, it gives an intuitive and easy-to-understand explanation of the model predictions by visualizing + and correlating essential functions. We compare the identified essential functions reported by our system against several expert-made malware analysis reports + from multiple sources. Our qualitative and quantitative analyses show that the pinpointed functions indicate accurate directions. In the best case, the top 2% + of functions reported from the system can cover all expert-annotated functions in three steps. We believe that the MalwareExpert system has shed light on + automated program behavior analysis.
Presentation Date: Monday, October 30, 2023 at 10:30 AM CET
Nikhil Parasaram
Rete: Learning Namespace Representation for Program Repair
-A key challenge of automated program repair is finding correct patches in the vast search space of candidate patches. Real-world programs define large namespaces of variables that considerably contributes to the search space explosion. Existing program repair approaches neglect information about the program namespace, which makes them inefficient and increases the chance of test-overfitting. - We propose Rete, a new program repair technique, that learns project-independent information about program namespace and uses it to navigate the search space of patches. Rete uses a neural network to extract project-independent information about variable CDU chains, def-use chains augmented with control flow. Then, it ranks patches by jointly ranking variables and the patch templates into which the variables are inserted. - We evaluated Rete on 142 bugs extracted from two datasets, ManyBugs and BugsInPy. Our experiments demonstrate that Rete generates six new correct patches that fix bugs that previous tools did not repair, an improvement of 31% and 59% over the existing state of the art.
+A key challenge of automated program repair is finding correct patches in the vast search space of candidate patches. Real-world programs define + large namespaces of variables that considerably contributes to the search space explosion. Existing program repair approaches neglect information about + the program namespace, which makes them inefficient and increases the chance of test-overfitting. We propose Rete, a new program repair technique, that + learns project-independent information about program namespace and uses it to navigate the search space of patches. Rete uses a neural network to extract + project-independent information about variable CDU chains, def-use chains augmented with control flow. Then, it ranks patches by jointly ranking variables + and the patch templates into which the variables are inserted. We evaluated Rete on 142 bugs extracted from two datasets, ManyBugs and BugsInPy. Our + experiments demonstrate that Rete generates six new correct patches that fix bugs that previous tools did not repair, an improvement of 31% and 59% over + the existing state of the art.
Presentation Date: Monday, October 9, 2023 at 10:30 AM CEST