From 5fea380e95fc2bb7e4738dfb24380f8a8bd74bd1 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 14:14:53 +0100 Subject: [PATCH 01/54] Create 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md Initial Draft --- ...aditional-Testing-to-Model-Evaluation-I.md | 108 ++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 _posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md new file mode 100644 index 0000000000..eefb3c50ef --- /dev/null +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -0,0 +1,108 @@ +--- +title: Evolving with AI From Traditional Testing to Model Evaluation I +date: 2024-09-01 10:00:00 Z +categories: +- Artificial Intelligence +tags: +- Artificial Intelligence +- AI +summary: Recently worked on developing Machine learning skill definitions and L&D pathway. In this + blog post I have tried to explore the evolving role of test engineers in the era of machine learning, highlighting the key challenges ML brings, strategies for effective model evaluation and finally tried to outline a roadmap for developing the necessary skills to excel in ML model testing. +author: snandal +image: +--- + + + +Recently worked on developing Machine learning skill definitions and L&D pathway. In this blog post I have tried to explore the evolving role of test engineers in the era of machine learning, highlighting the key challenges ML brings, strategies for effective model evaluation and finally tried to outline a roadmap for developing the necessary skills to excel in ML model testing. + +Machine learning (ML) is no longer just a concept for the future, it is now a vital part of many everyday applications. From personalised recommendations on streaming services to detecting fraud in banking and from self-driving cars to advancements in healthcare, ML is changing the way industries work. +For a test engineer, the increasing use of ML brings new challenges and opportunities. As ML technology advances, the role of a test engineer also changes. Unlike traditional software, which gives consistent and predictable results, ML models work with probabilities and patterns, making testing and evaluation more complex. +In this blog we will try to look at how ML has developed, how it might affect our job as test engineers and the important strategies, considerations and skills needed to effectively evaluate and test ML models. We will also try to draw a roadmap to help us grow professionally in this area. + +## How Machine Learning Has Evolved + +Machine learning has significantly advanced since its early days. Initially, ML models were fairly simple, handling tasks like basic predictions or classifications. However, with improvements in computing power, access to large amounts of data and more complex algorithms, ML has grown into a powerful tool capable of handling sophisticated tasks like natural language processing and images recognition. +This progress has made ML widespread across industries, leading businesses to increasingly rely on data-driven decisions. As companies embrace ML, the need for thorough and reliable testing of these models has become more important, which directly affects the role of test engineers. +Before we get into the testing process, it is important to understand what makes ML models different and what impact they can have on our job as a test engineer. +At their core, ML models learn from data. They are trained on large datasets to recognise patterns and make predictions or decisions based on new information. Unlike traditional software, where rules are clearly programmed, ML models develop their logic through training, making their behaviour less predictable and sometimes harder to understand. +Traditionally, a test engineer's job involved testing systems where a given input always produced the same output, making results predictable. However, with ML, this approach changes. ML models are probabilistic, meaning their outputs can vary depending on the training data and some built-in randomness. This shift has significantly changed how software testing is done. +We now need to do more than just check if a system works as expected. We must also evaluate how well a model performs, whether it's reliable and fair. This requires combining traditional testing skills with additional knowledge specific to ML. + +## Looking at the unique challenges that testing ML models brings +1. Unpredictable Outputs: Unlike regular software, where the same input always gives the same result, ML models can give different results because they work with probabilities. This makes testing harder because instead of just checking if the output is correct, we need to evaluate how accurate and reliable the model’s predictions are. +2. Dependence on Data: The quality of an ML model depends a lot on the data it was trained on. Testing ML models means we need to deeply understand the training data and its weaknesses because any bias or missing information in the data can lead to incorrect results. +3. Performance Metrics: Traditional testing metrics like pass/fail don’t apply directly to ML models. Instead, we use statistical metrics like accuracy, precision, recall, F1 score and AUC-ROC to evaluate model performance. Understanding and choosing the right metrics is key to effective testing. +4. Model Degradation over time: As new data becomes available; an ML model’s performance can degrade causing a problem known as model drift. This means continuous monitoring and retraining are equally important, adding complexity to the testing process. + +## Model Evaluation Vs. Model Testing + +Before we dive deep into discussing possible strategies for ML model testing, it is important to understand the difference between Model evaluation and Model testing. +Model evaluation and testing might sound similar, but there is a key difference between the two. Evaluation focuses on assessing the model’s performance in a controlled environment, while model testing checks how well the model works in real-world conditions. +Let’s look at it with a simple example of a spam detection model that filters out unwanted emails. During the model evaluation phase (validation mode), we will use a labelled dataset of emails to calculate metrics like accuracy, precision and recall. For example, the model might successfully identify 95% of spam emails in this dataset. +After evaluation, we will test (prediction mode) the model with real incoming emails to see how well it handles a variety of messages, if it can correctly filter out spam in different languages, formats, or tricky emails that look legitimate. Model testing makes sure the model works well in the real environment where it will be used, way beyond the controlled conditions of evaluation. + + +## Strategies for Testing ML Models + +To effectively address the challenges that ML brings, test engineers can benefit from the following strategies: +1. Define Clear Objectives: Start by defining what success looks like for your ML model. Are you aiming for high accuracy? or is it more important to avoid mistakes like false positives? Understanding the business objectives and how they translate into model performance metrics is essential. +2. Validate the Data: Ensure that the training, validation and testing datasets are represent real-world situations the model will face. Check for issues like missing values, duplicates and biases. Proper data validation is very important to prevent garbage-in, garbage-out scenarios. +3. Apply Cross-Validation: Implement cross-validation techniques to check that the model’s performance is consistent across different subsets of the data. This helps in identifying overfitting, where the model performs well on training data but poorly on new unseen data. +4. Analyse Performance Metrics: Use a combination of metrics to get a comprehensive view of the model’s performance. For classification models, confusion matrices can be useful in providing insights into where the model is making errors. For regression models, metrics like Mean Squared Error (MSE) or R-squared can offer valuable information. +5. Perform A/B Testing: Compare the ML model’s performance with existing solutions (if available) using A/B testing. This helps in understanding whether the new model provides a significant improvement over the current system. +6. Perform Stress Testing: Subject the model to edge cases and scenarios with outliers to see how it handles unexpected inputs. This can uncover vulnerabilities and help in making the model more robust. +7. Continuous Monitoring and Feedback Loop: After deployment, continuously monitor the model’s performance. Set up a feedback loop where new data is used to retrain and improve the model. Tools like MLflow can help in managing this process efficiently. + +### Let’s look at the practical application of the above strategies with a case study of an ML model predicting patient re-admission after discharge + +Consider a hospital implementing an ML model to predict the likelihood of patients being readmitted within 30 days of discharge. +1. Define Objectives: The goal is to identify most high-risk patients so that the hospital can take proactive measures to reduce readmission rates and improve patient care. +2. Data Validation: Start by validating the dataset, ensuring that it includes a balanced representation of various patient, medical history, treatment plans etc. Use cross-validation to train the model and select the most effective algorithm for predicting readmissions. +3. Metric Selection: Choose sensitivity/recall and specificity as key metrics. High sensitivity ensures that most high-risk patients are identified, while specificity ensures that the model doesn't over-predict readmissions, leading to unnecessary interventions. +4. Stress Testing: Test the model with scenarios where patient data is incomplete, such as missing medical history or inconsistent treatment records. Assess how the model performs when handling these edge cases, which are common in real-world medical data. +5. A/B Testing: Implement the ML model alongside the existing manual risk assessment process. Compare the model's predictions with the manual predictions in real-time, focusing on how well the model improves patient outcomes. +6. Post-Deployment Monitoring: After deployment, continuously monitor the model’s performance, looking for signs of model drift as new patient data becomes available. Implement a feedback loop where the model is retrained regularly with the latest data to maintain accuracy and relevance. + +## Skills Needed for ML Model Testing +To effectively perform model evaluation, a test engineer would need a combination of technical, analytical and domain-specific skills: +1. Understanding of Machine Learning Concepts +• Understanding fundamental ML concepts such as model learning algorithms – supervised learning, unsupervised learning and reinforcement learning, model performance issues like overfitting and underfitting and the types of models (e.g., regression, classification, clustering). +• Familiarity and understanding of model evaluation metrics such as accuracy, precision, recall, F1 score, AUC-ROC, mean squared error, R-squared, etc. and knowing when and how to apply them. +3. Statistical Knowledge +• Understanding statistical measures - mean, median, variance, standard deviation and concepts like hypothesis testing, p-values, confidence intervals and correlation. +• Knowledge of probability distributions and how they apply to model outputs and uncertainties. +4. Programming Skills +• Proficiency in one of the scripting languages - Python or R, which are commonly used for ML model evaluation. Familiarity with libraries like Scikit-learn, TensorFlow or PyTorch is advantageous. +• Ability to write scripts to automate the evaluation of models, including running tests across different datasets and tracking performance over time. +5. Data Analysis and Validation +• Skills in data cleaning, handling missing data, outliers, data normalisation and feature engineering. +• Ability to validate datasets for biases, errors and representativeness, ensuring that the training data is of high quality and relevant to the model’s objectives. +• Proficiency in using tools like Python’s Pandas, Matplotlib and Seaborn for EDA to understand the dataset before feeding it into an ML model. +6. Familiarity with common ML Tools and Platforms +• Experience with tools like Scikit-learn for traditional ML, or TensorFlow/PyTorch for deep learning models. +• Familiarity with tools like MLflow, which assist in tracking experiments, managing models and automating the retraining process. +• Knowledge of using version control systems like Git for managing code and model versions. +6. Knowledge of using version control systems like Git for managing code and model versions. + +By combining these skills along with domain knowledge of the business context in which the model will be deployed, it can be effectively evaluate and tested, ensuring that it meets both technical and business requirements while remaining reliable, accurate and fair. + +## Roadmap for Developing ML Model Testing Skills + +Here’s a step-by-step roadmap for a test engineer to develop ML model testing skills: +1. Start with Basic ML Theory: Learn the fundamental concepts of ML, focusing on the types of learning, model evaluation metrics and common challenges. +2. Build Statistical Knowledge: Study key statistical principles and how they relate to ML. This is essential for understanding model performance. +3. Learn Data Analysis: Practice data preprocessing, cleaning and validation using tools like Python’s Pandas or R. Data quality is critical for effective ML models. +4. Develop Programming Skills: Get comfortable with Python or R. Start by running simple ML models using libraries like Scikit-learn and gradually move to more complex tasks. +5. Explore ML Tools: Familiarise yourself with popular ML frameworks like TensorFlow, Keras, or PyTorch and model management tools like MLflow. +6. Practice Model Evaluation: Apply your skills by testing real ML models. Use cross-validation, stress testing and performance metrics analysis to assess models. +7. Continuous Learning: Stay updated with the latest trends in ML and software testing. Engage in online courses, webinars and forums to keep your knowledge fresh. +By following this roadmap, test engineers can successfully transition into ML model testing, ensuring they stay relevant in an evolving field. + +## Conclusion + +Testing and evaluating ML models requires a mix of traditional testing skills and an understanding of statistical methods. With the right testing strategies and by following a structured approach to testing, incorporating data validation, metric analysis, stress testing and continuous monitoring, we can ensure that our ML model provides real value and meets business objectives. +Additionally, staying informed about the latest trends and tools can help handle the challenges of ML testing effectively and make a significant impact on your organisation’s success. +Machine learning is complex but also an exciting opportunity to explore new dimensions of software quality. Let’s embrace these challenges as a chance to grow and expand our skills in field of software testing. From ddc1b62c80e2a76ec679384da90aa34772476af6 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 14:17:44 +0100 Subject: [PATCH 02/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index eefb3c50ef..cefd1a47ae 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -3,9 +3,13 @@ title: Evolving with AI From Traditional Testing to Model Evaluation I date: 2024-09-01 10:00:00 Z categories: - Artificial Intelligence +- Machine Learning +- Testing tags: - Artificial Intelligence - AI +- Testing +- Machine Learning summary: Recently worked on developing Machine learning skill definitions and L&D pathway. In this blog post I have tried to explore the evolving role of test engineers in the era of machine learning, highlighting the key challenges ML brings, strategies for effective model evaluation and finally tried to outline a roadmap for developing the necessary skills to excel in ML model testing. author: snandal From 9bc35ee4fe3e10d576fb97a2139700de0f2dccc3 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 14:18:19 +0100 Subject: [PATCH 03/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...ng-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index cefd1a47ae..0f916d6d59 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -5,11 +5,13 @@ categories: - Artificial Intelligence - Machine Learning - Testing +- Model Evaluation tags: - Artificial Intelligence - AI - Testing - Machine Learning +- Model Evaluation summary: Recently worked on developing Machine learning skill definitions and L&D pathway. In this blog post I have tried to explore the evolving role of test engineers in the era of machine learning, highlighting the key challenges ML brings, strategies for effective model evaluation and finally tried to outline a roadmap for developing the necessary skills to excel in ML model testing. author: snandal From 274bec84d1bfe2a3c5bb5c858b015e8d8130863a Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 14:20:14 +0100 Subject: [PATCH 04/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...h-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index 0f916d6d59..30c937eabf 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -18,12 +18,6 @@ author: snandal image: --- - - -Recently worked on developing Machine learning skill definitions and L&D pathway. In this blog post I have tried to explore the evolving role of test engineers in the era of machine learning, highlighting the key challenges ML brings, strategies for effective model evaluation and finally tried to outline a roadmap for developing the necessary skills to excel in ML model testing. - Machine learning (ML) is no longer just a concept for the future, it is now a vital part of many everyday applications. From personalised recommendations on streaming services to detecting fraud in banking and from self-driving cars to advancements in healthcare, ML is changing the way industries work. For a test engineer, the increasing use of ML brings new challenges and opportunities. As ML technology advances, the role of a test engineer also changes. Unlike traditional software, which gives consistent and predictable results, ML models work with probabilities and patterns, making testing and evaluation more complex. In this blog we will try to look at how ML has developed, how it might affect our job as test engineers and the important strategies, considerations and skills needed to effectively evaluate and test ML models. We will also try to draw a roadmap to help us grow professionally in this area. @@ -32,7 +26,7 @@ In this blog we will try to look at how ML has developed, how it might affect ou Machine learning has significantly advanced since its early days. Initially, ML models were fairly simple, handling tasks like basic predictions or classifications. However, with improvements in computing power, access to large amounts of data and more complex algorithms, ML has grown into a powerful tool capable of handling sophisticated tasks like natural language processing and images recognition. This progress has made ML widespread across industries, leading businesses to increasingly rely on data-driven decisions. As companies embrace ML, the need for thorough and reliable testing of these models has become more important, which directly affects the role of test engineers. -Before we get into the testing process, it is important to understand what makes ML models different and what impact they can have on our job as a test engineer. +Before we get into the testing process, it is important to understand what makes ML models different and **what impact they can have on our job as a test engineer.** At their core, ML models learn from data. They are trained on large datasets to recognise patterns and make predictions or decisions based on new information. Unlike traditional software, where rules are clearly programmed, ML models develop their logic through training, making their behaviour less predictable and sometimes harder to understand. Traditionally, a test engineer's job involved testing systems where a given input always produced the same output, making results predictable. However, with ML, this approach changes. ML models are probabilistic, meaning their outputs can vary depending on the training data and some built-in randomness. This shift has significantly changed how software testing is done. We now need to do more than just check if a system works as expected. We must also evaluate how well a model performs, whether it's reliable and fair. This requires combining traditional testing skills with additional knowledge specific to ML. From 22040aab753b8f2789dfb2b625a827a116e57ad3 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 14:23:09 +0100 Subject: [PATCH 05/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...ng-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index 30c937eabf..a79fd4c20e 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -12,7 +12,7 @@ tags: - Testing - Machine Learning - Model Evaluation -summary: Recently worked on developing Machine learning skill definitions and L&D pathway. In this +summary: Having worked on developing Machine Learning skill definitions and L&D pathway recently, in this blog post I have tried to explore the evolving role of test engineers in the era of machine learning, highlighting the key challenges ML brings, strategies for effective model evaluation and finally tried to outline a roadmap for developing the necessary skills to excel in ML model testing. author: snandal image: From f8ed663f266e789e029c6b960e92880bda76ed91 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 14:24:25 +0100 Subject: [PATCH 06/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...th-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index a79fd4c20e..d4517441d9 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -2,16 +2,15 @@ title: Evolving with AI From Traditional Testing to Model Evaluation I date: 2024-09-01 10:00:00 Z categories: -- Artificial Intelligence - Machine Learning - Testing - Model Evaluation -tags: - Artificial Intelligence -- AI -- Testing +tags: - Machine Learning +- Testing - Model Evaluation +- Artificial Intelligence summary: Having worked on developing Machine Learning skill definitions and L&D pathway recently, in this blog post I have tried to explore the evolving role of test engineers in the era of machine learning, highlighting the key challenges ML brings, strategies for effective model evaluation and finally tried to outline a roadmap for developing the necessary skills to excel in ML model testing. author: snandal From 5a76b41a3659cb69e7d6a057c1a8495a27cebe34 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 14:31:56 +0100 Subject: [PATCH 07/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...aditional-Testing-to-Model-Evaluation-I.md | 111 ++++++++++++------ 1 file changed, 78 insertions(+), 33 deletions(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index d4517441d9..e50947231c 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -18,90 +18,135 @@ image: --- Machine learning (ML) is no longer just a concept for the future, it is now a vital part of many everyday applications. From personalised recommendations on streaming services to detecting fraud in banking and from self-driving cars to advancements in healthcare, ML is changing the way industries work. + For a test engineer, the increasing use of ML brings new challenges and opportunities. As ML technology advances, the role of a test engineer also changes. Unlike traditional software, which gives consistent and predictable results, ML models work with probabilities and patterns, making testing and evaluation more complex. -In this blog we will try to look at how ML has developed, how it might affect our job as test engineers and the important strategies, considerations and skills needed to effectively evaluate and test ML models. We will also try to draw a roadmap to help us grow professionally in this area. + +In this blog we will try to look at how ML has developed, how it might affect our job as test engineers and the important strategies, considerations and skills needed to effectively evaluate and test ML models. +We will also try to draw a roadmap to help us grow professionally in this area. ## How Machine Learning Has Evolved Machine learning has significantly advanced since its early days. Initially, ML models were fairly simple, handling tasks like basic predictions or classifications. However, with improvements in computing power, access to large amounts of data and more complex algorithms, ML has grown into a powerful tool capable of handling sophisticated tasks like natural language processing and images recognition. + This progress has made ML widespread across industries, leading businesses to increasingly rely on data-driven decisions. As companies embrace ML, the need for thorough and reliable testing of these models has become more important, which directly affects the role of test engineers. + Before we get into the testing process, it is important to understand what makes ML models different and **what impact they can have on our job as a test engineer.** + At their core, ML models learn from data. They are trained on large datasets to recognise patterns and make predictions or decisions based on new information. Unlike traditional software, where rules are clearly programmed, ML models develop their logic through training, making their behaviour less predictable and sometimes harder to understand. + Traditionally, a test engineer's job involved testing systems where a given input always produced the same output, making results predictable. However, with ML, this approach changes. ML models are probabilistic, meaning their outputs can vary depending on the training data and some built-in randomness. This shift has significantly changed how software testing is done. + We now need to do more than just check if a system works as expected. We must also evaluate how well a model performs, whether it's reliable and fair. This requires combining traditional testing skills with additional knowledge specific to ML. ## Looking at the unique challenges that testing ML models brings -1. Unpredictable Outputs: Unlike regular software, where the same input always gives the same result, ML models can give different results because they work with probabilities. This makes testing harder because instead of just checking if the output is correct, we need to evaluate how accurate and reliable the model’s predictions are. -2. Dependence on Data: The quality of an ML model depends a lot on the data it was trained on. Testing ML models means we need to deeply understand the training data and its weaknesses because any bias or missing information in the data can lead to incorrect results. -3. Performance Metrics: Traditional testing metrics like pass/fail don’t apply directly to ML models. Instead, we use statistical metrics like accuracy, precision, recall, F1 score and AUC-ROC to evaluate model performance. Understanding and choosing the right metrics is key to effective testing. -4. Model Degradation over time: As new data becomes available; an ML model’s performance can degrade causing a problem known as model drift. This means continuous monitoring and retraining are equally important, adding complexity to the testing process. + +**1. Unpredictable Outputs:** Unlike regular software, where the same input always gives the same result, ML models can give different results because they work with probabilities. This makes testing harder because instead of just checking if the output is correct, we need to evaluate how accurate and reliable the model’s predictions are. + +**2. Dependence on Data:** The quality of an ML model depends a lot on the data it was trained on. Testing ML models means we need to deeply understand the training data and its weaknesses because any bias or missing information in the data can lead to incorrect results. + +**3. Performance Metrics:** Traditional testing metrics like pass/fail don’t apply directly to ML models. Instead, we use statistical metrics like accuracy, precision, recall, F1 score and AUC-ROC to evaluate model performance. Understanding and choosing the right metrics is key to effective testing. + +**4. Model Degradation over time:** As new data becomes available; an ML model’s performance can degrade causing a problem known as model drift. This means continuous monitoring and retraining are equally important, adding complexity to the testing process. ## Model Evaluation Vs. Model Testing Before we dive deep into discussing possible strategies for ML model testing, it is important to understand the difference between Model evaluation and Model testing. + Model evaluation and testing might sound similar, but there is a key difference between the two. Evaluation focuses on assessing the model’s performance in a controlled environment, while model testing checks how well the model works in real-world conditions. -Let’s look at it with a simple example of a spam detection model that filters out unwanted emails. During the model evaluation phase (validation mode), we will use a labelled dataset of emails to calculate metrics like accuracy, precision and recall. For example, the model might successfully identify 95% of spam emails in this dataset. -After evaluation, we will test (prediction mode) the model with real incoming emails to see how well it handles a variety of messages, if it can correctly filter out spam in different languages, formats, or tricky emails that look legitimate. Model testing makes sure the model works well in the real environment where it will be used, way beyond the controlled conditions of evaluation. +Let’s look at it with a simple example of a spam detection model that filters out unwanted emails. + +During the model evaluation phase (validation mode), we will use a labelled dataset of emails to calculate metrics like accuracy, precision and recall. For example, the model might successfully identify 95% of spam emails in this dataset. + +After evaluation, we will test (prediction mode) the model with real incoming emails to see how well it handles a variety of messages, if it can correctly filter out spam in different languages, formats, or tricky emails that look legitimate. Model testing makes sure the model works well in the real environment where it will be used, way beyond the controlled conditions of evaluation. ## Strategies for Testing ML Models To effectively address the challenges that ML brings, test engineers can benefit from the following strategies: -1. Define Clear Objectives: Start by defining what success looks like for your ML model. Are you aiming for high accuracy? or is it more important to avoid mistakes like false positives? Understanding the business objectives and how they translate into model performance metrics is essential. -2. Validate the Data: Ensure that the training, validation and testing datasets are represent real-world situations the model will face. Check for issues like missing values, duplicates and biases. Proper data validation is very important to prevent garbage-in, garbage-out scenarios. -3. Apply Cross-Validation: Implement cross-validation techniques to check that the model’s performance is consistent across different subsets of the data. This helps in identifying overfitting, where the model performs well on training data but poorly on new unseen data. -4. Analyse Performance Metrics: Use a combination of metrics to get a comprehensive view of the model’s performance. For classification models, confusion matrices can be useful in providing insights into where the model is making errors. For regression models, metrics like Mean Squared Error (MSE) or R-squared can offer valuable information. -5. Perform A/B Testing: Compare the ML model’s performance with existing solutions (if available) using A/B testing. This helps in understanding whether the new model provides a significant improvement over the current system. -6. Perform Stress Testing: Subject the model to edge cases and scenarios with outliers to see how it handles unexpected inputs. This can uncover vulnerabilities and help in making the model more robust. -7. Continuous Monitoring and Feedback Loop: After deployment, continuously monitor the model’s performance. Set up a feedback loop where new data is used to retrain and improve the model. Tools like MLflow can help in managing this process efficiently. + +**1. Define Clear Objectives:** Start by defining what success looks like for your ML model. Are you aiming for high accuracy? or is it more important to avoid mistakes like false positives? Understanding the business objectives and how they translate into model performance metrics is essential. + +**2. Validate the Data:** Ensure that the training, validation and testing datasets are represent real-world situations the model will face. Check for issues like missing values, duplicates and biases. Proper data validation is very important to prevent garbage-in, garbage-out scenarios. + +**3. Apply Cross-Validation:** Implement cross-validation techniques to check that the model’s performance is consistent across different subsets of the data. This helps in identifying overfitting, where the model performs well on training data but poorly on new unseen data. + +**4. Analyse Performance Metrics:** Use a combination of metrics to get a comprehensive view of the model’s performance. For classification models, confusion matrices can be useful in providing insights into where the model is making errors. For regression models, metrics like Mean Squared Error (MSE) or R-squared can offer valuable information. + +**5. Perform A/B Testing:** Compare the ML model’s performance with existing solutions (if available) using A/B testing. This helps in understanding whether the new model provides a significant improvement over the current system. + +**6. Perform Stress Testing:** Subject the model to edge cases and scenarios with outliers to see how it handles unexpected inputs. This can uncover vulnerabilities and help in making the model more robust. + +**7. Continuous Monitoring and Feedback Loop:** After deployment, continuously monitor the model’s performance. Set up a feedback loop where new data is used to retrain and improve the model. Tools like MLflow can help in managing this process efficiently. ### Let’s look at the practical application of the above strategies with a case study of an ML model predicting patient re-admission after discharge Consider a hospital implementing an ML model to predict the likelihood of patients being readmitted within 30 days of discharge. -1. Define Objectives: The goal is to identify most high-risk patients so that the hospital can take proactive measures to reduce readmission rates and improve patient care. -2. Data Validation: Start by validating the dataset, ensuring that it includes a balanced representation of various patient, medical history, treatment plans etc. Use cross-validation to train the model and select the most effective algorithm for predicting readmissions. -3. Metric Selection: Choose sensitivity/recall and specificity as key metrics. High sensitivity ensures that most high-risk patients are identified, while specificity ensures that the model doesn't over-predict readmissions, leading to unnecessary interventions. -4. Stress Testing: Test the model with scenarios where patient data is incomplete, such as missing medical history or inconsistent treatment records. Assess how the model performs when handling these edge cases, which are common in real-world medical data. -5. A/B Testing: Implement the ML model alongside the existing manual risk assessment process. Compare the model's predictions with the manual predictions in real-time, focusing on how well the model improves patient outcomes. -6. Post-Deployment Monitoring: After deployment, continuously monitor the model’s performance, looking for signs of model drift as new patient data becomes available. Implement a feedback loop where the model is retrained regularly with the latest data to maintain accuracy and relevance. + +**1. Define Objectives:** The goal is to identify most high-risk patients so that the hospital can take proactive measures to reduce readmission rates and improve patient care. + +**2. Data Validation:** Start by validating the dataset, ensuring that it includes a balanced representation of various patient, medical history, treatment plans etc. Use cross-validation to train the model and select the most effective algorithm for predicting readmissions. + +**3. Metric Selection:** Choose sensitivity/recall and specificity as key metrics. High sensitivity ensures that most high-risk patients are identified, while specificity ensures that the model doesn't over-predict readmissions, leading to unnecessary interventions. + +**4. Stress Testing:** Test the model with scenarios where patient data is incomplete, such as missing medical history or inconsistent treatment records. Assess how the model performs when handling these edge cases, which are common in real-world medical data. + +**5. A/B Testing:** Implement the ML model alongside the existing manual risk assessment process. Compare the model's predictions with the manual predictions in real-time, focusing on how well the model improves patient outcomes. + +**6. Post-Deployment Monitoring:** After deployment, continuously monitor the model’s performance, looking for signs of model drift as new patient data becomes available. Implement a feedback loop where the model is retrained regularly with the latest data to maintain accuracy and relevance. ## Skills Needed for ML Model Testing + To effectively perform model evaluation, a test engineer would need a combination of technical, analytical and domain-specific skills: -1. Understanding of Machine Learning Concepts + +**1. Understanding of Machine Learning Concepts** • Understanding fundamental ML concepts such as model learning algorithms – supervised learning, unsupervised learning and reinforcement learning, model performance issues like overfitting and underfitting and the types of models (e.g., regression, classification, clustering). • Familiarity and understanding of model evaluation metrics such as accuracy, precision, recall, F1 score, AUC-ROC, mean squared error, R-squared, etc. and knowing when and how to apply them. -3. Statistical Knowledge + +**2. Statistical Knowledge** • Understanding statistical measures - mean, median, variance, standard deviation and concepts like hypothesis testing, p-values, confidence intervals and correlation. • Knowledge of probability distributions and how they apply to model outputs and uncertainties. -4. Programming Skills + +**3. Programming Skills** • Proficiency in one of the scripting languages - Python or R, which are commonly used for ML model evaluation. Familiarity with libraries like Scikit-learn, TensorFlow or PyTorch is advantageous. • Ability to write scripts to automate the evaluation of models, including running tests across different datasets and tracking performance over time. -5. Data Analysis and Validation + +**4. Data Analysis and Validation** • Skills in data cleaning, handling missing data, outliers, data normalisation and feature engineering. • Ability to validate datasets for biases, errors and representativeness, ensuring that the training data is of high quality and relevant to the model’s objectives. • Proficiency in using tools like Python’s Pandas, Matplotlib and Seaborn for EDA to understand the dataset before feeding it into an ML model. -6. Familiarity with common ML Tools and Platforms + +**5. Familiarity with common ML Tools and Platforms** • Experience with tools like Scikit-learn for traditional ML, or TensorFlow/PyTorch for deep learning models. • Familiarity with tools like MLflow, which assist in tracking experiments, managing models and automating the retraining process. • Knowledge of using version control systems like Git for managing code and model versions. -6. Knowledge of using version control systems like Git for managing code and model versions. + +**6. Knowledge of using version control systems like Git for managing code and model versions.** By combining these skills along with domain knowledge of the business context in which the model will be deployed, it can be effectively evaluate and tested, ensuring that it meets both technical and business requirements while remaining reliable, accurate and fair. ## Roadmap for Developing ML Model Testing Skills Here’s a step-by-step roadmap for a test engineer to develop ML model testing skills: -1. Start with Basic ML Theory: Learn the fundamental concepts of ML, focusing on the types of learning, model evaluation metrics and common challenges. -2. Build Statistical Knowledge: Study key statistical principles and how they relate to ML. This is essential for understanding model performance. -3. Learn Data Analysis: Practice data preprocessing, cleaning and validation using tools like Python’s Pandas or R. Data quality is critical for effective ML models. -4. Develop Programming Skills: Get comfortable with Python or R. Start by running simple ML models using libraries like Scikit-learn and gradually move to more complex tasks. -5. Explore ML Tools: Familiarise yourself with popular ML frameworks like TensorFlow, Keras, or PyTorch and model management tools like MLflow. -6. Practice Model Evaluation: Apply your skills by testing real ML models. Use cross-validation, stress testing and performance metrics analysis to assess models. -7. Continuous Learning: Stay updated with the latest trends in ML and software testing. Engage in online courses, webinars and forums to keep your knowledge fresh. + +**1. Start with Basic ML Theory:** Learn the fundamental concepts of ML, focusing on the types of learning, model evaluation metrics and common challenges. + +**2. Build Statistical Knowledge:** Study key statistical principles and how they relate to ML. This is essential for understanding model performance. + +**3. Learn Data Analysis:** Practice data preprocessing, cleaning and validation using tools like Python’s Pandas or R. Data quality is critical for effective ML models. + +**4. Develop Programming Skills:** Get comfortable with Python or R. Start by running simple ML models using libraries like Scikit-learn and gradually move to more complex tasks. + +**5. Explore ML Tools:** Familiarise yourself with popular ML frameworks like TensorFlow, Keras, or PyTorch and model management tools like MLflow. + +**6. Practice Model Evaluation:** Apply your skills by testing real ML models. Use cross-validation, stress testing and performance metrics analysis to assess models. + +**7. Continuous Learning:** Stay updated with the latest trends in ML and software testing. Engage in online courses, webinars and forums to keep your knowledge fresh. + By following this roadmap, test engineers can successfully transition into ML model testing, ensuring they stay relevant in an evolving field. ## Conclusion Testing and evaluating ML models requires a mix of traditional testing skills and an understanding of statistical methods. With the right testing strategies and by following a structured approach to testing, incorporating data validation, metric analysis, stress testing and continuous monitoring, we can ensure that our ML model provides real value and meets business objectives. + Additionally, staying informed about the latest trends and tools can help handle the challenges of ML testing effectively and make a significant impact on your organisation’s success. Machine learning is complex but also an exciting opportunity to explore new dimensions of software quality. Let’s embrace these challenges as a chance to grow and expand our skills in field of software testing. From 22847f6f8e1652a1667bd346cdae2332578f2dd5 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 15:40:03 +0100 Subject: [PATCH 08/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...aditional-Testing-to-Model-Evaluation-I.md | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index e50947231c..d6b1799c79 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -40,25 +40,25 @@ We now need to do more than just check if a system works as expected. We must al ## Looking at the unique challenges that testing ML models brings -**1. Unpredictable Outputs:** Unlike regular software, where the same input always gives the same result, ML models can give different results because they work with probabilities. This makes testing harder because instead of just checking if the output is correct, we need to evaluate how accurate and reliable the model’s predictions are. +**Unpredictable Outputs:** Unlike regular software, where the same input always gives the same result, ML models can give different results because they work with probabilities. This makes testing harder because instead of just checking if the output is correct, we need to evaluate how accurate and reliable the model’s predictions are. -**2. Dependence on Data:** The quality of an ML model depends a lot on the data it was trained on. Testing ML models means we need to deeply understand the training data and its weaknesses because any bias or missing information in the data can lead to incorrect results. +**Dependence on Data:** The quality of an ML model depends a lot on the data it was trained on. Testing ML models means we need to deeply understand the training data and its weaknesses because any bias or missing information in the data can lead to incorrect results. -**3. Performance Metrics:** Traditional testing metrics like pass/fail don’t apply directly to ML models. Instead, we use statistical metrics like accuracy, precision, recall, F1 score and AUC-ROC to evaluate model performance. Understanding and choosing the right metrics is key to effective testing. +**Performance Metrics** Traditional testing metrics like pass/fail don’t apply directly to ML models. Instead, we use statistical metrics like accuracy, precision, recall, F1 score and AUC-ROC to evaluate model performance. Understanding and choosing the right metrics is key to effective testing. -**4. Model Degradation over time:** As new data becomes available; an ML model’s performance can degrade causing a problem known as model drift. This means continuous monitoring and retraining are equally important, adding complexity to the testing process. +**Model Degradation over time** As new data becomes available; an ML model’s performance can degrade causing a problem known as model drift. This means continuous monitoring and retraining are equally important, adding complexity to the testing process. ## Model Evaluation Vs. Model Testing Before we dive deep into discussing possible strategies for ML model testing, it is important to understand the difference between Model evaluation and Model testing. -Model evaluation and testing might sound similar, but there is a key difference between the two. Evaluation focuses on assessing the model’s performance in a controlled environment, while model testing checks how well the model works in real-world conditions. +Model evaluation and model testing might sound similar, but there is a key difference between the two. Evaluation focuses on assessing the model’s performance in a controlled environment, while model testing checks how well the model works in real-world conditions. Let’s look at it with a simple example of a spam detection model that filters out unwanted emails. -During the model evaluation phase (validation mode), we will use a labelled dataset of emails to calculate metrics like accuracy, precision and recall. For example, the model might successfully identify 95% of spam emails in this dataset. +During the **model evaluation** phase (validation mode), we will use a labelled dataset of emails to calculate metrics like accuracy, precision and recall. For example, the model might successfully identify 95% of spam emails in this dataset. -After evaluation, we will test (prediction mode) the model with real incoming emails to see how well it handles a variety of messages, if it can correctly filter out spam in different languages, formats, or tricky emails that look legitimate. Model testing makes sure the model works well in the real environment where it will be used, way beyond the controlled conditions of evaluation. +After evaluation, during **model testing** phase (prediction mode) the model will be subjected to real incoming emails to see how well it handles a variety of messages, if it can correctly filter out spam in different languages, formats, or tricky emails that look legitimate. Model testing makes sure the model works well in the real environment where it will be used, way beyond the controlled conditions of evaluation. ## Strategies for Testing ML Models @@ -82,17 +82,17 @@ To effectively address the challenges that ML brings, test engineers can benefit Consider a hospital implementing an ML model to predict the likelihood of patients being readmitted within 30 days of discharge. -**1. Define Objectives:** The goal is to identify most high-risk patients so that the hospital can take proactive measures to reduce readmission rates and improve patient care. +**Defining Objectives:** The goal here is to identify most high-risk patients so that the hospital can take proactive measures to reduce readmission rates and improve patient care. -**2. Data Validation:** Start by validating the dataset, ensuring that it includes a balanced representation of various patient, medical history, treatment plans etc. Use cross-validation to train the model and select the most effective algorithm for predicting readmissions. +**Data Validation:** Further start by validating the dataset, ensuring that it includes a balanced representation of various patient, medical history, treatment plans etc. Use cross-validation to train the model and select the most effective algorithm for predicting readmissions. -**3. Metric Selection:** Choose sensitivity/recall and specificity as key metrics. High sensitivity ensures that most high-risk patients are identified, while specificity ensures that the model doesn't over-predict readmissions, leading to unnecessary interventions. +**Metric Selection:** Choose sensitivity/recall and specificity as key metrics. High sensitivity ensures that most high-risk patients are identified, while specificity ensures that the model doesn't over-predict readmissions, leading to unnecessary interventions. -**4. Stress Testing:** Test the model with scenarios where patient data is incomplete, such as missing medical history or inconsistent treatment records. Assess how the model performs when handling these edge cases, which are common in real-world medical data. +**Stress Testing:** Test the model with scenarios where patient data is incomplete, such as missing medical history or inconsistent treatment records. Assess how the model performs when handling these edge cases, which are common in real-world medical data. -**5. A/B Testing:** Implement the ML model alongside the existing manual risk assessment process. Compare the model's predictions with the manual predictions in real-time, focusing on how well the model improves patient outcomes. +**A/B Testing:** Implement the ML model alongside the existing manual risk assessment process. Compare the model's predictions with the manual predictions in real-time, focusing on how well the model improves patient outcomes. -**6. Post-Deployment Monitoring:** After deployment, continuously monitor the model’s performance, looking for signs of model drift as new patient data becomes available. Implement a feedback loop where the model is retrained regularly with the latest data to maintain accuracy and relevance. +**Post-Deployment Monitoring:** After deployment, continuously monitor the model’s performance, looking for signs of model drift as new patient data becomes available. Implement a feedback loop where the model is retrained regularly with the latest data to maintain accuracy and relevance. ## Skills Needed for ML Model Testing From b73e2b43800d6bfd6be5873fc95d43eb80f565f3 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 15:53:14 +0100 Subject: [PATCH 09/54] Add files via upload --- ...raditional Testing to Model Evaluation I.png | Bin 0 -> 52902 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 _uploads/Evolving with AI From Traditional Testing to Model Evaluation I.png diff --git a/_uploads/Evolving with AI From Traditional Testing to Model Evaluation I.png b/_uploads/Evolving with AI From Traditional Testing to Model Evaluation I.png new file mode 100644 index 0000000000000000000000000000000000000000..33d05c19694ed4fec0c055f33a340055b6010294 GIT binary patch literal 52902 zcmce-Ra6{X7d46n4H_&sGz52oOK^7rjk^b@aS0Y6xI=Ic?$EeHaHm6Xmqvm+{C&=M z#y{@seYp3bM|bt8+EumoT5Hca*V>V4Dzcd9B=?~iPS*_%4g>d}4}747aRMCNq>#Ln zxR#gEaW;wxzBJLWu@6JOCyv&41_tF%gTV|89?!>^7~jp#I*IV#5VV`Y69{<>tidJF zNDm-H%=#))iQG<{tvc3wuF$3Ju)WzU9p1gUX&#@Pe7f!AbK3Glt^=|VMh%+PCI9f+ zUp4#Rk%mHLeEt9aX9Q@=>Hc>HE_2cUegQ`#2sHk^3dmwPfX?{O^+1+XF0-)Ye}54w zzOR!Q{oi24(Lid;|Av+Me+q5ViAiUpP=)BJYgvn|ER}3)Se|`?$5-raqX&&TM$`@*-)Mn(h7RM@aeYLm64| z**NUoVrlR~6s`5CP9pcSU79SV+S(dOOQP$V zs3_iBii@CyICtTkTkekx@s-FW@@ymp=KnWU1b}A3GC}2jC>5XxU7I!eN9hXd$m5kK z;lmQgk?+X3Aq^TGQif!fz?SiRjz2z8_8{SCb%3|#ADCGP5`-|g-snHSoTo$LU`;vzFe;;3w z!Hm}q8;x5ZKbm6UwQrDv8C}n>uhOo_pPt5xA@a&TXL{fq*G&*n3%Zb_s*a9 z%Wo=7Qr`LQC1VedW?DLe;>y~8!$U(#iB%&p*ZK-eaDq8vtH`9^wG6Z;xg5#x`)=F< zkT8DMM@FLk69JIlJVIkTeMWW2@8l&#W4H4wVYbWPS70jwuru&Zbsq8+9~`p<^MPufeWDJOqQPcpG)BBQxkkSreU+2Q;=V&YA?DHUD5?g^)7yqf8F$y z$77*cCLP657pRwUwJbb#oikZ_&dj8(_81|5Za?c{h8TKgTUc3*w(jRJ$B91t6n^^K zf6@K=tQt0wCmymBf-UO2h-EQXqWr*wR%w{Y=j0*!Mx#`P+#j2e4(QkOEJ}8Me$M1v zi+}DZ@7%E&Tg+eUXPVdXe6zd@8$QXty0zplu^|^1m;17}cZA`;S;WK9jHw8H7LpN& zuJwsghIDgLk(^`xRUjf}N+of;GEmZjD~|hhu!5Pr&f`bp>t9FP8?^+SN2_NUZ-+zl z{ka=I*xWw6>?Em9oc^q?WKl#vp z+E$?#YUFcBx34cY@NU)rcJ-*KoY&`;<}IcI8~q58w^sS8_@GLq#|lsBsBy}P;M>Y(ZM=~Mka=Y zZkRH#BZm@dQVg9djT@4t<~8=e{(Lx77>=%%mK*q@8!P%?vjn?8XxaC;F5DpCewV58 z--NrQ8(AmKw0_?o=d}Dm-sX?Ch7Z|33z|j$_PTw1>y&wn#HY=pWu`#I?CM9kVedGF zKj)A9>kk6vqc!y(gof`MMI4EN49pYp)RINI4OUFwad0`@57b%@OEN8v=F6Cfy*5)x zg}rLqIt^3=cJX2J?Ynck9+=j7^G8v72wUta4~ko>{AH#{KFNJG6e+M^2Kb3g^Y!I< zf#7yOH*hq^_q1%duS-?z+1-05K{o57^rXN(S?f`G#m;t|SZW5hb-DB7#TdKgXforG z6|GUU8_Yt+yX%3k3+sL__xr9r@F>4R(cWntt5eu7Rwk6@2IvKDr)K_}p@hhhGKarA z7>h#nJzcohhlH^HI&?s(*4r?S9-IAzSMGV^St+NkCz9X!U0VFy2emp?zrC4=-}WD4 z^B>%2cGkXeK3oy=zc?oC(#>q>_q|JgZoB*87R=Fe8d6_xub~jg+1_|Y;aGI- z?37H$fpywagEiC@@GFZhU$?`nuEw_3Q+x(=sO?^)$h8`I&4oHPv z`u_w*eFLsO#KltGU6Xspg0a(%nYokg9T+vwM-Iki2E{}upaL{-xPh4zfln^? zdznsI#R1M$xPCE0Z!2*6dOA3G`i5Gg!AhBc3by3US&pT$&UvzTKi_0_wG-0F z%!@m3sp=HfP9!=*FbP?K3vvgMP0aIk;IX5HiqhV*=-K{Jsd$l8Ff7ID48@R^7NXS6 z2MOWD)w>zy1im_q3Ptd(c}Y%1I{XWG>$&mtAr$B8-U4`O8u}Vo6gw<$X}2%eEyogD zj}|4-9g+fz)YtoqCuH+AX^ZxE$2iwX6GE0bIK>TpL5X;DqS1BX{FEzRJZ0kC`YblMRLX)66J4ZG2jl7 zdO({K>70`5hAZwE(T&WEXG!+Bef0x%m#(Xh17G0*n6LNyl+UDHcUw_?;8OEg0hy%4 zi$1zxmsm`Z*%&y7eNj6v7I!yU%Po4<%!t*832nDB(SJk~2i#KgX(yK8@0EcmEwpKkov3 zm~!sI!~~2hS;xuC8q07TqJrn==ld+phoW5;Z|>Ko)(73Y`){O<&WjiK zvHV*ReY@dgt`Rk!XKP`(XKhe6Yt~$+tAm-aG5dw`T~Z#Kcs4$XK}d(Uo8cPR#oV9s z2Sr4CJTUaPACU~9Nbm%*_N<&<3$%%;eg%0vU{5pNEnVu#@ZdVGSS_Q`%BMT8^Se?3Lr1 z)CkF*E+jcn;z2nt+*KGpk@PMdhx`Ad#)=k!(}O9=?oWi8LqRqezfNU zN87=JXgZ{h;;wACjWNalakKnk&4oPrijhR4)x$3sGHU@oZwFl8Q33 z_yEtfqqw1QDN(IY5d{6;J&x;!kA8~E&~asBRjb7&`EMXt<_`Q&ul|>d97hj0&Tc&! zEP3g(%iGM$92vakH*mxk(~Gg&ufNVdeWq$s3STuHdNwWf)6LDKj}g7=`_l5MNT!?^ zwEb%4)@CqQ*bu$qi~VY?7)f;?r(5fFZU#D>$Pv+R=s*Pg{W6~P@2Z}F+d#}{?3jxG zMVG$Jf66V@>(|9dB6n^I=qyQyNmb*hDgH+Dl+k0Y0}zpiWS~A~5#9OBt)CI&xRxFO@q@x6@Itvzyub^N672yke>R+7Ng%u@Pr?7akv;qE?OBn@7>WGSE01-g+~SpG{oMd= zV(XJwWD>2sQhBl4J{{<{sQiA%#G&idS>Tv&RO%g3(Zbp!Uh$W)aX3gBo0%?=-q9-@1ugZ-YWuCnA@;x7pblU4ICw(ji`O2?{aHPI%vYLoTJ9rDg<(pH1QHY! z6`_lr;K1F~*1*w4d)~xtcp{9;@SZZ&9w4}g+X`=#gNT8-=f7Ca!~dJj1tbTN9|U7W6%cfk|ZQsy#5q6 z%g0gbkdryo%j93_@Ou?n7odR-Z(}8ekSt!-nE$6Jzz<9|r6sZ3pFT-xi@hWiHGiQy z9~ntxCN0{{-dD)xeuAICXmLym4>YFQ)~QUDVPu?b4j{M9Nv17}M4KJBG!k0+#c>&| z+}c|2r=ME+YdmpRcd1@@u7lSh0k^a57w1I!)$jl4wOK@V{3vt$)5pWa-!xC5%}`fY zT6#WJy?G|@>ej?xj12KR9Pgld|Fh#jFnVOde;UdXZrum6|BUhfN+bFINP7AIb6B~o zsf{NG*leW|OSrXzpxR4dm%^_7{e49ViQxH4T}?YkH7n?7K4D-$%Gx1nP=oyEjfCCd zY%v$;NNZp7J`Wx`ogV_@StGg{&%c{tItmO2)H|*U!U7EjUy2md*y_jF+W8gDB(=3u zpKg|&F3!4N1r?=)&~7@R)$K>?slPvkKf&0{7>r^udV70MY9y?Bt{eU%0mv&TxTbzl zwX!N%OoI_LTSK%De0(}qQ#s_$hOIg`f9%;|7}c=Y7y^U(V~w_Rv;aWAu7*3i=kf14 zt>Kn0OI$E&6T2qMHZn3&f7bP=07E3%be18mSul#=cfFwF6US!M0jzImp!@joGfY+} zgdD+`>7u_br)8ZM07qjX0|uE%sfkl`VEV*atk9`*vw_Gx>;?S%AFj~&YhdcdJq)Sr zSK8njKe<+F@+gzsYU^o#9oHt#zQqcJ709 z)hAb1SHA|UiHlYDIpr=G+a9u}P{?vZ_-6*rf-%@of3w|=!f5dhH_fQH+`;zbbVb8lMbt zl)xA;+o^)YJZu#%E?{VebcrxtMFPI2nF1*SkG@%lHfZ`Kvm4ElloQiu?uR zK{eekdTZYMzID~I{-KF7`(rtgP*J?yY*D1B%bafV-bbr( zVu6#!xvVOng5vYT^)4*-uo*N3Bl@w$R@#`&loS-cXSvVHU;MGJd6WFQ(HjDiO2-IB z;7jLAZ)>}N@%u6cmdUM9f`d$_E;Oi01p|&cQffO+Pr%<^6nkfZ5Kf<~18NisuRI=` z=^oww`i8{rkNPENzDZAfd;aorWz-)@TM~c7=qb2nKh)oZ8(Y{xm@Su|EGPdAHf$uO zS+0=-VmQW9Nn(CLJ?wbDP`?jkOE|P%rDZmLTGVau(vEpn4|wktQ}o)bX2)x@w;t<# z&xK*}vZ^}jBrCR6smt}ca+NyXwqrD++o~HEb~11CTd6*+Ce3)NQ}d<$OwxMds2_{5 z$ZnyW4u#x3KFf8KvDhcuYb$cWXTZp^|DZ5O#1}c4G20iWHJ(It5=-81(G+0$j^$#t zfpL8o2r88a#5fh|_;GXx)Hb^+6`VWeKqhAxiX4jH{28FH830hGdvfR>#+NXck_Hv| zy=+80n8e;bQ;gL(1=LsaQxb#^g zllO`lN2G-RvbAnr$(Al~{l1`Riiw2U0G4|XvAvG4nrU#YKf#^g%93reFpBWZgONH^ z+g)5-9tPw-*|@}0S`B_DIFfM@x4^;f_2oFWhjE%eoa5O-IxsZ_dxE>axJ;)e$0HuK z1C}22NqD~iD?#CTo7_{vnuZi?ReDW{oBiQq@2#`)CAS6GV`u0@MUAE^Z0AZQ6Q|V( zv(j!4b2cxshzN0;VW{42snIs+vMO*nnP^}9zSX%4oNCYkO^`9|4Sw@uG1wjbOVV?r z)4EZZc#Gre{LwF`?}d4JpT-h4kGZYJr!OpjitQ_SQ@H-37WB(CH*T zmUCj5Y$?(TTZIr;!eE;A;yOSB5ayfKCc9dE?6TbX?2&*G3XPh!G@i#bN*37q4()5Y zXr?4XycNBM$&fCl^N5qTUlFn5VGsVT(tM@iOVqLvJ zkDqWf)ORk-yx4Irz?7V}7?FK6M{uAsY$3*+wh38{a6ayEt`r(&zU8$7qd@-%bxFo5b2KNE|wE-`79(TMH;wh-%A)!RNa?p0wB(N=w4?U=5Rd zX03oAhNig}j4B8Qls*U44pN3T2NVH!rVd^m(-G5o(J*e@^>m^5j7eUy98ph6!Xh{j z@cb}>MxRCH|9G=7DTYsn#waK)W9R?+d>cm2))y`FT$1UC?rT)}c)y{NhanBI z6WW8P!vRX)W6t}7!niD_XL97_U9gKac;kW;Cdplnuu}oCUHSv=awgxWp~mo8p2gY~ zND7~ITs8(_(UjM z7E)9|PPl;i#ledK*m0nDTHH&B@d&LiL{MhMSvYp*F6C3dPQ1w^TID`vNAM2QjF~)lEL0b=R;TQGkd{ zw-@!L<_}AAt3i0wIIIlqFgzh8yVt5r9DM$rxqA8$HGh+BKg+S>T!!E~nWu}CqMZ5~ zY#|F;V}FY2kpZ@iS+3TfJ{W6d0VHLJJ_Gti>o$n)+{3w050CCf0$ zabJ@qdG#1f>n+E|T~VCJ?Ai{H@N%j0{|ie#7Do%ms^GUM~M!fgXp= zOuR20+~YhA*A2rAc66?R!w^M1${DBW$35{~7;%_G5r>!!wl+{6;olNq6AAOQK8pLD`^GMdP)ue#`uByR zwP8WV{L@!50Mzz!x_KG|IQhK`j`Jb2IP6B5Z{7JpcYk1r+))5~B2%3cgxITgh_~VU%bl z%&WJa^spZe>cT1;O`LzC?{b@qaeW&}pQE;ExbpCYlzILixekHxj;y1WT#6FvmhH)_ z;pxEABy>h2pi8=N-Apd?b;sI5p=qt&B8z4+11;RXwuYIcspgkuB9H$NgOif`RozN%A?U$Gno2649{ zGfQ_gRYg8-5wYP2BG;((?D)^C%f4~f;z8h16MjaKEETbxrFAedz_H-@yA$CeUOCNS z=5GpYTt=P21*}xJ9Wlk2Zra9`dT$YTSUHlAkZ>qH^^&{Do4C?jPr$0V2$NBzvLph6 z$0qzWlz~s2J(W{+GzAC03gI7nqgZgHBQF%t;TM!-L*~!@1c%;&ZLJ49x_O}Ss z!4d*%u!L;9iXNd}Nd$Cv=A^PJgg^hVW-EUHX5_1GAVM;z<#;27cFxjE_*Re)sX$-n zw`c)og)Oaf=Yg}Xn3msrM}iWqUggli*_b$V6cDi@BMk`=HXF-miqDA1J2L&DI&*0z zdH0l1<^o25(|%0+27j%TOE*sDcGWsithQ~IR|{#y8hpy&G@9JqsrCXETs>oa+**n4 zetD?SqY8P0p*`J?B~M?I|-85r2%fcQwl@0=$JPH+qY{d|3z1ns(XtI2><0gUvq zD2gQ5-+{&dRMSAOW=mV(4yx}qlHdSv6&gxjmq}TH|J9=GWCD=+=xI_eb6JH)!#ru{ zeJ?&$30UR+O=8Y<+_CH6%#i|fthNp%-N$#C3VY=X%aLW^J0yAI79h!q1BIZ<(v&XA zkG4n8_-`U@iVs79S_cO%{I#pc3s5?7jpD(LfXUgvJe02Bd6i}9=4#!#XU>c%eI53s z#)o$Kqy{W}V{w~JZ92}BX1#Y+jAr98$1tEa?9PNjbm_kT2rEeVvFbIV0wFv|T_Z{K zLiPlvBpB;qL`KB5juSsV`pdjUQc9?Y!PWKMG~-k5Z2e8263)x_@SDCcfafPuba+Kx z(lC1{ptQXw1gpw`$UXgIZ(*hXD^>&W^ip}QTuFp!JIqn)rg*-FL2FMaM11a)P2bH;cLXbz-*6E+bsI^`q|+Tli!k4tjpz$~9aUHy8w zZz+}K1S?EV5|; zmXZ0ET7g?HI*0b~aC1Uh%Ldtii6S=+YaL5Vc1s&T{(wsp+*3spV_1hod!O5uD0a1uOx}HU`>Ei%Q}=VpNv3bMjZifVs8>PPu4ATe z{9)V_UUR|OPswT9FpkIp07~MGx%SQoN6z(D!vLeK65X;4AFPFU%zfi?_`_o&B0A?t z=vn1qL1^^Y<`hhJ^r#Vg^BnKm?47;+sHgJR{{zZ>fY3Rvbyeb$@S)Q>pKHOH+u{Br z+veMK?6Uobt`yCQ=EsMCFFe1}?wu?&8Fnj|w=(}6GF`Syj-RQ{ocQJXr8*UmZ9Xc* zURQbF5?N1BjbAhW8oMRf*}1=HaCM&6#Ju2iv7&xj8(6&Jw|ZFJWu=gM1QJ2HVvjfGLT)m5pdE3%WiHq{{&xp_>JDro-H zyAHFGOL)X*Dyn;xpOPpR9wQT%UL6yaZtK{v7jVl!XDz|JA&oB+QZ%U$|2< zZ3Q}dCcs{!h{O!{cs_(cAX`MpSZM&9SE!eRw`Gm!?+uQ4^WAm#$tCjkY+Y>Qcdd!3 z154GLkHnSu^XfHpQ%$2_w$HI4^_#*gk8f%hox4Np+hj;@6v<0E80So)@YmZ38M{i4 zTagEYnA1Eq$W?PVkUdQZtbBp1sLrMSuKBCfa@h}YP099sZ3gt!l&QwS?VYFImFi$z zfJ+r_oRw7NBI6M;JMtq3)ra4GL!JX6Tz>~TvAu^iq6y|PZh0a0*2O0g4MoG5xAOgn z?FxQ+^#c%{t11PR{9gKRZG{S-*^A_$S+}9W#6>sH zhBd46T>`o4C2dd-dKn%Sf2K??RF7tD5G2_O?i;el^AsZ3;xz!q^{kkrB|~FT7xK3Y zHhe>)JR2#NdnVKgJ<+nU5vG>Cw%N8jY~l^a-<1rJ%w)8K_obt?c=7>KU>`%SPa$KT zz2unLWu$o+WXx=|XCPC3cOKMklXJKTZ6xhl=JLu~Y^10b^G`UNYk+h@24k~X1I`{! zA0e&*>UB56d)DBnJPY;+#T&^bE#xfv{$*Ua-oNAp{9gQ`5*)z0ROA6-HkUIS-n{L- z2OlUVC@K7nQ>#YmfVTJQ;ldj!xf-L+oxANgh~QW&O^JZ4P7$tHWW~fIj(GbZD%ctD zp|!>J zmd_+Pn~1dtje}#X-_B3PDu0iJ&h>6_pJ!1CpGgX~&Q~u3!Dtee$QXMed!((giL~Gq zVa8v)eGJ^P??U$+MzoryPdD9)hLcf>p)H;Y5X#d8ws&Xof{!hln@RQqnnpe)y6aCL zm~~o$m}j+UNL;9!g6{LYE9_2I6$HC|v)wb@YuxYmunuqLI%n^*?TclWhY~vzUxq7l zlTB`%n$2VL%Nr!o83#~%=iPaWy=Q(Sc`Esp6c)of=jYRWC02}CwwYPNeL6n*{r+L> zU1rN>-sO(x+V#PJJm6D5F;=s9;bDwOUY$69GTmv&xAgo);o>%DHdJmWH4fDlAY4Zb z2{}=W6Bz%kWsqx0KBx-4cUqUqu51{7z$i=4)m9w?Z zrc1Zp#NhO;AJj-c5!u|OD$hiF-JVeCLy5NHp_@%yYTbug@&f9*&L?#xp`(UKS(<-6 zvwXdSI~n;{dQk(21nnt4)mD3L`n*FSHpoSd{t!^zky`it+hY!Kt;tNE|9EGA=hshf zV+SP<5`9_cC6Q{gpb?-&3{Ial^@RM5bezjgikIMJjXydpI=y3Mx4pN`#SuGJ*L20w z0vh4`5RN((WqF4s;8s!;CD}lhr$9xs+^G*Bn%6-SLo-088J!GCQ=&?%Y@sY>{>$=s z9aly;z)g7@Z+l|Qz;i9eX7;Yy%dM()?UZicsX-|W^6*yPcJ0%wV5Z=FW6pZ8WopHS z>tY<=^61Cm4axY=rXF(CgA=2*#*ioSkthpqq>*oS_y@HdV!osR7M=(!BxG{R{@Fu@ z)ovDVKM{cpm$g|G1KUT>^xp;v2+^zqBoKJm%Lxx*MP;SfA%_R zb4paZL^CRV}%_o zcMB=tjJ;5c5N`@XgW0Hmm0H%O3wGZCx=*q5!AdD%=bq$%$R7lWrsyM*u>*)n8Ur>8s%CW<#8r1eymmS@v|<>Q zqB5;_x;oiCTcUOGw|1lX2pBtL)hBf}$P$&0G?gY)DV8*Fr^~s9I`2wEb1O+X^M}^M zBR;3J3NwM}W|Mo^*9hiclbSBT&r8%{S}Z*Xk{+%t{l%NEZVIM`J%YP$bPx&4Ht2at znl`@0IJ}9fFaZuX;d$qqqBm=#AS2PjpPzEpQSz1IXY*%w5t#+zplz}Ur=USTfb&Z_ zYBdOSy1<#ixjSPRiyuc9I$w?vTQE-X3_zzZcNEmB^pPneA_1ZA2w9GZ#9IuAHhc{} z_7aAYlaI!>0VyDP7zjZ?<^f$0Jp2jr_weA|m89Nv&9l|EcUVVye-Tkn#2dGeS!dw+ zNC?RW@aq&KFx2zSt7}?Mk5t?$WxxxGq>xAkc`JS`r3Fs2a`YnVPfFBOiuQ&T43+WU zbDs+R6LHT!)J|)UP$5Ze?~YxE6T{P) zRK99-^*9+y=@H2X54CW#Qcuw}WbT=OJ!V6|8uB;U9&PdMTQKp-| zG6vqVk<^J1OFamdM8J zwp6Z%>eW34h&!(r^_So50_uO|jeE`zYV^2Jbs5g|*LlrxtKAe?In#f)^mn>9V>Iz6 zs>Z%97cRX1jja94WK8<=V5dw+hZ!6*OA?0L2{DVtO4OI-BCPZ82xjx?5LT~bebBcX zMt)Diy4YlKFuwxd4GWkI)triOs=X>36G?sYkNb#W+UZpIz!>U>7_mPVNb`i>9)uiu z=XwGBI2`gZc8^uk>Ur}tObV$CR9~|%**VVa?#mH=MzM=5qn2N^-xZF)N?a=m0&D8R z30{%;^E4DA9TDbxaw}k|=uC1(t`QO(a$0X3~ zC5D!4{;AD|UvUHeAWqT#s;_7WiZ*jwu>(Uo3YV0~6@F&W$YU1rF{w*yY47S5pc?V< zea|sZaVe!fo_H;i`l)anFM+NJvfcmVW^D0*NyHxF)ob<%@QzICRHBC?xF6bvfed*Y z)6Ul%N$({1KrQ+0*FWWJN0drK7ITUU>m2V7JxxJ*tsB7@1zZsFeQCsK5OdZt)_J%z z(~7qa!GRJoCP@)s!{C4i?PBi_S80K76{FZnJ!%CIXoEA+>!`{8*1A-*m5}-9EreQ& zjkOxLTC9t`CZ%UJmk60?!-Lk8ZmF*GQd;{%1bOUO(;dRFDi)8F8?9x~&ku7xf01^8 z$aNQOks1|3S@hg+Cg&bwVQs2Ud$5u-d&`Pz6gm^KkXp&Or{{JI-_cj(Tk+t>+}~aA zRE0S|#eXI^*!aA%VK&BVMlYhs$~zN!1rN+dH&3V?0@uk=_qe{xO)Of>vr)H*r1^l} zYxSoTN1GZ+Yo1lT#ILSTon9i6p*!9^meD;yN-3X;HSA5?y4qf}N+lD_rc9~RV4PQc zrk+fC0ijl$4^OevuS0TVS}k3Qz+o+t`8sEogWys^1H}JGK2bS2O*b9 z_kf%C8%nS3q7G+F9Q9oh6(j&Bs8=Ev&g+Pr<_gwcjUh*@r%UDBN=f{8!{h3!eInem zb843wl~q}PF9e>*Lq3we;vM(~{B53<9UPINm5vscf%LQ$#h$8OD02vhde(aBOTnRF z*UyA-!ZH_S7JsmU`0XK+NHdYx^5m)DiK^h2ZV{(AZiG95`p3jV`$(?+Fgm7UOMiLC ztP&c=nT|x#VtLmj!2oL0fh4iojvn@c0^3NvwESfqmu)S~)CTOEsn!p?Kg%91Eld&x z*hF$r�SIfGs6~`k3R*N0|;84i^GJ$ryW&W}NLamHY2M4-Z<0X))#_N;yxLy^-9O zG#lJdVIpqsZG&|}#PRBZzpl{B31a8O+pf3>%q+FD@h;}SBNKH0xdQ8tZNt3v+zrDb zRMh)P%pmBSn376AAt4?7n<^YOZNLB}a**NFB|L(f%$_(p=dfqvpTwMOO+UzA_dR-Ctce>eLP zx9AGL3m|8EgoY-F2mKLDUXN;b^qEl$1l0*-H+3h)&8hH<(h0(#RK&JAvl7yYU48^t zFpb_%x?iY~$%NN*{x^a38-F;t_J5&RoZ{Y&Q4$FQpkj{;?4ntXNb=YAX$%GEO7Gdo zuQSGlK%wITFXH#l|j#LJeRC>OO(no{)QBNx&faT;THzir@q`*@z z#ribI_+%^4v*n&auCwzv;{>z}deto}n%FulO6wNu^J(wdDU`+ki5rxE;U8%>`icD4 zs?F&#-i{c%2hgZ2?@j! zK`i))f*VGMj=O7sg@ierl$*oY5Z7TG(7s{XknLpgJdjau)i+hY@uFkl1F^T&r%Bo5;DUx)5ak>(~WRFS=Ak9Y66++XJ{(^?Of{2I_gSekhTmDfjd z^L^PJYjypEC}$ws*rOmWX_&a&quh6R^5uXSmuzz@x-V!DZBD3gCnBlXQn+!CN}b=rj1TLmG-~3BLloe8q41MLcWnzCWR1Z9@J-x2a(s7+liO~v` zGR&|-uaD|#ib}q)&j(}(7^Y_MF3@puivKxnZg*efehOivVXHGzCLq{_$NkBv4s4|% zR3DI%=Bp($&1h3ZT)LYO&Iq`}Ok@Gkw%Yo4B7gm5p3V9mqiF2ca?!#^*~1HGD!bAX zP)rfKK1);Pu^n4xtEd5Ba>igz8M;5_NS{mN+v%Hv3y}G3W%W4LgXmq6c(j=8JuJqg zsHCr6Dvly+bKfHAztrp#W@~JxIJGV#7t^d9D)1n2$f%5E7rwboYd;d$S_QlliF2vSMi@1w44`@+?S2Hr#g4nb zITGPMduTcIxpr#q6SE?337J>XdS%el?v;|1n^E63(}EH${=|!jjF?UTX#n8SpzX5y zK7Qz#TbaotY4_2-sq^y?nfd9r3B25N8d&+=ATcwXQu_JK@ND$SH#gBo`GyDL;tP#s zZ~4Td<+p{L{F^jkxa$DXyF z&ykT;X&9)PIFNRz%5@kgIm&k=v5-s=(l+m0bI4Go=E!SJzYyP-&Mj=!JK+CmvXqk4SQSjnZq?>}y((?s;354mnSgI=;;xF?Yev;P zZk>GjfY$H?;Z(|2%x#{oOtY3s_Qt_1_P@BK(>+pnV8>WMCuFjU;`*kX^ukrp-^jX| z9Ii{@bYwBaGWjN|s#^MY#U>ks~KAmy|#6f=wN2;)_l=wdZZE0+@fvd*VYVAgE_u)8+yt+)^xBg)Aq`h zIfKy7gKVf@(E1`rRzUh>&-h&`L!^v@FFZq|^MxncpcP1Ty?L%Fv_*HaRecTDoLBli zO*96L^P8dIU!nN)AR;o)I?7PlHR@4>^S)(`VrWko>EYxj-4w~HT{p@RRS+7{3T3ycguOzfhsDu zpp#;qgK`08KE<2w$|d34l;~R)uzl;SWYUC1{KcMm zL(+=&M6Y7!feicDbOlGfrw|Iji7OqT@{@!b5vncWlV-H7K^D=zb(zDDJ524|ygk*Ea$)PeqzNZO>p37AnHAF6IV7ZWx_Mki` zg!DQs++^ix7iU6WBw~3?G|?J#h?}hSP9eY|E|%Q$E##v^z9 zbsalo_+_v8ZFqoEn zzl`Wlpj)SC9r-;-Xo^hMdC2u)YF-?UfF)9Mw22!cGRFJy#3(}FP ziIjmKKyVg_td@v4qT$__yWw@4a~`M0YRB2ndg|vxoH1$1b5r5PBqE%HYu=T2v@*$xF=N7ejN?mT z7M(EP{Pk-F&X|cD;I&m|L4hvEJ5-ETxI&;Z#+lKUOvkAYHZ7sfB?YxNC8Dcb67-Emnd{A!u-ScXuo9Zo#DxT;JT!_x;SDlVr|5XYZL=*P5}s zo>Ji}v2jMM-7KfQ1gvlj9YA4X8Q+DVlqK7j%o|Ki_dYF8{y($(Ds@R%>h2A1sdcPG ze}{zn1;#TG*_!LB{;1aR>xN9}QuBV$H%P4SfXFJ#UnHU@1a8YRI;{61YnH^=CDi05%E{SQ z_{KoQdQ=Jq7J8$^uW7W}Ts?K+AjA$@o}vR@Gj8U}#3c ziVXYnjA}5B#6bR?%zT(7L#Aw%T)wqVEJ9Uk_7e}W)yGJNBgTpR2-yhCxK@!YIcu%$ zd#Q5{!FCe?RV=Hu$Q3cbI7aO>=Y9o?m4*b{5sxdBXHL`q$_K(c+m`FMgeHcWjmZ+Ig7 zvpF^xGJp7fLCX5R{tDbbz~Yb;!KogeKl+z7`&fA@ZsoZ&P%foXE7X)R|Lrdz-{8@8 z#I??6UHJe2n7SU8#F$#R&ZD%9!(&-Ar!cB02q2?cFSarsRXlmB_WX$68t+m!1w0;L z()^Ri;?HJ$s=X61)v2vk$#DFfs6|)G=GIl@Q@vGVq*b9;c$vZN$gOX+=A4+6N{8~5 zGl?dKP(JE_xj;*iJuTyM*@JBj&3@;`u_<-yshN0jR?-_t^-ca|(~rRJ`9}sg>o*&| zP?gh`tHVLNQ+O5jvn4pOn|bXeoZn6$4$|oetswWpqM*R+-KNj_p@?%x!(PAZ`OkNah(R(Ql&7*fx{1mvSPEWKn-=rKOVY3~alYj{-1Gmd9}q%Vie(^jD0{_&Jco z?BnVHCb$0Xy__>69}26ZQ4qG+p=~LYBP(!&HKSG5kf30|1=0~0;6SSk_ezO;*)|b$ zlZeWgeUf1oEo;NrF_9%^OsbY*#mI6~H(!kz_P99@<$Hv*N+0%@4+f)jWJ=?787u%6 z5<`TsA5EoA{3`&NjY zHk4CvDPC4`Hy*PS5HK%ONj^;K9!)m$gd}eoEB|!CGCFr zlKrVs+w$)!*Cb{FcY8yE{3Pj*O??4*io?!0;rOfUfFnN!k%&(bE>77Y`xWX+b59z? zWQU7mn&tb=kB|cMa3PDO#h;muk!0WzWf5gytz##K6Dd)HF)82P{#?PP62BrkxXxll z?a_W`EB#A@YRKlp+9W^N&HG=hZ-MTd)npPw&iztyvh zihlYW*P6qHr=a`&86lWP}(ua~Pd3k4(bEzdrlGs6DoxD7%JQ-&dy1a%s44$w=tW36 z?sMc%oXR%JwCbKQ4}RzS`}oS!>bl%{{lb`UIXH%MFspUmrZhZ46jq4Jq%E(JBz=g1 zg)@32s9t0j!7k21h&?2cb`GSAR=5{e@pt)ibP%EDX$1Y#0A3jf3yGxWiV0_P`ZB;< z9Q)Wt62Yy1vYJd5G}Ru9eB%l5$X+PNTn<^Xr{q8*J1%(>gHMAsz!dhwxc@rlOl~# z`OD@gX&&iZwhY7q@Id1lMoLx@<*v6Kfwb#!jTIol%1Ai;oO6@Srl3jtS@@{_xS>aa z$z3kb=eY&1s_brafW^CW0J7Bd<*QcA=PsBrnZ4gJuwz@#tNER+xa0vG%%6xJZz`{@S6x}hPp7&mF5~>wX#yWX6}ojAyM*}U ztMe6WHL(NEpUjrU-mmVnuUGK0$5-9$DeJ^lNlR0D?_q9Jd^+lXl?SdyA78%~kS>H7 z8=5qsR4QXMFtUAO;-Qf1H>AK&J;;i%Q%J&~;j@p5h+rS=6UI7SxE7gD_5C)w9+i-! zS@12}P;HOX_0vQr?nM)KI|LcetS(q032sf+R&@)h4BL7GKj;_daGAq4omv?_z0$Xo zQ55>x$Rl+5UqddhLn!256_ieg+8zE>cfT*{?uPS5n0jvDaRVA0^bBoRVA(WZY zugPn?_YM?cz)LwsuOHYXJK$?ISUd0RM8Ay64wJA>3Gy>ogh%^^{oKQK%7t@SZ7Bu% zh5<_Lq#Dc%T6h+#j1T=jho<%=D2tGj`aRXw4=7h(JdgeJB~5B`Z<=b|{TxdZlYt#s zS~en$C7f{T(5-t3hc~bTi4&qmyjh*nRfE5W38B=gHY9c;oiA6{p3955^<=8*=u?sAt`QOGZ@qi?cy8uBh$bDJ4WLM7hXrWUJc?uXuE zc3>#FI@b(|#@{f5w=yjC9m#^a!hEL^rWTv3lt9+!6VecDB1$obq5xkE|PhyI^qFyjo3%$FfN#)ik@{{w^XKmXpmHiBr@VAa_IydDs^^y8* zk%|`_`a2yp0^>(1eW}{8o6buUCo$Hu<*s4VI5&Y}7;mY)5j?y!Ljgr0lXzJNdU-+@ z*Z0}EIrr=&eKoU0pr)Q4F)=+p3foZYPi_#M^0v+uxaJefBfoIx%PVH(a(T|8Ub*)0;$j4g$}MlHGVD|x>9^<)vX8XtUtPWY81uaNKI*Y={^?r4dFyJ1mU*vOP=a`CzNe+o?O=^y4P}L z{l}w%dcno?w-X1&3`?g=^FQ6KaI+bknH1c(Y*QL$E1mJ}EE;>@$RB<&r;gNX_V)w% zqDLNC{&m;fhE_=6TMON~C3z&~RdBfZX$P47aQT_~bt`bf5$kVRgc{k3S*%<-k;gB$ z=UvpmGuKZwom)yi5?TQ-phpd|Pip-?a(iAmTJQ|X0={g02dRxI(qpWRa?_Z)A0f0)r1Wu69Sa9}oD$R_E@Xr_!t@m1C# z(GN^Dq;W*+F!-boPiz%Jgb7J+cC1OqL#w-{hZ2Xvh8`dCW0MW_nizi2*#dx-IxJgn zH=Pq{_Zdt&4Jyo>-d`Y`C^G#BDD#zz-qlKVj!PGH##5W*b4Xr0e#vrSX1g6l=A6JIIr4}pbOkmyff?`ZOmCN4zj!exOMO$SR*{t6-G5iG zA5I(*OI=OpwBKCJs3TD#po;-&uxYQ1;^rlvg)>Jh`?F)JF_Q=7=-t+Rq8aV z1!TQNA#}dAZnNrI@k@-wOr|A#2i>j9k7vH(@Nj}$)1RLOA3a9{+YGmP+b=rf^w#lp zA7U_AMKW9{fC#3MD(0y#FrD{*zN7NW+-ql!fYViIX%t^|ttQh7m%LXZEfbTIE0@yD z9X2b{8a65`^?dIxSL7>+a`_WZXbNQN*?u0lJ(^xqxfe6*cNvVgI#U`%`pTfGSGOv) zU$*H-z{1Y-e0whCC>m!{k9}t`%(F#aCg}zTKue1kdL;gL0;2p)mo^Qy)|I;~^|bs6W?EZwq2Ce_jYRDGHgUirkmp3)^&#v$Vx(xBOggP%X-6>J^Y{ zQ9NCoU743X>AMVk%RSZ!0CDKQ>C1IO_z%O929-OjVJKPgSW+jL6kZWi z_t~FMpE;M3HEz?f<8nC1Gbjp3Ly+;~d#v-G%bvZf;3(WX{r=j`_*-NuKnl0(s?FB* zeja)c;MPq4=r1uP&#%T59$r#X73%sklXlS&f$%IMo*IXL)}1Vksx_-34lf*t(Ddmw zRT$GL`ral~iL95x;y2q?h=x))a1q$@(RaqLhVuQH`6N&jFa9sn{p#_0FG=<(P3dx( zG7ebI+G&tQ@|vmea2@qw{0quC7Mc))b*5jLFdu}N<&X9zK4W*P)TBQJ^2arY`I>}G zpWXL;3Q9Q}19A5{!Cd7me#sDRCgB+p}9->c*MQTg23 z_dZsg7GxKXM41~eqT0x}iEMKawTwTJcmWnE`4hL53fhq9$1oG*McEx(2P8`%MH9Ri z$dd#GXoM}%-xq#Xv>UiqVVQka#X${KE{^rJ6J~c2#6Uc7=?68Bl^kY!2#1B+`@XFl zru#TmFMh2uFE~cCnt0Qk{OXhNnLDLkS?GIJ+rG==+4vN=@_uXK<)@M9Uf*{Fu+#Z8 z<88nvODKP5ps&T6pV?q|IQRv3blx3|0HhH(e`GK+3#BO?hiBD9p&uIkDEE){W2| z)X=sIm=15r5Ti{PPD4%v_x}GMDyT4wN5I zloGrRRd@p*nn_|i<yEI+ukQQ87=F;Z@^?55pZ4)%b#pHrhFQ*UJXy$*N!!Pa)UlhnRTM!hG5 zlai88b-v`avf^U<5n=|JF~3X@XaFm*b>tHBx14J9Vi7>WtSPX*Vg@7PthuH4-1&g_kn?cZ#CrKT~G?J!yFc9IBa7 z)fu|&9|D;g4S?aZ$n@^VgLnI7nCtu9>8Lx6&0Eu`!uSS?zUQ^Y!kRNV(LQekf7Q;@ zhrJ_*m@Z;}_ff^U_normu?5`cU)c?zIcU_V2wv}-uiE&S<1IVP(I~pNcRXY)6Y&=$ zmuvCcYx>tTsW$aX3l%{vB`LuXV5B8Nz@3w8ABd(4LUzPa1-QvM4 zL2b<5{C5^+>|WhYg>WgGdPl^^60Qe)&Nf+p{JG8cy0C@(x{}w~QBze&yQ|NF1O5oU zYQ0E;)0#86r@ZHLa2caY1t08~45-K1ue7;;t})<9t(XYsxi3w0308gm9y9p ziE2=acO4UG334so*h}5my8|Mk#U~BcB*Y!vM$?4RqvmbbIXlym#q!dWMbo3!X*&Y_ z24v(Oww`wViV$PxGG$;=ixO;&*FV^)VNv0$nmMq%5`D>wekd?N!9+t7yKHD9|45KJ zc$H{2MrSCbbu}7cDt!n}Ggta{YWE?4Zo(xLH@HGPM}+O&)qni^N@!9y>zZwk8Rp5S z1gUJ^;v&U#n~=#ryA@jpyjw0YyA70N<=2eL-DHj(50MC+Mt+-NvwfTmM*mq6zrQr| z=9umupATo#mX?pNecIvf6>^(mZa(!e4mj~{eLh$#SlcvgvhNs|Q>4^GJ$8-Hbh@ea zK{XQl;O5aH=zK`L@Q!4=P*(_1JbsuH4BK1HHFY;Nc9YRI5Y!k));{}#7N_C)fj9{n z#H#h7s7<%C$i^<&XB6HmIkze=iDUY|zRF@&|AJ|v%P_*aQ|0dlne&ZaT@O6h}{OoW}P;Z5RBoB{$ z__{tq+sNb$BJr6f54%tqi}IqaPy>9#Yt;Fhg)<%DB7$-zBKmvlHK*h0CQ2`sRro4; z-D{yS{a^rRM)V7&E#JERudfMhErC;;gvf_sEmSA$Tk6^E9-G$fPX=1((n%-|%g`M| zPb$twYe@J$aou;Ec(z_Y?%eF`&3rZaGE+y<0 zRi13!*L>jNJ70r`_*S^}qA^-ZGOJ33YRE;xL5rlTcsQz-Us1I36)9sJ8AiTr2@l{5 zlorU|4Ed*x7eW2vhRO>g%q=zas(x^E`W>NWbO{>QQ8g->Tg%#T!uiOEWQLS-{Gcs$ zCRS{WS7D_7p-a6`PVa5C;S{o=kEJk4>&>nM6*=Bv}-2J?PBsG=2v$L%OTo0y9<9q)$yGOOfiNDgcvgU6& z#|4+{dGlY*#m!&B=<`{--}xy{;cF?~UJ816q5W7l=$f;tug*vO^2O%pxL^@KsCkew zuXR!f8xwkk7wGf2)LIOpUNUx$0gBv)UgNuLn#9mvd7Qlj9M@SUJIK;8XF?FO1F*LP zq~+k97ynd@5 zIIv6)|I+E=sl&V%p#+9|z1S;45mCy?(Kk^Gb-dOsLWDsPmrTzszxIHs8|6cgw^FGT zHZJ0A{Bt_K*zOiOHK2?`Sy*wLhWDEQBjX&BT%bu8@_jFK|0QSCu66-FkkX*5#%YV)__p<=G+kdaKxvy8~7IfY(7P!>$Q z5{~IOp8PbDDx#88wDogwv^FKPV3ICg60l&U%vZauzk3xkAsz?D_y{U2sS_zqu!)H^ zVNCXBuxBH?T*owwq_p|p{bOZ33=uEbMqPmXrC+JXh*U!^^>VG9XV4U*)zR#^bshbP zH>z1wD|kNe2i)g6^=GZBT~>M>rXPX`3IIlbLm`dfj=}1kj$Zr}z(8%%Ae-M){DF{$ ziL447F%@lMP8E+A8}d7HhR?`m+O{TE=(;4ncS#e2KdE#s@RJvzVPbMkzb5#G zkG?iVSv3#>NiG|6 zNGQC})%ML?aveC*nTIjde$eVXP(*e;95A};dxtF34}Vuc4CvJj*{|zODG9S0cttDy z2sqVBo0m~c{*Ed*g7n(~rtfz*MzD@+r&Azhrk9h8ZlhH)lR*dFws1*?mh?xV@1(!rNu_ zIzhRWi222D^A5*#P+a!CxL4&kOzet+vknebNbszLa#tnxt5>uhtHClZ_(ZW89o67x zuGHs*VnpXCm1}Wfvg(D8Ie_z2U?i;;Y~IAP&c>AI^yl?3vk0JoKd{9rW!fMx^=P(~|@) zK{WG&Kci~Ie_fGo*+Q9kt^aKf4(0q@!e=Z|6dsbiP z0zeq?J0i8nlj9NhTn5>pEGnEXjqOyOy`r%Vj)UyHR>aNW8SvlyVMv{5yM=6JoJegp*i%AhOctQR@A!?b7r! z(F5;G;sy!-U)_2nZHeR7uR(H5TJ6J=33u9tblzzMK16SOdNle=tjWJO+GX!$@8!hx zEbh&U(W(**V+Ew@nk$lTxT5~DrJYykQt;_>%bcn!#S0A^R_%V<0xUh5*acH%5fw_TGuX6YT238ccWN%CLt&)#fS{~?J;+t zn0P4I+aAx7+fw|GCdSEE<3ins3LRgW%)e=P6%Ihelf#W&75}N}D8VHtwu>q#ZoJVd z<{Yb}ZbJ&?E6_&|{Vsf2a)fR5q&3AjUVx-So2Bzku63tM->tyN9hnL3cdWzf6ABD2 zmJ8KRtm(2~Fxvx{_sNu$C@#umPG-$OFwU8PTEy;FlhSsrp8@?Pl`bZ!i74-zu~v=`cxKYq8CLt7H=u zx=S8sA<6@8Y`YY{se1@U$N$6X9SUs=nRsLA*v0hc{nWImg}`6`tCjZq!q>^#0`3qU zzQk~!kDuaecr=9EW!c5CcSvphW4O}}R+)@z1-DzA!9LPDzk_|FgSdNv{Od0k!>l-u zT}S$Pav8fhnQW>?HGTUP<0?Ev1bu00MT3&z<0MmtOvKX06zvB74f7}Ka*_y(g#7-E z5|U)RghY#y9_{y>D|=~Cg}3g*a(egwE(CGnvwQ~c(lTOo75@{Gcyzu7)*Z@!kE!8T zP6v6%*J~Y@E?FyYXoVLej5~N{GoZvBmBC_=>(cEre0*w!g*d?>}lA%_n_qfsE}w7YB)3!1L5_t`BleULAa*5UyfpG=1D&miHM` zVvlilqW#WCUer-bTO0i92kCE4pTEAc&Q7ZCdX@^QEpG?Y6M0MBNbRkoPi^|bXmef# zXdAn&avo3|K=&TZHYm7+%!ehor&ZxeUDH6vgH(wQD5`%7rHWa-Jkf{^ws0`66Fum==9^FV5MZvskG?% zJ@_wobMnbaF=Y=9-fyKv4ZajPFzH3x@n5CxamoO7%67M9vG5F?dTxQy%xi<|UgQXH z&AOW5kaG2m$6?rVlUNLTYWG#QtX{0^*C!5j}dwn1ijK7YI!f0xOQoHy2Rf2-7!S%YN_?>3C($YY2*=f+%|FiXf3pE9DN zQ@0o7w|8zo_4#1FvT)I;T?i?68P>e^$}EU~yGkhru5s~rm35OKj|e{aPNz#N z9M_4SEB4OCgPb68MM54H|BkPQ@yF}=Vy2NHiA<^o4;$EsGe}I~!I^^ZL`?$%Vb^y~ zF?`CCDLvN`fhOJ!2FBD8CF0B79MJbu3+7TB`g)1o$LQI|Bt$4`fMF~brIQw$iF~Xb zGgCSs09(NFV6hADZ|0F(Po4wZUJk$PGA@2L;f3Jf8RjW(5ANWpp)_G3w4gdShQcjH zc(w#+esqp}4lY(_MJiAs*fcCT(s>JpNI`o~>L&}8+Id{%9p7xW^FL)f@aRRf;u2g3 zY*l)WAu9`3dRNOJPv7T+z)3Y~O4F^i*#?1VD8g0U3WmwH)TyeiN&LlqJhqlr#4*-s z_w&tc1bg>c1>d5Tc3hxm+EX~hpwnKyt`fu+)K6YjlR(i)s1s0)ie-Y12*0l?%u~4NKGOk9x+f9lcBNN6QowmOZR=~+UeUNF%}u6S|ko;6@dNx z>qSRoorJgPJ5hKWRz6)GK0V&9cd3i040)cMx2PqSnwwV3eOU^auP9yzn$#R1a{i#g z@VnVGEpqLYhn!B6|I^3<;-;<$Vsomrr@Gt2bJcKWDVz|N1z8dh3--3n1n+Izvhne? z0Z9y!;K$jIYw<8m)POtuNM>?YQyJ}scOC~O$dfBr$N=zz`$fgCWv&Vp9%I)KLGB+d zU-CA0lkyzC|ISb> z6yJ)I#5qKP+2t_ge(WEul$~@G7+0Reb))lceLTI}CJCcx--L2#D9j!2^;x+m?y2^G<1u)RTxCRcAw?z zwywm<2H#O&i}8TMF^y;~YFz75sANFIz#UpVe@Xj_AHQ7P5&t757F!Hmk+<7$t>M#3 zkl)q0wyCT5fBhs5a*mhPFx1U0?_uF;)LrtHGg&)8JaRyu)rbh2%Rr!pwZV9q=RCvV z(7Y_txQR?kPlt;_pTa*ONDW)eXHZ2(yz9kfcd3^d)#+=Ta6!J_aM$hcNDD_=c~~WQ z{82U8D2yMul1{!$%8H5`EMbzG;TS$itl%?oZht9}$qW&Zp1!>{uC2%I*Iz3i#*m*; zll#btbf#MlgEtnb3@v|$tH8$D-$HI|ujl4RVYVPWVtS#w!Kdb8Swj-ZnNSIige1qH zn5hCbVID7r<#aUI3gkqOIwo#n(pvuw`=DJM1IKDLr7j0pqPX_5m`w*#ck5}?esX21 zkDvW=NbUdx;ShQ`!YO?h`@pWeF~E71Hi_s7|7S<(;RMFeQFA{n^d}uUisR3yZcd{sVKV-hN z?~63PsCWSIqy)_x$q-Z4$SHO1&uXN@s42jD@Q9E($cgC(ZDpux79Ss9-NKLTHn}Qj`PHNR zbF`Y!6ClVg%_Wy4fj_JPZGVv>jlLSs$X~Xq1HJQR4@bj2f9vSQS7&stA>{;Y(r3FYS1CS2SRgmz^SWfO8 z8L`{0+$>Bb!UV;3p6=TeHgW$*arX2J&zGeD@NI26f@uSW_~!m^mQ@1pby2~Xdx4hf zhMWD>TsYAjA$Ge0!JFm$Rn^Y7i;U}@ksbc1E>Gcl8__+Wjf9yKWw_){vUBw)RA^TE z7~nLTjLXCuPCTm|64e>Qro8Hg?wvqc#H5_u_6Hgcvz0f;dYmqV8$v;zJtIl^_1NH4@F+enM z^%VBX?%=%B>{MzwP93VEfFybb}^UIs~CnYBbk1YQ?i=QwaAL)~OZldO$=jGBin zHd$l6WnHWNZ(n&}&^{4ixewca+Bc1??12QeXq7tzag4-74f&ZZd0a5 z8^Asc*tEnuMiv|EC)m2$MpwKF;{Duoz;?c1@7P^xc|YK)Y}>_&x86L_d`vu}uwljh zTY#iRdyV!2k>ttiL*DtGK=lU;_5+SLPdicE7o8A*nc6x)jR>EObO9TY@1R$kS(U1Z zrR!N!K1r8$78@}E-6n#e=pwax9s2|Sk`eqxw@l7Dgxe8%OXp|2bHCju*$R%iRaxSx zmC=CEa97IrveJJpXGGwS!)qn2k+$g9a~b=eqO6j!y?#9SVPM6=8BN7&A%}^_5ASkr zAb8aGi)n^aqC(*0v6xGGfqKaxml)rHim(~(UVzMG4J>BpbWqx@75|A8kBE3P-kv38 zILlw|08!Br>ysO-4xdBIH6h4&>ih#ea+cMxOm!RBhq*oDqRWt(Xmd0_qS>@93o^;8 z?<%heui)zJwG{xJBlD(V1sH{p-O6&E?3$+S zJ9A%$-;g*OzdQ%cF+!)}ERS7!7t!ihxG~(IA+GC4diX7S$GACPum|O&Y@%GWwq~Ap zy_E|%K6g#AHN?5u`&TAf?#IPwO%qnT0fuB08K|TjEOJjIi}SO@Q-3 z&`h$F9f*a6BZ)D1Bxn{a`aES8op*sGtjJ0D6RH-x;OnMlU@5y0bTL+aWB5F-BW)gO ztB+r)jXR@vuo@*>SHW`@i=?b!4RUEiNWw$39OD0uh&EHqhPRGyaWDA=de$@L#G>pA zmS*ubZ3k4=EAc62zvOel{ziy>Kr!p=>#aF-Q=U+>%@GqSnCf-;JIz=qo#Ftayg%5h z{%@riM4w{pckBW28N?jBg{S<0dej+MJ3Mw}A}e@V;Tlte0-eRhVNhd}3n_i-S$CKq z9fhAw7^+vHof3T1An;oLb$&v%^p64(-;*RVxNvTQZbVD$+hJfb#;qgDL6wb2D3u%* ztVD7({Hm3_Eo|;I)_5SC`$IwGrQ$ctoZAn8T7xYTPvj8+_-_5 zH~rl&T2iInqv=xLP8Ix5Tmt7-1q-j>W%_YY2@$32Ow%a3k#&6_6Y8wWCVtZW z%RVWKMRUA*7adpJAji)<2EQ@LP)>Q~U$?n)*-o20@vO@Oj)-JZelix^|a zkc{77F6dvhyjjiJf~soO2a*aFl;+McYc{RBU*QgV@+eJ{cGBJL3j#Kt!VXt`x3a=G zI@+hb=44K;(*4!BUq~<{YUphVYcesU;nG%g)8*~^hhoCk22uLqCy52zrnkIhxeupn z{ShgZ?*0J(>e(nDA4-`69gTFz(J7M9`U{?+3W3Q6k%vr(W<{u72L$5Hki~k+T``wn zaN&Hz_MDTCGWdk@)#BjHI7X&Kw9!m0UJmy&^q>xe?oL1cUPp9$5X(4mKif_9^f#Y!-pD=VV1?tb5s z{bCifLH6@6*X3&K>hlY@W)wb~Znz2mS=ICm_Z7T*Hgyf>i4;S(2!oSM0XTV?_idao zS|Zm>;rOJF18iv2I)h8;ZM|G%G%HZK%H%qAHf^BST(E z88rgX{1ZFKyReW~R3dHT-noZIu(d6l2qmcV z_&Cp_Hy>q@_~8CJ50rZRCHN~Fo`WAe91M_U)hYcqJLWwGF$~3TdT{`452uoOh-e<% z<*Y5~PX*gdBpIuM>^1Z{5@j!>d=)Y?CvwFn?>dABiqw(IkcZ=y&hS7?9oLFzFMWGL zM;(w4~Jdr8{<+YBp-`w718GL_v`b&oG#D{*6fNqDj#!22$MU;% z)B=pQDwTHw-E-mi%Myrh9Z7F!Ox94gJh_NcaP53ii~n3UJ!)|kCcIqQ;1b%fw`Ia$ zliK&#Oa02@z(l;-wA8|21keK78`U+I!<~Er3EvLb%MM76l;}F1AU6fkp;~L2Zt3(G zw4kB8{t#Hdf_o;bgd3Lfw-6DO`8GupsGjZQz58}Ya_<@^M1Pp=T?Hw zt|`uHIb7Zp)c6-aC`4Kot5PlF_-`ukeXN47{?>nxA#j4mp?~A=>zd`IzXtYfmKrV? zW+p+W+A880X&w_NLSSk>EVTtP@%DF+MVMy9PcwD*gQp(f_)sBT&3dLUTRUQ7Xqu-Q zlLRx(*y&_>9)E8bM4Qn-B5E>#turoEMSn37BWfXdLkDo*TbQ4j!bbK?35_ zm=y!s#SL1wKX+V}*@L02DEI;Q^N~>3^GCRGN@CFAAFBG82Rg;)ybD1IY=Sy@fUApS zQfzEg6Cje4W(Iutuq+aOF-Dc|>+R7tu#G373(sqmbFhJEbI zV+{+5Td#-M5j=I}B^u!OMU_e%}F$Cf`Hgcvxn0Cl46$?p;v21J#u z5SPqW$QKMtCTWSmK1|I-!0hX@z)u|KnaGGT1bWoIKJpaGo-PCJZ<`t)0>f(M{u@C) z_Sw53wIrUXp|V_%2xkdHd->bV(5=h2`wiMye*kbZqJCY z3@0(Q_u%S9TSgJ_@mxlxx^JlBb}MBFnXH|GKIE;0nqdu4BEzKm7)xEou1=)UBoH@a z3$E$$QvI(4Z^w}5bcKnn3_D*(64)As{Wk6QGV{MlUGf^!I&?H}Qxo$k0=gVxV4ZRq zAH8-Dy+=$vE!Xu2nhLS7-i5&+YKE~O@C4|P6oe-h2{#PVexygm|L=(aD?4uw$ZAVt zSZ=qW+r+l8FoN;a2y_#>smH6OnEzW2rK0E>)NPh#P!{7|8dJw%@d_FRU1%UaF*Ha> z&VW+7nf0&#qhzCjQFZM9z+idta}C)F^}FyymvYkG|2GxB)G1n&p%%;2JkJa~$ykNP z8GZmw`wbefF7 zo<`Wk{^Wx&1aq&z60;}aE*H4*7(hY^w0eE<@)`toRz+c=O_JsSYA>ahm`rdnIdn0GY;$b0iA6 zjlL^Pd-X8%tTbdERO#v4r|R}J+4t?}?XNQp&vo%{ z073yT*+N3rGb%+Mwzf?2AD6gQIK`lx6^s1*ak>6^f;o$Mh_XO-Q?dE8(0*i8xE!>i zl}Wu;Zws;Veh#_c4<8lbmxq;AlugcmBBiG#4a|cMnYg-4eBVEwnHs-N9tTlVpqwnE za47Q`q8>G7*HdxFdwbv5SKA#};uSTuoFtAHo^C$#%F7M2aukn`SI5n?4(IjGz#)w3 z7BQ85{Sp+o+gc1e*5pyS>cbL#!#AR<)@j<+vS6<@zP@!R8}4y#+05O}zlALpP^e5B zz>=--qTm8?0TsTT`pni2%g4-vC{aWm9IE{8=g_X71RQaTBtEd`8UMU!Hq>u+2A&nP z0O{NhC08&lu@z(&3;TLppr3RllCk$JJc@yI-`&)S_Mar@Q7s}|ayu|jmy$wVK3^Om zmEI_HF3-sQ(_;S;sm zn+*8&*;^q)`XFk}I9=Dnk;fE6qUjwdt|Ly;b}GyVRTY4!Ysobp6UdyPvXuRie#HL* z^Pnu8$mq}^wfIx#l8K&l=}V{DcwSqt8<{mK3ARcT(!S||!K46$$XfQ6nw3@g;FZnT z$REO0Dcx)w=qj_5os~cyC&VH5pYAe}AqA=r77y$fYUAi>?S2}v|4#~QS4E$);kluB z+3fJ_1`SClSsqk-%7{QQ<^~)bvgcDUtOkK6*S!&NFm_(J{WS4)lp;fp+(Kc3z|=Vx z9X9%TWsmFL3BR})SY|Tyu_sv>v_XBbQ5+cz-6`KrRCGj?-zA|C0d(Fue2}CdYAqeG zOi@iO5nA(8yE~h%ov$v6`6{4atMT#8;9rWugG*k8Zo3qzm0WPCYC5MK6EurizG%A! zS<((RT-M4s$IC~;F{vg%&nx^BhIsB6g+MxMQ@UjlAf0O_e3LXsjbFH&Gde1zc4TK) zAsK_d>Y!1kU9|;}#h7MX3;Uy~FZoFFhjST1S9x%sMdcD|r`k|VK;wM%3BHX3eV&wE z*~{evmgPi!K3vu5N^w^1mad1=3!7?k>Mq%gV??A@ICMK>M7_I&2kuQWE#B&U{#@tc z{%<+sa%LC zHbf^Hx@L_027C;B&AkNh0rBw4aZqFbkf}@kA5T{q7RRz}g9HywNN^1ff#43o2X}WT zxDFcJ-QAtw?iw6|2Y0uigY!D)ocsRqH8a)K)wOH3thM$a13=Pf8h(6;#CO7$0~;(O zj0C>YPrI>HyzMXlwpLpx)uJ7TiB+gFnpC?$#5-t+mn>49s?sk=DaDU6rnQqmc% zqxl6v6m~3X2FL_xqPXWAW%X6?FR#P*EQSAzMX8=?}WZdsi}an#_)8jiq*^? z*95@m68RrbtNg~ZF*@Bi-@{IzDSaAF(DRr)$N&egZYyg*$KMV$bn<3?*1tFv$9+zXr_JC09M(TuAy z8XRcU*^%k&!%Sa_N|#ftYMmBYp7~}=Vrj?I*%x9}A6oAt8;8*Bx!aQyMqa0 zzOEbXyk5ClXrKRcU@e(|2Mw>K#XC18Bcot&l&jdx*c>TwxJ!D@z1F)At^>jq&EEk@ z_P-U7=XPFETF$RE6hFnu6)y8;kPH#F<07M_!o)D41`JcwRaY#4Sz+&&Hb|LOi2xm| z@T@(v7t`c(8T@Yi3WAmH{pe>)VOk%nMhvACSUNq4zW`+v-X{d`Ug176MyBymB{ zO?2sGT2r9!5{GVQ&%-9!mETiOAU=E#pmoG0ma74+B<1E!H>-Osntc5a#mhpQY-gICyR%eB;^c)xwucAEe%Z3Z8~mc_^P8vZFe>^>)Q z(d%yXz*l%U8B(y^MVO171}jl<%2oI>I$clV`L6XP2z2Flj=xw}wtwEEygvz_bS55H zHJoi(d%R`6u`0P1xc)3Di{#lFr7UqFFUa4pDHlPF{mt$o2YKgZ`5ciZqBHwC-v-}) z2Y|n#fnP3)rGvYY>-y1z_teNyv)G)U!fiVh?G;zR_Ro+nTRT89Ee3#}mHcV9X@SUo zyuZeJaJ45fpX9mv2L3bg=%f{r0MJo#q=8ZW+f9$|qT{I!!9jpP@-M!~yS9xA$K7B4 z4@X{jVT0LTibMol+ULY10;>!QEBciNOdr3TIp5!1$-x{YA`4eNpPlZEfPzL|qxhIl zZb7|BrDVzy0IU^}ZPY9l5cmy>s}KU}Zl(6Xi#IM~qs7F)YQu&v8vneH^~163GUb-w zhj3Y#IXWSi^)DoK(FFQPm$p?G4!H$U#@2y78tmh6Ts+<^2c2o@?Z#xVA^lk>>j&;V zxvmrv2&#|#Cr`E-b&R)w@(WORYklz`jzq)XnQ*~SAkT8w&R*+7B;;mo?E7=^ZV!?T4!?=U(74v0bjA$TpXnKW#fsd)nZND%@rIqVNu#>296vVJ` z7S1^z;h~rFDf7Y<2~I@FM!HXNW%A`_0}|3WOA%CZyd*vk`jvLblW z^Tir?wToTju{h;n++r)%1C(bdK|f>Z`yjke-=$O8RRr45ivw@MQ&6+sgB_fCw79J* z@E`&YG3^%<=}=uT+%t<;du1WL?{=MqE{wwqLr)FC+bQ77EI^XM0Nr#peN?UaVG+_JqYI0dFPa|yzw$p`UiEHow^x_RH?3av7M?LE z9W5D?V-EUU8hpp0nU^mXcs&BOnBBp!7~NJu0liDQ&%7l0x)t|3n0#IxJ6p1e6CmRa zAv-BIxyDZ(ducwVL=q0yYb*I)@Esf-5cFSrBuVS;uR1^Il12i%9lAnTUdhRNZ*)8p z)6a1gBJ;VpnUua7K9P-Rdm6SV&ISwv9ceq*&jAM*;PdZSc`M)# zcH7k<3l9L-&fF`X-($1&uUyE(Gom;#ozCsrAVS@=E1vq>mZ}i{tF>f=-$G_t`4CDQ z@3YZSIK4bHR2x1suZj_OJx1*hc6vd6?*43aY?Pws;|8v4bVk`;l7Be}=int0rZZ>8 zVF&9_24COZ0VjXjylN53JDAaMttRX{BS6&!WG|R*6!j(TZ`i~Ih|Ab_)%7?>D{2Tk zS-!Q?-^=?<&`?{dRb0t$;c`CR@Y%%=?qZ>(4v9g&x!*foKu(8_QHT5V2LP2elq8|& z&*T((8@&J&LGJLjc9{f?&*q*eD(xLXrpbhC9X|^?*Ejg-3dVjgWs( zyd9m{0J7?mtf{DNceUAx;u5QhO&~y?V;3v((EDSjK>{k1mf10**y>}R_r}^Ipg;jr zNrle&0+{7ti6pK_H?ug3@)XJw9Atr=oE`lFfqI_HWn;==*_mTF7oIThLJfc>cmiM= z3;jCe!6xr#_r-4`&9psWW}Y|D(H=0z`1^ssTXvW5mGA6yn?V3zyiuF=0JRDFagRPTVih(4!ouBEGvlBnn$w&q>YH7#$2B z^DJoTm{`Mrd~dR5Ll|C-b5XvbIt#rn+eun`LsR@7D^iOtwSsYeB=H4kCiD z|04l4%)<@S2|{npqLXPeO2>Pw36G6U7^j^$+(wTa3`-{j-S?@*o>|u&y#?HglL*k2 z!5qJdT}MyAjU3vAl*D&xC-;G2VaouKuroUDbLRWbME6$gNrci5Rz4@RXFjptN%3ms4buY9+&m z$~zIq=X?rj(pF)lIrmrDk06%0 z29h}k)?@iFJZTBQENtXFqy6Hj#k6h()m9 zAhF|~H?@r6rUgEN01Tn*OA^zT--eEHc@x_+l> zn=HH>Uz_=Y(Ajbf|aR_FRHGUjE&t|q2+&qFt$&FZBqGOI1{)ApMy!+*{q5n=i_GSSxP%ye0)caC>CesWqp2pwFcYfnm)ghAsD11#`g zAJQcB`j<#bgejK_+EE1x^uD)Y*NlI5tAsaUw>l$8L?-Nl#+Og!s;M*U?)gz}zV z3@HwJvppBrqWiHWS-S-6ljUp;>1+4xtuWjY#JBsJ61rQASl zYmhzU2jy)7%>dLed%k5TkP{AHEUNQ)sir|S1l7Qxj0ie4iG3!*DGa+tLqPYF?#Mg< z&If0SMh^Wn)CQwR)Ii@slfZi9qB8Dm;jjCkmG{T3zw|xut;9INMllENAZ~WNV|l_h zWeUBZd*$6hOEl(*~K z?r5FCCC9oPxrOcuG1elp+#=YSW*1(7@IPrNcfHE3Tx>h6nIvS%YQFg5)vmWV4R@$echw!Lh9y?M+?2tY4ZLZ= z+m&-ChP#lmzNKtDMHIHBUiRm=hC%o|t}|5=!nSA6s*F(O$q$4co!4JQ#;6W-=SbB}^gr87yMBWGJ3L-uTW~Q}2*dR@_ z?x4VRq|c3_0-q6f2S!sfqQcIkHCQ##p&bQgVz;>G`R?Mcq?8!8a*IT0=56KioPykw zcH+0*VCZE88N62VOrGc&mE?8H0;M?YqIKbXk@5(gJo?cLWL0D7Qm3c;f;6t|r(X*0 z_^N*Lt`>76_t+oDB!iBBy*2O^Boh_1-}spye-lQ8XSr4WJ-t++FNixh8JV!>XC!_@ z4cZ{wRR@o?C^dQ_!p{u?4Qm$*a<NG0cKS7m;wI>1W@yzyaZM?pN$U0F=Xom!N@kQZaI9+{Pa>`bno1@IJ!St!Ha=68QXb57A3gSOIPSj6^nS;mp#}IxkbDWqJy#Sj)}Xrn*~?i z!bMw?-qUUTwpcM=?j-ky3u@@=1TnoFek}7&J2G1(QuzC=5i9!$${qR<6lP(1K#A*$ zC((Go7?y}RaIGGt0I}I}NFIC?J81HUYu|$Z%h6kB3eFM%f{=&V59;nI0V(hpOyRH3 zYTGNlnRF1!-qcy@HIHQ2jvhx;%`*N@ZK?rZ1)71Mu%w3CAZ*`Ng<6qV^SaLiLWrdnHj1wK8pnGg!I+GTfPD#xxsJpc@R`5kiAaIlgNsU2la@eymm?;O8H>SH; zHgT|Fvwc*@JUyC4?H=8()GWkpV8g92dK~(BeVzRksecV&-JK9x90;SJwUkKVF^HZ~ zoKuUDU{KSL3TxDHPQZ)t(yOg{cC4_ScxTFe8%wu^HFj9X$m#ry7lZwQF@bVsv)P8P z5(H1)+$%#W!o|`%sl^D(h2?|8i}0ptX&Pq~sqRD^W(l;_R3fKqrv@I#94(Sb&loNX zCuSAwAW1EPv>z4BDfnW1Q`l}FnoPpzF&8;wR+JaP7i+qd-)JCencbJac~5s) ztmjP>W0&|yJk_R!nEy<#g8YsisJ&Je;!5@+p4*m564*4jINrfO z!k(NZZy1M6mpNcye`i~cJ}j1H5VY~={dhawrSpMu=v?5@A#@E7a+$BE<6{jBLyYg(9G4SxWl z6AJ2;Ar>Q7g*8=2>r;MHrqaHcCqZei`@}34=}gE*jU`_aV8{_ZW`2-L@tA_b7JA1L zYR$!@_}xx-)}!%txF7NID81@!gek*2iZcoY%4Lc*Ko?yx6BQ1$z${LHv*EtiKD;{c z=8c1N3xtv^6+Qvb!w_40wDbHv7wJTu=2Rar<(*X4=K_%H`aE}m(%B@YXc!DtbsH>@ z)KPHSKfNAi5BD$X=S3eRvv<30D#4w)Z;O)%kwY+lO2^y$PREGt6rl@({Y47rO4Re` z*_+Qp5=eBbS_LBK9samDA7Oy3kutS&Yp&h@@d5+TUA&6NODiKM&IaU43g~3XqC;o8 zSPCT_i6SMt+)b*KRBaIyAv3_+^GhF$@c7aHn;uYBay=}X0XHYHHimyNN_q*F7QX@t zfkzi;#5XC6FUL@+%NQSO`nc7n!5G@G zypW((!iw}=Ui`bq0JgKQm%#NM`DGABSpr@DTJ>4%^_LXSs{l=H0&RPeoRV$n=$*F< zM0plFGx&m#n}`n{0>h3jAD7Ug8uW!?nG z{ZaG6ZWR3a)b%@Eh0$|30{3RPU+~CEAe=snVAkrD$k5B#Ir$Qv$&loopUz>p+J%A) z3b6&8pFc<94W`U;wvvqytJ?$!CrnrjlCuLPHD}UYVp4r}tr4nW_1+m8XtEF7@YYXISjVLb(8sd_C~+GO%xEYCr&(M*uhgXDzu50?>o`_Sr5zmI^oZ(<)?_J|!9dEobs;Yp zZl6kGN%KV}@DIke7urnobd(#QJ6_FzuOP2`clT)}mm?8BWW>DXpe)y+XZLG;Cwhup z4u{f~TOckp`&~Ow&ZA(6{BuJOvE7XjHFkpRUG|%QO=rJ6w^=HqwNAaGa!^ ztFlmK&u%%}1*e{p=3F*@3#Y&R`4r^A1~mWZ+NE~$FmWplsbyB6%!5v{!uDsOwH(NU zg)d87)ZQ+Gt|2&GZd^S(j1!BGqHs%)E3o^DzV5W0xL{fBVE5gPlaojU8>@%~Y)OqS zFKq*&de1A6f*a*~{T4UFxzcI9xco3BysN-Tdkn+%0*q_fp3p>&27l~y)Lg9X@FHqp z6i~*KK%P2be-38&_&r2_g7sYcTMzpA%=t#AgVQT{*bn#<_}reIjzJ=@(|6=lC)fh* z?go|jc3%lyXQ%!OVnRVverQ-qxoY{da@CmL#{3~*=#7#Z{52161sf_SQ@|#G*)4Tk z0&i!d(;~lTK;dK(YsXA}&)D$hc#F&_(9RSbNi; z9&r#+5|!W^(DrU(T_K-@XW<;C@(mWb+|N3n^m6jt72Kq^ri-rPZ%JXzQ$D8lA+Wmg z!6IBpt`$3ENvF>Hx69ld&Cc9}B1Hi#LRSYL}C$VNs7v}TfA|qo(Mc8+b1?2jK z{qp%@5fjJsYB~-r+TK%?mYWDv#?6cNl4t96$*ooh(mxCRZbFO-{I^3}UUQ$F5FVzA zU`W=CA`k0Bur`~unk)f*@`)MbOpZ|MHs`}Klm0(;mDrV(P1|{OqJnm^#aV8(D-PrD z(qKZbkl~sTm4fv7>_(Fku$)SycQoIJ^z}Mycj)edF*td1)(sw9IXRmW60qMDiL+KS zTF^A#?&Ig;S=MJEFN7c0%k6jEEZMF!4=0E>?xbH9NVn*!i(y!Dq(wdW)xnn@C*7}1 zTNWR7Hrqtvv9UTgX&D50ZUqf(?=Wap^4L(FXG99}kw?Pcgw4XG&&*lfIhk^X18>qNFuY_=`qC zyvG8^%{1Q{P{{iN+3k!Ypzv;~D)ER&SXs^Zvd~S8W}3Az?!EGedC~9Ov&whv7rr|} z`c*|Fu#KrjcNPu$Y}M%ZATJ6j>({rFGw_PWb& zl}$Iqi1Y?#|ERZB-OW$jMHFcPr|CoAbv{()mm0eP4V5=FBQ4F~pUq-UJIx~Gp~;&r zhWgzEm57L8)p$=BjCPyOvHMT3a{q4R$`zbuhmk*#kk|!IDCVJ z$z+%z8BcjGuh3~z4P-vk3m{OHKSS#uE@LZi;BV2F$dAznWP4iE>UY%C|JBeLo0YY$b>r zuJY|dwEJ#v_nRV5B%<+;h#6=QPICVa8IIjBLAp|JEqWfxGA^&QKdkTbZw<)`SU+k| zSX}%AU8_+yL}VG{#XhDU|KK)axb-ku9?5^i{5Oc?ohPPTvqD3i1eEZ|5%oVh z?DUDSAF~WCBK=aVb?Sn**6RMbpJ6GAA%Qt+O}s_K(hMife%*gl7scEbbPP$2=}p+n zh93AA*rra8g95Ym&&RP8$9NL;1F*3~H2=ZQ)hJVldG~YwA~2+#;e?{g|L-CzccNXy z4+ZpPzTdGPf6;9<>a;NV0)}i|S;;9#rRtBA|E*y!hE)?$zp8{mY*@#K`hRCe18w^O zlj3J78p)n8E#t&VZ!6@#VL$MX6Ubv^)ubTP5VnYS|8Ke9ZqSkBgZ%{j1tUUFiRAso zBrJB?|MxtCVpm8Z(SArVmh2M$E-6SvyU zG%Q81J64g;;Zlx>sa@mycP()QMd8xEoKQYq@L@$heMu6vgZ-TL6uLl zbUO6k#{n>T?^Zs^ZJdk}N&hl7!oN2&l*_mmFc8O7g!Acc!TE0nh9R+dc;>pkFsrnt zav3rIf&%|;vNfiBE(thY1?7+!GFd5@iT~|BSOXE9A6&E(WF~u&k?mLGC}C%2093&T=0{@B3%-vUdM z@LP(%S2V_J_y-9KSOk4ZfS_YZGhduZLdPQKKdS!f^i!}+Ss03V(K_1zg)HKfl?|!tRb*aiQriAs=_-9`&GEWI!^#^2< zi-qYsMwT?i<7)kLJrs_%fL2^YXyEK4aS>sQ*+1q;sfli`D|w+HzlkX;W2-9u?+~GI zi66{Sacq&v`&uUdXEeJ3@}%T+E<_7iq*=q^ry2hl0vXhHIQSk3sXy6P7kt0B`*cbd zbGOqkh0GevNxXT6BO&CHwPW+!Qg$iqzjcBOz_b(*jc5Ff^j1kJqw|mlWqe8=naL?N(3U)Abob# zX}z1RviNV;@W7TJ;(($m?A9|ovkF!M4Tsxt!wO*?1pf{&C;wi#J%)v(0O|7bIneaq z1Iaxntj9`Y3oYOKj<{zBfeh%9r4|2_L^*kBNu3myuNa`b_VY(vcH%LU{J-|00*@m> zn|!UK_q5%-5hfB#9)~VHF^VI{gOx~L&}b~>Ev8zy=T-f8@+QRtIG~BPS7H%NcSNBb z3>l)_0>j}37L5cc48{9p2MXJ5_E1nQZtve0pH#9PM*h71O0*HwXSq{Dd;c~L>bdJP*;5*ZieufhinKOzN|}qIEp|W zX~N>8MH@YWp%~QuPK#22YC#X<-`y2El{Jq6dOF1bSu#6pCuVR`E==#hxg4*XvVrFE zF9T-xtmC@amZMhwZraf(-$U63ap!@!SK^r&Jm#I&P2TsEU2MaWV#8+hHOf4+G7_;b z-&qm<0cej#z0D(?LV)E?etPok!YN9iNLe2o9Qa$8Xi^|3)M#1>zy?K0IOysg$1yH! z*aTpmtJ0sEE2qWK4vpN%=|&*V{9Q*CA__spNIWtc$(Oe=)=QTa^kz#;I}$R=glODgoo(mO7$jB20c^FTxx8vj>c@~5QQQ8 zmFZUVI>g?qA$9{8ukhiNS3bgqc@*=~q8*`30%`;4>#aLA^cpU-wC73s<%f%hfC0@^2nJhj9#>p2qY9_0{hQ>yZ7z7j0Hmy44Q_NK|NJha#zF%fKlb620%#Nc*Y>1<*k$ne9lqtpG%_sGX z?P*(5h85sV8Fzw8GHEZ^;wbq)lB%$Y&VG>-UH|;1XxI z`Mp8kAXun#<3<%77<)B=`PUkh+0gNaoh|V$Vkn~du0yU5U2_QBc5}MI%@sv+Gs3$b1t*&uu zVqGMm?<@}=;)`XIY<-?|KY2eFdPi|U(|riQUVpx>^I2u;MATFphy$0ZJHW=%CEZ!a zVMvuq#+moa8|si_k9T5#B*o|{({afdYHVb1K!~(Sm?kgTSGzgE zSS6)f6VU%YJEoH2?4`r@+`>>p(+agx(JPq`qc({TgC_MA?i1x-*;gWO&j6QsM2pgg z^fWe02?xWxg1HLH($W``Tz!}D!!GIg0nkhQNe01vK-AiWGONXk$hsE0tpQQd}~sU!tFuo>T?bqx&_XT>BG7;`;sbY5<@ zZJSIduMguAd(%dv5tDYbL1J!Cm$&woRYjunLaRVG3+}w zIsNf$2~As*4Y?_u!wU1Dl$d+al{LnA8ruxyXXG>4WbL+MV;VLC=9mon5l!De<`-nq zC7bLyqiVLlH`LLzMmd=6nRk7-t_DZoQ?KpZ*T8a3Hi&jDLPV<6?e1gSQm>f#oZ5{e2i;=t@P_n|afg19eg%4aDKw)d{Z=Ry9uYuc zWyksFkeDLs z!@X$o5YFDs%kQSKAVmsNHjTKW*|>45g%NzFb}~7E%jU6E7M-o*hd?#$I6VT3cC59Y zdhErd78zXvKFuJ$rx z$n@pBYm|Xw6-W@+2d;qkGu_A1Ic$@Pv)rXVc|N4pB@OkyMc$!t=Q+XjW_k<;;t zXcXTIC!j+pariCi>qOc3gLcMBeWmBkOR>XB>T7(*?E$s(%+Zjm(x*di#;Qnf(TUDK zyw4tUmpDcF>ngPl4kmYhHXZujaU5LIF?9)Nc|Oo^9fgqZ)P-U>FJc5kl9{^hHD6y2 z{LyYcAaHh_#DR8G_p{^UqXcx{<70dVYQ8$~M+>ENT@zUhZD)<;bt1+A7_tlNPWe~E zlT+fT2^lBf?vL7RcO#T!Dbvo^3Y1lN)cT>5VL!Wsp`y;unu@Tp?oZ+!;m3ulVJ8s;#y;sVp2UpLbF5jjew% zot)my*K-qM#A6Xj;{dSykR?#(;{`m>>$95sG-4)GjtooSRXd9NsN{P{Dq|)jX(E%S z{130?h611jB?^6#n5>(G45xSCc{$UyZousjXCyF1uzwdVFuhyV{%yYFkFZ78^4aA~ z!tnI0y^bdBYJ-^z@u!@>3^n>hxw9Y?i?zHyVkVFLfzZK*|yfb;U7?{)*(zt{+C%c~VG%H9-P0LP}mY5ul z@TQbPuDq^H8Lqt!VnE6wa}u&sf@j;ZoVub3$#Mes%JWrP#q(MD5iu6ZO^%62#E=FBVcj(@k z>1iju7E>*yPjX4+<27V>s}2NQh5?!}`ZGl`EO$@H$|7vmzvUS_+(K47`ywyTw+1eM zYS~IDdg_0_%L@$OQ5>Xf0bRX!xWQ37s`0!z<)F>$4l5igFS(H?k$67oy-Z)+6iUYxqE*C(_}MbcAuRQlLJ zXU>=M`5%dG=PhbP#B5Sxd^ZIdcGID>-Y?H)H5IxIQVS-;l_W?BqTWcJ_YSAPZ(AJ` zFB6cKO|$@-E#HJ6c#c_j{|!`SNrn=SBQF75u>Ov8LU!U)gqj4m4T1(9+fQy2p{6 zT>11aLf;cSR(`n>&NTD~BYGqQ=nQrR9Kwvle0+~Cuewe4{=D~NGnXk`^Tk~eR?(eoObMe`q3>qd*rz$&;_?YCP+=Xk$YsJI} z$m@x|WOF5zDiyzv@{6b^aB-nk1A_MHSdV|1f`OYDQi1o0?T9oo@L?*U1R%)*I0+Dr zo@yoQsuym{ebh4RZV#@W52aa#OB!p1mK?j8@zBkJ_OU(9Jp~7oTpT9mPL4mCLsF>I z>C*VOwPHdiS=p$%dN=<210x*t&n_oc07kOo=3M^zY!0-iSyw7 zgh(avh)Q^E%UNKn$8~nMohcXVi;UuabH>oL;uinnHQ50f37GtZO%$9w5dJy^$>Q@E zVICwZ_klUqbvP`9V&uY2Vk^_qi{je*@CprcUm7xFh+SZHpT_T%)_LA`-s!NnXm-3V z=s7!pt4F_g#0`2H%iMPLK_%e)rJe78V|!pdvAd%r>%y-=ma7QR0(sUDXv+VlLnucA zus}yh4wrO=6!%|T7k9%j0Ns)0$##Kd&pS7}N8) zz-w}mOtF7eYMdrDyZ&er!@~s2`Q_H{VvyRRN0saYs%m@@M)GMxt`mCHbWah z8h+##Uz4Y!7b#l_3JF(#BQpAU&<2bDbW-1h>@E^b_4~g;C=DNLI&IQZQo$)b2(C*V zDQ&O7H$7rHq!6y*DYeb2P(pazST3qi5b?9)Pw9J;BPKxMv(b9hl=~i9Qd$DV&Ww~L zWS!>x*W=lv3(zpl8)QlGY1k}VE+!wI=%b6_mc*?}an$cGZ?L6M?tR-4TG8<|0TFiC6`6tHJ(}S(-z@vUw^f5yR*9+$j8y? z-zZB|mHp|NLg4u>|K(l699>m=duEB%YEeU{&)t;x<$Bla(R#Jz94#1<>6#&zBvIl7 z5X(kBdWQau6BQ|v{OCallthAaZwFba#_p5ylfnp}{oMeg=tPT|C&t2UIc^2K0h0Tj-Y#5;;)%@Nd^v$R zY=wC57mX;s)}e6~%_!#;*8X6V>$Qy1qJF zQDwO0bs8yfpC%~nrWR)d?MfTHN#A+Nu|FJUVdXD(QDnyMXg4S|C2EI)- z+lgGzUAQuwH4MXO)g2l^#b^H(qx9-p+s^A=Pc-_%=P?|&J9@(dGFI-4A-fTZh${jT z8K6XXA>}xA+Kj#akiwXmj9KMytQu)c4Jl^B(8jrKcUrV@E!G8BzqqQFGfgf?yO6^L zWpYXLVnQrqFoZ{1jGtX`!S@GHyV!7YS1mF&HRT2Y1Ujwa^H)YKO>=`D!&?0w=w}ar z$n}6qr*`LGg8RB1O?Twe{{}o&v*h_QAOeUnv{h(yhzV5LRS1BR=%-whDPQjem~QC1 zo<``LcC=M>D~qu7?_M{;nG`YH1gq9sqwu>4ktWQnT#{Z+TWLE!r^P7vkB+hb>H%!# z%Ey4S`JD8!y8|Mbb-x$g^%^%?*{nVBkyYYFqh2D4^TckLacCjdi?I^f_HYCT>V4^V z2iAS4Om5dW1GINaI)BOVqrXNOzZf18topouMT@U^;%#VcLE+p&XxW+`rI^$%w&Z2`(ba z^r^3r>X=@sXHi6x^b4H${fffvpK@oL3-N}%UDF>Oen{@|G*DAM?E7E+Jm`Tn_?p_)Mg^6~UX(W2x&Q~~Wv2ZutP;9G};Mb~IAh;t_f*FlVb zq0Vw6x0xQO?ws+_DHe_YMIwsz*u>a4#c0>_d0&ugFI`Q*(eoMpn#n11Fa3)0Omeiw zXX=t4o`|~byfNMQgYQDS#h1m=j7cgi{u$>+o8=;K!Afm_uQ)NKvt_ZQtZEXg?Yy-A zhQQOE(c#jM^xMG%FI>m0?;C}kgi}7>xw`r&27Ur~(NaV%oSBlH3ebY9RVtAGoqMlF zfB$;n<3fbawin{Wg!nSNG}rKRUql=w_1yW4V;NSHelBVRS21YiomOFY38Yq*%mVP; zaQrM78wQ+2;$ODa1C`N9Uuq?-?22YHydRzTnuxA=4{+t1C`XZi6ByHjz*qVA^&$T22PosToMui~ z3?JN3%GaC^uMRjM;NaSlrwmBQZ2xUoBZ4DNU~%So(!K(AHhxF;iegG4C7)RQ8YVS+W>ik4=1t99MV_M?&S zqz{F@+TziX1T<8cPH{d6J~a1cQv%$FpATUr#(#Bw^%$2KsAsjBuZl0PY?ngiJrFcI zy}ST^)rQsde8|%|zWtgk_^EDoYk%szka+*3_CuRejvsKn94=}0p4zxU3^hek^(^-8 z*!H5K2N}jrX)#gt2#HlUz%f>_@xl&i@tCyb7!Sn|4Rq`U7jxNTDY7tWKWn}OBs3`U z&Sa=#sLZvzELrX5k>Y~(Co&6d*E?CvCJ4aW4GS5zpvi}uemAQ zNaR$F!ue!(4<5lbD;+vE(u@C!#xDYwvh8&|&avBKmYulS7i1B1qa@2E%Kv!%#Z1Nq zSx>$E@Vx7_GO6Xf^I7#F-I~yGcItAnH_uVC%O*3!YcF*gkhI#>A)oDhz2(6v=x#B# zS^hMd%#^(FGa{DhlWamC+9?$kfsbeRPQT;+L}ETXYC}?r^9Tl z_{jOy-eIl4>)ls=yjNe@MaLr`)l&?pDU33xNJs<@u;%e5{-mOUmClSwNjhTfALH7U zo+$N0t}zKnNxRp{HL;N{W?qvu2LmwWBfxf{H`nX}6=T$U-75@-5wc?(ds~EuxBadJ z5ble4YJAsuh#O`sN3DQd+$8^N#%!sImQClTm`2yUKrq2CfDlw3zhi7xzI^TrZ1VD| zX2q`jZL_k(^}Ux*4CzfV={7B*SzZqo&u7x)DSJSm@_M$IAIy7skUwpitjaQnki~AT z33^Ck;FcK=#MLH&c_b#q`Xn9GK-!NvQzuwG1848F*0O)}!sB#6;`MMng%~5?=F^_> zjU)`mk9$8|t(%ydj1ZZ2VevmoFEv`zRYek#2NTW9t$9B(JbVAT-#>nG z$DglCcAVsqJL}w*wU{ml^|XLh$%c72;BIDUb2%hq&}xkP8adQ@9S2h8?b%I6O)Rf! zl>&^O>IRUqfVo|k``boRGi^prKu4DUg5v(+xXPq2=%%%EfYnQ7paIvaJ{Qu840a8O z@4U)%!Jcq<#yBtXEsg|Gv9>S3@4H~h)zdjUa+XKVueZ1{?A`VqIIWvyFLkMfmduV1 zC?A=(6z+HOqbd7B0-Dx6m1NlnZ*v?c6mqYRGZtIdn3Re~0QZs!NTVukG8^roq#mn- z066LIBE7NN2|`FhG?nM4I-g9+;w*=-Z;MB`EY(dIrjMsx+3k1JprVh_@%U9;x{hM$)Yd&SqJXsiRP}5_WVC`foHa4`?xJ5ha#u&7pt$g4Nd67HnsEjGSoHjk~n{ zO>sH#_vx8Uuns!570gQ{LViXsSo5+Q2CO(Ptg;x*(;vD?Rm7IWJYpBRmt4CTX( z_@nU^T+#5~_fu!yEh_fQ&Kz0zXYZ&tSbi#h6ogS*E#4CoJ$vib!YFcO968F&Pouzt z?2g^_;R{pMSLU7~jOcFNLGIF?SER%ma4 zoY;H>N@l^`hCix9Bgu5RE0c3J6l34mm^AoiSIziOpg*!GwzkJ=1Iy-&oXJ$4XsyRF zyTj&2c!n*R(OCu;ZTFfFM2GLwiW~CF0|?@JeA-n07;%pFnan(PJ50+AR+0LH!9uGj z2>jULdUJCFSuYKN_S=i!-ZH6@&AfKZaPz+;P#7$W4u4F_W_901(HVK zhtgsvuJtPQI!z?$P7+xJd@ft4f+}iL29lW`QvQzBx$~ZU74}CFiMDT@9gP! zw?fvf#0Yf<9Xr|QM)eU8wHd-!h{5svkGPBsdS{@vv#?G;#nkZ-*3P@9qS0pW&uY1% zzMX8x(NfKS#@#I(pO$7RGNj95{lKcT#{xv47HjQ+vsHR}3(NHu@mDg&tlwrO8cofH z|GKZXDcg9@Bcm)HfLOyO6){LtGtKkQW-yD_k`rIjv&Zf1`fRC|_`3h*Q{SpA?1~ci z85h|s`j^P@7NgX(Q}sjbB7EP(B-Q2n9e7_A{o73&7=y(0WI6`T`k2Tf0<9(+2HWn zUXD+~FwF)y8raGb0n~70vl4lsB(#aRa3GP%^VBcN%Sl0Y3ANXcp^j!}D}}CmvU~#n zSKZbAGrhpkU-Gad>#E(lqs6CO451rIMOrlvDa@i@m=Sqc7rD|}GnZKRmJmjbR9KlY z58Wj;iWG_%n;KK1Jk2m+Mss)njQi&M>b&{BI4{oUe9k%dy`}MR%nK=;^tb}7=4%m})}*8}X9FXiiNR1h8e9>2vCnES#QX=$1RO9DsL0Pw4>)i|M3O85Zfpx~Q;W9*U6i65{# zXT7#z;5oU$k~kR=^-)osFwkC&JAJ;0#}hZWEr31#g3=n<-g9+VvJ%!y#kKvYukVNF zZKK0E8@8B;^0SHwL)>M2!Rjuq@NPUkSmu)-jqLHwxzAw zZ^XNr;Bw9f#4sI++x+ccU*m3x`DN2~`)(~~E3oPXV=#{0dH=v~>IHf$=h2sc{*+UO zxhS5Bx`>D9U|}#j!nh|Xz7tMiRaSOIMn+ycyKmf~=|-qnK8Hwjb8}lIa56wpB6ZEK zz_;s>@XMFz`-nAKmW6YSBZ3_PnV1l!j|P&1A333qNL;)g8-JMuvBJ#IgSDWD?;RPM z4}GydqMFdR+83G^rb)#f7L#f3()Do{uvqLOe^QWIR#w)&I5RLdh1Ssz9H;etDCyaS z23PK0F0ZIqxOsaIPFdhlWG7x9@S{MBwct|jln^ZUS4bvx4R##$3M}g--GT>hQ4@UJ z>^&ypm%9IW*Tj@*L!ZtNM5E3Z$D{$n|Lhm#dnJ6x>_C=Ns)ka6j>~mrODE90u&QOj zD}7fiyf5k7!*+A%$9k@ImTk?|K^%qYJvr&FXI~?IQQ0(;jq=1CQAp;+u^vl(VWVrAd<~Zc zq81cB8o+4n%Uya?hC%r8=+;jU-zzpx?!$Jh&+$b%q?TJxR1*sCKh~!l&;Mc-q$G`} z$J@Oz?lU%Z^_%^%bCK7abtQ z={wgw5Ap6tLNPX;FlvQ0u_f@y)z+1vwl^pvP4j3~O1it=`V-||=)B}t$$qQtq0VL- zUBJB5T~PN%M@>_pE{T(EaEY?YWi118_5tu?<95u}}O)ST-r9)L2z;{StnVbZ{ z?uQ}_`afSXa(PG^?w)1R)UZb36Ny6d_*8J4ti=U=hrxvx4|+n*%_(!co~aCL&T;+B zXln8uwQS!!88Vwu6}VKJt^s`4Q#UKY2qAc;wQrkumFxk`93+0=hqMlt*oQ}8{iQ#N zub0M+ng1n34Y$GiWMMtD)~OS-wRu%z*#lzHVd3~TFFtOa zjUh~T`Wga9he{S>MxD*L4e!|@Ts`3Pqs=8M-EjlS^p8O3xZ|QfWNlur~&_BHU3BUl?0ddu52d*Sl6X`0dPHj>R9E`pyYo6^O3P? literal 0 HcmV?d00001 From efee9542a39bba6080ac68347b938a22bc7f8420 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 15:56:26 +0100 Subject: [PATCH 10/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...ng-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index d6b1799c79..f99434bc0d 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -14,7 +14,7 @@ tags: summary: Having worked on developing Machine Learning skill definitions and L&D pathway recently, in this blog post I have tried to explore the evolving role of test engineers in the era of machine learning, highlighting the key challenges ML brings, strategies for effective model evaluation and finally tried to outline a roadmap for developing the necessary skills to excel in ML model testing. author: snandal -image: +image: "/uploads/Evolving%20with%20AI%20From%20Traditional%20Testing%20to%20Model%20Evaluation%20I.png" --- Machine learning (ML) is no longer just a concept for the future, it is now a vital part of many everyday applications. From personalised recommendations on streaming services to detecting fraud in banking and from self-driving cars to advancements in healthcare, ML is changing the way industries work. From 371cb5a2c3b684437a06b4e9f7443fb497a5c4de Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 16:10:03 +0100 Subject: [PATCH 11/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...aditional-Testing-to-Model-Evaluation-I.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index f99434bc0d..8bd56eb3a7 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -44,9 +44,9 @@ We now need to do more than just check if a system works as expected. We must al **Dependence on Data:** The quality of an ML model depends a lot on the data it was trained on. Testing ML models means we need to deeply understand the training data and its weaknesses because any bias or missing information in the data can lead to incorrect results. -**Performance Metrics** Traditional testing metrics like pass/fail don’t apply directly to ML models. Instead, we use statistical metrics like accuracy, precision, recall, F1 score and AUC-ROC to evaluate model performance. Understanding and choosing the right metrics is key to effective testing. +**Performance Metrics:** Traditional testing metrics like pass/fail don’t apply directly to ML models. Instead, we use statistical metrics like accuracy, precision, recall, F1 score and AUC-ROC to evaluate model performance. Understanding and choosing the right metrics is key to effective testing. -**Model Degradation over time** As new data becomes available; an ML model’s performance can degrade causing a problem known as model drift. This means continuous monitoring and retraining are equally important, adding complexity to the testing process. +**Model Degradation over time:** As new data becomes available; an ML model’s performance can degrade causing a problem known as model drift. This means continuous monitoring and retraining are equally important, adding complexity to the testing process. ## Model Evaluation Vs. Model Testing @@ -99,26 +99,26 @@ Consider a hospital implementing an ML model to predict the likelihood of patien To effectively perform model evaluation, a test engineer would need a combination of technical, analytical and domain-specific skills: **1. Understanding of Machine Learning Concepts** -• Understanding fundamental ML concepts such as model learning algorithms – supervised learning, unsupervised learning and reinforcement learning, model performance issues like overfitting and underfitting and the types of models (e.g., regression, classification, clustering). -• Familiarity and understanding of model evaluation metrics such as accuracy, precision, recall, F1 score, AUC-ROC, mean squared error, R-squared, etc. and knowing when and how to apply them. +- Understanding fundamental ML concepts such as model learning algorithms – supervised learning, unsupervised learning and reinforcement learning, model performance issues like overfitting and underfitting and the types of models (e.g., regression, classification, clustering). +- Familiarity and understanding of model evaluation metrics such as accuracy, precision, recall, F1 score, AUC-ROC, mean squared error, R-squared, etc. and knowing when and how to apply them. **2. Statistical Knowledge** -• Understanding statistical measures - mean, median, variance, standard deviation and concepts like hypothesis testing, p-values, confidence intervals and correlation. -• Knowledge of probability distributions and how they apply to model outputs and uncertainties. +- Understanding statistical measures - mean, median, variance, standard deviation and concepts like hypothesis testing, p-values, confidence intervals and correlation. +- Knowledge of probability distributions and how they apply to model outputs and uncertainties. **3. Programming Skills** -• Proficiency in one of the scripting languages - Python or R, which are commonly used for ML model evaluation. Familiarity with libraries like Scikit-learn, TensorFlow or PyTorch is advantageous. -• Ability to write scripts to automate the evaluation of models, including running tests across different datasets and tracking performance over time. +- Proficiency in one of the scripting languages - Python or R, which are commonly used for ML model evaluation. Familiarity with libraries like Scikit-learn, TensorFlow or PyTorch is advantageous. +- Ability to write scripts to automate the evaluation of models, including running tests across different datasets and tracking performance over time. **4. Data Analysis and Validation** -• Skills in data cleaning, handling missing data, outliers, data normalisation and feature engineering. -• Ability to validate datasets for biases, errors and representativeness, ensuring that the training data is of high quality and relevant to the model’s objectives. -• Proficiency in using tools like Python’s Pandas, Matplotlib and Seaborn for EDA to understand the dataset before feeding it into an ML model. +- Skills in data cleaning, handling missing data, outliers, data normalisation and feature engineering. +- Ability to validate datasets for biases, errors and representativeness, ensuring that the training data is of high quality and relevant to the model’s objectives. +- Proficiency in using tools like Python’s Pandas, Matplotlib and Seaborn for EDA to understand the dataset before feeding it into an ML model. **5. Familiarity with common ML Tools and Platforms** -• Experience with tools like Scikit-learn for traditional ML, or TensorFlow/PyTorch for deep learning models. -• Familiarity with tools like MLflow, which assist in tracking experiments, managing models and automating the retraining process. -• Knowledge of using version control systems like Git for managing code and model versions. +- Experience with tools like Scikit-learn for traditional ML, or TensorFlow/PyTorch for deep learning models. +- Familiarity with tools like MLflow, which assist in tracking experiments, managing models and automating the retraining process. +- Knowledge of using version control systems like Git for managing code and model versions. **6. Knowledge of using version control systems like Git for managing code and model versions.** From 5064fa2bfe01c1eddbd3b4bfbce3e0cfa1c4cb35 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 16:16:53 +0100 Subject: [PATCH 12/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...aditional-Testing-to-Model-Evaluation-I.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index 8bd56eb3a7..af57bd276b 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -99,26 +99,26 @@ Consider a hospital implementing an ML model to predict the likelihood of patien To effectively perform model evaluation, a test engineer would need a combination of technical, analytical and domain-specific skills: **1. Understanding of Machine Learning Concepts** -- Understanding fundamental ML concepts such as model learning algorithms – supervised learning, unsupervised learning and reinforcement learning, model performance issues like overfitting and underfitting and the types of models (e.g., regression, classification, clustering). -- Familiarity and understanding of model evaluation metrics such as accuracy, precision, recall, F1 score, AUC-ROC, mean squared error, R-squared, etc. and knowing when and how to apply them. +* Understanding fundamental ML concepts such as model learning algorithms – supervised learning, unsupervised learning and reinforcement learning, model performance issues like overfitting and underfitting and the types of models (e.g., regression, classification, clustering). +* Familiarity and understanding of model evaluation metrics such as accuracy, precision, recall, F1 score, AUC-ROC, mean squared error, R-squared, etc. and knowing when and how to apply them. **2. Statistical Knowledge** -- Understanding statistical measures - mean, median, variance, standard deviation and concepts like hypothesis testing, p-values, confidence intervals and correlation. -- Knowledge of probability distributions and how they apply to model outputs and uncertainties. +* Understanding statistical measures - mean, median, variance, standard deviation and concepts like hypothesis testing, p-values, confidence intervals and correlation. +* Knowledge of probability distributions and how they apply to model outputs and uncertainties. **3. Programming Skills** -- Proficiency in one of the scripting languages - Python or R, which are commonly used for ML model evaluation. Familiarity with libraries like Scikit-learn, TensorFlow or PyTorch is advantageous. -- Ability to write scripts to automate the evaluation of models, including running tests across different datasets and tracking performance over time. +* Proficiency in one of the scripting languages - Python or R, which are commonly used for ML model evaluation. Familiarity with libraries like Scikit-learn, TensorFlow or PyTorch is advantageous. +* Ability to write scripts to automate the evaluation of models, including running tests across different datasets and tracking performance over time. **4. Data Analysis and Validation** -- Skills in data cleaning, handling missing data, outliers, data normalisation and feature engineering. -- Ability to validate datasets for biases, errors and representativeness, ensuring that the training data is of high quality and relevant to the model’s objectives. -- Proficiency in using tools like Python’s Pandas, Matplotlib and Seaborn for EDA to understand the dataset before feeding it into an ML model. +* Skills in data cleaning, handling missing data, outliers, data normalisation and feature engineering. +* Ability to validate datasets for biases, errors and representativeness, ensuring that the training data is of high quality and relevant to the model’s objectives. +* Proficiency in using tools like Python’s Pandas, Matplotlib and Seaborn for EDA to understand the dataset before feeding it into an ML model. **5. Familiarity with common ML Tools and Platforms** -- Experience with tools like Scikit-learn for traditional ML, or TensorFlow/PyTorch for deep learning models. -- Familiarity with tools like MLflow, which assist in tracking experiments, managing models and automating the retraining process. -- Knowledge of using version control systems like Git for managing code and model versions. +* Experience with tools like Scikit-learn for traditional ML, or TensorFlow/PyTorch for deep learning models. +* Familiarity with tools like MLflow, which assist in tracking experiments, managing models and automating the retraining process. +* Knowledge of using version control systems like Git for managing code and model versions. **6. Knowledge of using version control systems like Git for managing code and model versions.** From ce3ff32852e95c205226a7ca22fcb2f22abc2fc4 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 16:21:56 +0100 Subject: [PATCH 13/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index af57bd276b..0b2c1a05c5 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -99,23 +99,28 @@ Consider a hospital implementing an ML model to predict the likelihood of patien To effectively perform model evaluation, a test engineer would need a combination of technical, analytical and domain-specific skills: **1. Understanding of Machine Learning Concepts** + * Understanding fundamental ML concepts such as model learning algorithms – supervised learning, unsupervised learning and reinforcement learning, model performance issues like overfitting and underfitting and the types of models (e.g., regression, classification, clustering). * Familiarity and understanding of model evaluation metrics such as accuracy, precision, recall, F1 score, AUC-ROC, mean squared error, R-squared, etc. and knowing when and how to apply them. **2. Statistical Knowledge** + * Understanding statistical measures - mean, median, variance, standard deviation and concepts like hypothesis testing, p-values, confidence intervals and correlation. * Knowledge of probability distributions and how they apply to model outputs and uncertainties. **3. Programming Skills** + * Proficiency in one of the scripting languages - Python or R, which are commonly used for ML model evaluation. Familiarity with libraries like Scikit-learn, TensorFlow or PyTorch is advantageous. * Ability to write scripts to automate the evaluation of models, including running tests across different datasets and tracking performance over time. **4. Data Analysis and Validation** + * Skills in data cleaning, handling missing data, outliers, data normalisation and feature engineering. * Ability to validate datasets for biases, errors and representativeness, ensuring that the training data is of high quality and relevant to the model’s objectives. * Proficiency in using tools like Python’s Pandas, Matplotlib and Seaborn for EDA to understand the dataset before feeding it into an ML model. **5. Familiarity with common ML Tools and Platforms** + * Experience with tools like Scikit-learn for traditional ML, or TensorFlow/PyTorch for deep learning models. * Familiarity with tools like MLflow, which assist in tracking experiments, managing models and automating the retraining process. * Knowledge of using version control systems like Git for managing code and model versions. From 364b10737b2bf8b5e6d17cac475d88259afd5437 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Sun, 1 Sep 2024 16:34:02 +0100 Subject: [PATCH 14/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...ng-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index 0b2c1a05c5..c5c93a3312 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -1,5 +1,5 @@ --- -title: Evolving with AI From Traditional Testing to Model Evaluation I +title: Evolving with AI from Traditional Testing to Model Evaluation I date: 2024-09-01 10:00:00 Z categories: - Machine Learning From f3e39369fe5988e0e7ee006ea1e6010821915179 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Mon, 2 Sep 2024 09:31:38 +0100 Subject: [PATCH 15/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...ith-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index c5c93a3312..25d4867817 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -149,9 +149,9 @@ Here’s a step-by-step roadmap for a test engineer to develop ML model testing By following this roadmap, test engineers can successfully transition into ML model testing, ensuring they stay relevant in an evolving field. -## Conclusion - -Testing and evaluating ML models requires a mix of traditional testing skills and an understanding of statistical methods. With the right testing strategies and by following a structured approach to testing, incorporating data validation, metric analysis, stress testing and continuous monitoring, we can ensure that our ML model provides real value and meets business objectives. +**In Conclusion**, testing and evaluating ML models requires a mix of traditional testing skills and an understanding of statistical methods. With the right testing strategies and by following a structured approach to testing, incorporating data validation, metric analysis, stress testing and continuous monitoring, we can ensure that our ML model provides real value and meets business objectives. Additionally, staying informed about the latest trends and tools can help handle the challenges of ML testing effectively and make a significant impact on your organisation’s success. Machine learning is complex but also an exciting opportunity to explore new dimensions of software quality. Let’s embrace these challenges as a chance to grow and expand our skills in field of software testing. + +*Hoping to publish my next blog soon, where I will dive deeper into evaluation metrics and how to choose the right ones for assessing machine learning models effectively.* From b75287315e0b6fde09c124fca376bce061f854f8 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Mon, 2 Sep 2024 15:27:28 +0100 Subject: [PATCH 16/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...ng-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index 25d4867817..e665d63680 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -125,8 +125,6 @@ To effectively perform model evaluation, a test engineer would need a combinatio * Familiarity with tools like MLflow, which assist in tracking experiments, managing models and automating the retraining process. * Knowledge of using version control systems like Git for managing code and model versions. -**6. Knowledge of using version control systems like Git for managing code and model versions.** - By combining these skills along with domain knowledge of the business context in which the model will be deployed, it can be effectively evaluate and tested, ensuring that it meets both technical and business requirements while remaining reliable, accurate and fair. ## Roadmap for Developing ML Model Testing Skills From ed8a828b08975cde2c503cdf702a4fb570e4ff27 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:54:07 +0100 Subject: [PATCH 17/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...ng-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index e665d63680..ad908d7273 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -54,7 +54,7 @@ Before we dive deep into discussing possible strategies for ML model testing, it Model evaluation and model testing might sound similar, but there is a key difference between the two. Evaluation focuses on assessing the model’s performance in a controlled environment, while model testing checks how well the model works in real-world conditions. -Let’s look at it with a simple example of a spam detection model that filters out unwanted emails. +Let's look at it with a simple example of a spam detection model that filters out unwanted emails. During the **model evaluation** phase (validation mode), we will use a labelled dataset of emails to calculate metrics like accuracy, precision and recall. For example, the model might successfully identify 95% of spam emails in this dataset. From a3a0d8b9d2465f60545ac55355da22c2058f8211 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Wed, 4 Sep 2024 09:06:04 +0100 Subject: [PATCH 18/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...ng-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index ad908d7273..ef3593cdb6 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -147,7 +147,7 @@ Here’s a step-by-step roadmap for a test engineer to develop ML model testing By following this roadmap, test engineers can successfully transition into ML model testing, ensuring they stay relevant in an evolving field. -**In Conclusion**, testing and evaluating ML models requires a mix of traditional testing skills and an understanding of statistical methods. With the right testing strategies and by following a structured approach to testing, incorporating data validation, metric analysis, stress testing and continuous monitoring, we can ensure that our ML model provides real value and meets business objectives. +**In Conclusion**, evaluating and testing ML models requires a mix of traditional testing skills and an understanding of statistical methods. With the right testing strategies and by following a structured approach to testing, incorporating data validation, metric analysis, stress testing and continuous monitoring, we can ensure that our ML model provides real value and meets business objectives. Additionally, staying informed about the latest trends and tools can help handle the challenges of ML testing effectively and make a significant impact on your organisation’s success. Machine learning is complex but also an exciting opportunity to explore new dimensions of software quality. Let’s embrace these challenges as a chance to grow and expand our skills in field of software testing. From d3de04c08024e96745bdc3354c0c4edb05b5004d Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Wed, 11 Sep 2024 08:59:14 +0100 Subject: [PATCH 19/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md Implemented Review comments from Matthew Griffin. --- ...aditional-Testing-to-Model-Evaluation-I.md | 25 +++++++++---------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index ef3593cdb6..d3a0872c72 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -2,9 +2,6 @@ title: Evolving with AI from Traditional Testing to Model Evaluation I date: 2024-09-01 10:00:00 Z categories: -- Machine Learning -- Testing -- Model Evaluation - Artificial Intelligence tags: - Machine Learning @@ -17,16 +14,16 @@ author: snandal image: "/uploads/Evolving%20with%20AI%20From%20Traditional%20Testing%20to%20Model%20Evaluation%20I.png" --- -Machine learning (ML) is no longer just a concept for the future, it is now a vital part of many everyday applications. From personalised recommendations on streaming services to detecting fraud in banking and from self-driving cars to advancements in healthcare, ML is changing the way industries work. +Machine learning (ML) is no longer just a concept for the future, it is now a vital part of many everyday applications. From personalised recommendations on streaming services and self-driving cars to detecting fraud in banking and advancements in healthcare, ML is changing the way industries work. For a test engineer, the increasing use of ML brings new challenges and opportunities. As ML technology advances, the role of a test engineer also changes. Unlike traditional software, which gives consistent and predictable results, ML models work with probabilities and patterns, making testing and evaluation more complex. -In this blog we will try to look at how ML has developed, how it might affect our job as test engineers and the important strategies, considerations and skills needed to effectively evaluate and test ML models. -We will also try to draw a roadmap to help us grow professionally in this area. +In this blog, we will look at how ML has developed, how it might affect our job as test engineers and the important strategies, considerations and skills needed to effectively evaluate and test ML models. +We will also draw a roadmap to help us grow professionally in this area. ## How Machine Learning Has Evolved -Machine learning has significantly advanced since its early days. Initially, ML models were fairly simple, handling tasks like basic predictions or classifications. However, with improvements in computing power, access to large amounts of data and more complex algorithms, ML has grown into a powerful tool capable of handling sophisticated tasks like natural language processing and images recognition. +Machine learning has significantly advanced since its early days. Initially, ML models were fairly simple, handling tasks like basic predictions or classifications. However, with improvements in computing power, access to large amounts of data and more complex algorithms, ML has grown into a powerful tool capable of handling sophisticated tasks like natural language processing and image recognition. This progress has made ML widespread across industries, leading businesses to increasingly rely on data-driven decisions. As companies embrace ML, the need for thorough and reliable testing of these models has become more important, which directly affects the role of test engineers. @@ -36,7 +33,7 @@ At their core, ML models learn from data. They are trained on large datasets to Traditionally, a test engineer's job involved testing systems where a given input always produced the same output, making results predictable. However, with ML, this approach changes. ML models are probabilistic, meaning their outputs can vary depending on the training data and some built-in randomness. This shift has significantly changed how software testing is done. -We now need to do more than just check if a system works as expected. We must also evaluate how well a model performs, whether it's reliable and fair. This requires combining traditional testing skills with additional knowledge specific to ML. +We now need to do more than just check whether a system works as expected. We must also evaluate how well a model performs, whether it's reliable and fair. This requires combining traditional testing skills with additional knowledge specific to ML. ## Looking at the unique challenges that testing ML models brings @@ -46,7 +43,7 @@ We now need to do more than just check if a system works as expected. We must al **Performance Metrics:** Traditional testing metrics like pass/fail don’t apply directly to ML models. Instead, we use statistical metrics like accuracy, precision, recall, F1 score and AUC-ROC to evaluate model performance. Understanding and choosing the right metrics is key to effective testing. -**Model Degradation over time:** As new data becomes available; an ML model’s performance can degrade causing a problem known as model drift. This means continuous monitoring and retraining are equally important, adding complexity to the testing process. +**Model Degradation Over Time:** As new data becomes available, an ML model’s performance can degrade causing a problem known as model drift. This means continuous monitoring and retraining are equally important, adding complexity to the testing process. ## Model Evaluation Vs. Model Testing @@ -82,7 +79,7 @@ To effectively address the challenges that ML brings, test engineers can benefit Consider a hospital implementing an ML model to predict the likelihood of patients being readmitted within 30 days of discharge. -**Defining Objectives:** The goal here is to identify most high-risk patients so that the hospital can take proactive measures to reduce readmission rates and improve patient care. +**Defining Objectives:** The goal here is to identify the most high-risk patients so that the hospital can take proactive measures to reduce readmission rates and improve patient care. **Data Validation:** Further start by validating the dataset, ensuring that it includes a balanced representation of various patient, medical history, treatment plans etc. Use cross-validation to train the model and select the most effective algorithm for predicting readmissions. @@ -116,7 +113,7 @@ To effectively perform model evaluation, a test engineer would need a combinatio **4. Data Analysis and Validation** * Skills in data cleaning, handling missing data, outliers, data normalisation and feature engineering. -* Ability to validate datasets for biases, errors and representativeness, ensuring that the training data is of high quality and relevant to the model’s objectives. +* Ability to validate datasets for biases, errors and representativeness, ensuring that the training data is of high quality and relevant to the model objectives. * Proficiency in using tools like Python’s Pandas, Matplotlib and Seaborn for EDA to understand the dataset before feeding it into an ML model. **5. Familiarity with common ML Tools and Platforms** @@ -147,9 +144,11 @@ Here’s a step-by-step roadmap for a test engineer to develop ML model testing By following this roadmap, test engineers can successfully transition into ML model testing, ensuring they stay relevant in an evolving field. -**In Conclusion**, evaluating and testing ML models requires a mix of traditional testing skills and an understanding of statistical methods. With the right testing strategies and by following a structured approach to testing, incorporating data validation, metric analysis, stress testing and continuous monitoring, we can ensure that our ML model provides real value and meets business objectives. +## In Conclusion + +Evaluating and testing ML models requires a mix of traditional testing skills and an understanding of statistical methods. With the right testing strategies and by following a structured approach to testing, incorporating data validation, metric analysis, stress testing and continuous monitoring, we can ensure that our ML model provides real value and meets business objectives. Additionally, staying informed about the latest trends and tools can help handle the challenges of ML testing effectively and make a significant impact on your organisation’s success. -Machine learning is complex but also an exciting opportunity to explore new dimensions of software quality. Let’s embrace these challenges as a chance to grow and expand our skills in field of software testing. +Machine learning is complex but also an exciting opportunity to explore new dimensions of software quality. Let’s embrace these challenges as a chance to grow and expand our skills in the field of software testing. *Hoping to publish my next blog soon, where I will dive deeper into evaluation metrics and how to choose the right ones for assessing machine learning models effectively.* From 373f75ce8444f54cc676f558d918c555712bd817 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Fri, 13 Sep 2024 11:06:56 +0100 Subject: [PATCH 20/54] Update 2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...ng-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md index d3a0872c72..cbc1a032be 100644 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md @@ -1,6 +1,6 @@ --- title: Evolving with AI from Traditional Testing to Model Evaluation I -date: 2024-09-01 10:00:00 Z +date: 2024-09-13 10:00:00 Z categories: - Artificial Intelligence tags: From 66574ddbc3d0857980ca7073d3430c9a97f11392 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Mon, 28 Oct 2024 10:26:14 +0000 Subject: [PATCH 21/54] Create 2024-10-28-Testing-GenerativeAI-Chatbots.md Initial Draft --- ...024-10-28-Testing-GenerativeAI-Chatbots.md | 130 ++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md new file mode 100644 index 0000000000..33e858e2da --- /dev/null +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -0,0 +1,130 @@ +--- +title: Testing GenerativeAI Chatbots +date: 2024-10-28 10:00:00 Z +categories: +- Artificial Intelligence +tags: +- Testing +summary: In the fast-changing world of digital technology, GenAI systems have emerged as revolutionary tools for businesses and individuals. As these intelligent systems become a bigger part of our lives, it is important to understand their functionality and to ensure their effectiveness.  +In this blog post, we will discuss how we can make sure that our Gen AI-powered systems are not only working properly but are also efficient and easy to use.  +author: snandal +image: "/uploads/Evolving%20with%20AI%20From%20Traditional%20Testing%20to%20Model%20Evaluation%20I.png" +--- + +## Understanding Gen AI systems + +Generative AI (GenAI) models are designed to create content, recognise patterns and make predictions. In addition, they have an ability to improve over time as they are exposed to more data. + +GenAI chat models, such as GPT-4 by OpenAI, can generate human-like text and other forms of content autonomously. They can produce outputs that are remarkably like human-created content, making them useful for a wide range of applications. They leverage artificial intelligence, machine learning, and natural language processing to understand, interpret and respond to user queries. + +## Two key aspects of these models are + +**Functionality:** Gen AI models are programmed and trained to perform a wide range of tasks, from answering FAQs to executing complex commands. These models possess natural language understanding, contextual awareness and can understand and process diverse inputs, providing relevant and accurate responses. + +**Learning Ability:** Through machine learning algorithms, these models have ability to continuously learn from interactions and to improve their responses over time. By analysing vast amounts of data and feedback from users, the AI can refine its understanding and prediction capabilities and hence very adaptable to new patterns, trends, and user behaviours, making it more accurate, reliable, and effective in dynamic environments. + +This technology is revolutionising multiple industries like Healthcare, Entertainment, Marketing, Finance by enhancing creativity, productivity, and efficiency. And hence there is the need for its effective testing to ensure that these models meet the desired standards of performance and user engagement. + +## What all does Gen AI testing involve + +**Functionality Testing:** Verifying that all features of the Gen AI model under test work as intended. This includes validating input handling, output accuracy, and error handling capabilities to guarantee that the AI behaves as expected in all scenarios. + +**Usability Testing:** Assessing how easy and intuitive it is for users to interact with the system. This includes evaluating the user interface, navigation and overall user experience to ensure that the GenAI based application is user-friendly. + +**Performance Testing:** Ensuring that a substantial number of queries/inputs can be handled without degradation in performance. This type of testing measures the responsiveness, stability, and speed of the Gen AI model under various loads. + +**Security Testing:** Checking for vulnerabilities that might compromise user data or privacy. This involves a comprehensive evaluation of the AI model's security measures to protect against unauthorised access, data breaches, and other potential threats. + +**Compatibility Testing:** Making sure the system functions seamlessly across different platforms and devices. This involves testing the Gen AI model on various operating systems, browsers, and hardware configurations to ensure consistent performance and functionality. + +## Challenges in AI Chatbot Testing + +Testing AI chatbots presents several unique challenges and limitations due to the intricacies of AI algorithms and the complexity of natural language processing. + +**Non-Deterministic Responses:** AI chatbots exhibit non-deterministic behaviour, meaning that they can generate completely different responses to the exact same input, under exact same conditions. These variations in responses are not necessarily errors, but mere deviations from previous behaviour, making it harder to define what constitutes a "correct" and “expected” response. +Hence AI testing require sophisticated techniques such as probabilistic testing, where multiple iterations of the same test case are executed to assess the overall performance of the model. + +**Extensive Test Data Requirements:** AI chatbots require substantial training data to accurately learn and interpret user inputs. For specialised domains, domain-specific test data is necessary. Collecting a diverse and representative dataset can be particularly challenging, especially in niche areas. Insufficient data can lead to reduced accuracy and reliability in the chatbot's responses, especially in context-specific scenarios. + +**Managing Complex Conversations:** AI chatbots must maintain context and recognise references to previous messages to effectively manage complex conversations. Natural language is inherently ambiguous and includes variations such as slang and regional dialects. Testing these scenarios necessitates equally complex scripting and scenario planning, which can be particularly challenging and resource intensive. + +**Lack of Standardised Evaluation Metrics:** As AI technology is new, existing evaluation metrics may not fully capture the quality of the user experience. Establishing appropriate and comprehensive evaluation criteria is an ongoing challenge. Developing standardised metrics that accurately reflect chatbot performance and user satisfaction remains a significant hurdle in AI testing. + +**Continuous learning and Evolution:** AI systems are designed to continuously learn from user interactions and evolve over time. This dynamic nature poses a significant challenge for testing, as it requires ongoing monitoring and testing to ensure consistent performance and to identify and address any unintended issues that arise from continuous learning. + +Despite these challenges; regular updates, monitoring, combined with robust testing strategies can help organisations ensure the reliability, accuracy, and user satisfaction of their AI chatbot systems. + +## How to test + +Testing Generative AI (Gen AI) models involves evaluating multiple aspects such as the quality, relevance, coherence, and ethical considerations of the outputs generated by these models. Here is a structured approach to testing Gen AI models, along with some tools and best practices + +**Define Clear Evaluation Metrics** Establish specific metrics to evaluate the performance of the Gen AI model. Common metrics include: +Accuracy: How correct the outputs are based on expected results. + +Relevance: The relevance of the generated content to the input prompt. + +Coherence: Logical consistency and fluency of the generated text. + +Diversity: The variety of responses generated from similar prompts. + +Ethical Considerations: Checking for biases, harmful content, or inappropriate responses. + +**Simulate Real User Behaviour** Use diverse and realistic test data to emulate actual user interactions. This involves: + +Testing with data that mimics real-world scenarios to uncover practical issues. + +Conducting tests with actual user interactions to gather authentic feedback. + +Establishing feedback loops to continuously collect and act on user feedback for ongoing improvement. + +**Focus on Conversation Flow** Ensure the chatbot can maintain context and handle conversations naturally. This includes: + +Testing the chatbot’s ability to understand and manage references to previous interactions. + +Evaluating its performance in handling ambiguous language and variations, such as slang and dialects. + +**Monitor Performance** Continuously track performance metrics to ensure the chatbot remains responsive and reliable. Key actions include: + +Regularly reviewing accuracy, relevance, and coherence metrics. + +Using monitoring tools to track real-time performance and detect any issues promptly. + +**Incorporate User Feedback**  User feedback is invaluable for refining the chatbot. Best practices include: + +Using feedback to update and enhance test cases, ensuring they remain relevant and comprehensive. + +Iteratively improving the chatbot’s accuracy and relevance based on real user interactions and suggestions. + +By following this structured approach, organisations can effectively test and refine their Generative AI models, ensuring they deliver high-quality, relevant, and ethically sound outputs. + +## Manual Testing + +Manual testing plays a vital role in the development and refinement of Gen AI-based models. By incorporating human judgement and real-world variability, manual testing helps ensure that AI systems are not only functionally accurate but also contextually appropriate, robust, emotionally intelligent, culturally sensitive, and ethically sound. + +**Human Evaluation:** +Given the unpredictable nature of AI, involving human testers to manually evaluate the responses based on predefined criteria can be beneficial. This can help assess aspects that are difficult to quantify, such as naturalness and appropriateness. Human evaluators can also provide nuanced feedback on the tone, style, and clarity of the responses, ensuring that the AI aligns with the desired user experience and communication standards. + +**Simulate Real User Behaviour:** +Human evaluation can easily mimic actual user interactions ensuring the chatbot maintains context and handles conversations naturally. Testers can engage in varied and spontaneous dialogues that reflect real-world use cases, uncovering issues that automated tests might miss. This helps ensure the AI's responses are not only accurate but also contextually relevant and user-friendly. + +**Test Robustness:** +Human evaluation can also be useful in ensuring that the chatbot can handle a variety of inputs and situations gracefully. Subjection of common typos, incomplete or fragmented sentences dynamically will help evaluate if the chatbot can still understand the user's intent. Also, scenarios where the conversation is interrupted and resumed later will ensure the chatbot is able to maintain or recover the context of the conversation. + +## Automated Testing + +While manual testing effectively applies human judgement and real-world variability when it comes to testing AI systems, automated testing on the other hand provides consistent and repeatable test results, ensuring that the chatbot performs reliably under various conditions. Automated tests can be run frequently and at scale, allowing quick identification of issues. + +**Efficiency and speed:** Automated testing significantly reduces the time required to test AI chatbots. Scripts can run multiple tests simultaneously, providing quick feedback and enabling rapid iterations. + +**Consistency:** Automated tests ensure consistency in testing, eliminating the variability and potential biases introduced by human testers. This consistency is critical for identifying regressions and maintaining the quality of chatbot interactions. + +**Scalability:** Automated testing can handle large volumes of test cases, which is essential for testing chatbots that need to support numerous scenarios and user interactions. This scalability is challenging to achieve with manual testing. + +Below are some notable options available that can be used to set up automated testing for AI chatbot outputs, each offering unique features to support comprehensive testing. Please note that these tools are relatively new and still being explored for their full potential and hence are expected to evolve as the field of AI chatbot testing evolves. + +## Promptfoo +## Botium +## Testmybot + +*At Scott Logic, we have conducted several projects exploring the effectiveness of automated testing for AI chatbots. One of the tools we have used extensively is PromptFoo. This tool has provided us with valuable insights and enhanced our testing processes, ensuring our chatbots deliver high-quality interactions. +Stay tuned for our upcoming blog, where we will provide an in-depth review of PromptFoo, including its features, benefits, and how it can be leveraged to improve AI chatbot testing.* From f63d625438de0c657d709befc1cba80f615995f4 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Mon, 28 Oct 2024 10:32:49 +0000 Subject: [PATCH 22/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 33e858e2da..b4ef3cbdb0 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -5,6 +5,7 @@ categories: - Artificial Intelligence tags: - Testing +- Artificial Intelligence summary: In the fast-changing world of digital technology, GenAI systems have emerged as revolutionary tools for businesses and individuals. As these intelligent systems become a bigger part of our lives, it is important to understand their functionality and to ensure their effectiveness.  In this blog post, we will discuss how we can make sure that our Gen AI-powered systems are not only working properly but are also efficient and easy to use.  author: snandal From 923a6ec3cd0ca60441c738f8a880a23b63a6e8e0 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Mon, 28 Oct 2024 10:34:22 +0000 Subject: [PATCH 23/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index b4ef3cbdb0..159302ba9c 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -6,8 +6,7 @@ categories: tags: - Testing - Artificial Intelligence -summary: In the fast-changing world of digital technology, GenAI systems have emerged as revolutionary tools for businesses and individuals. As these intelligent systems become a bigger part of our lives, it is important to understand their functionality and to ensure their effectiveness.  -In this blog post, we will discuss how we can make sure that our Gen AI-powered systems are not only working properly but are also efficient and easy to use.  +summary: In the fast-changing world of digital technology, GenAI systems have emerged as revolutionary tools for businesses and individuals. As these intelligent systems become a bigger part of our lives, it is important to understand their functionality and to ensure their effectiveness. In this blog post, we will discuss how we can make sure that our Gen AI-powered systems are not only working properly but are also efficient and easy to use.  author: snandal image: "/uploads/Evolving%20with%20AI%20From%20Traditional%20Testing%20to%20Model%20Evaluation%20I.png" --- From 9ce25c5b33782712dc5b67e58bc3e6c1e66ba267 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Mon, 28 Oct 2024 11:47:28 +0000 Subject: [PATCH 24/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- ...024-10-28-Testing-GenerativeAI-Chatbots.md | 31 ++++++++++++------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 159302ba9c..cee79b29ed 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -15,27 +15,36 @@ image: "/uploads/Evolving%20with%20AI%20From%20Traditional%20Testing%20to%20Mode Generative AI (GenAI) models are designed to create content, recognise patterns and make predictions. In addition, they have an ability to improve over time as they are exposed to more data. -GenAI chat models, such as GPT-4 by OpenAI, can generate human-like text and other forms of content autonomously. They can produce outputs that are remarkably like human-created content, making them useful for a wide range of applications. They leverage artificial intelligence, machine learning, and natural language processing to understand, interpret and respond to user queries. +GenAI chat models, such as GPT-4 by OpenAI, can generate human-like text and other forms of content autonomously. They can produce outputs that are remarkably like human-created content, making them useful for a wide range of applications. They leverage artificial intelligence, machine learning and natural language processing to understand, interpret and respond to user queries. -## Two key aspects of these models are -**Functionality:** Gen AI models are programmed and trained to perform a wide range of tasks, from answering FAQs to executing complex commands. These models possess natural language understanding, contextual awareness and can understand and process diverse inputs, providing relevant and accurate responses. +## Two key aspects of these models are: -**Learning Ability:** Through machine learning algorithms, these models have ability to continuously learn from interactions and to improve their responses over time. By analysing vast amounts of data and feedback from users, the AI can refine its understanding and prediction capabilities and hence very adaptable to new patterns, trends, and user behaviours, making it more accurate, reliable, and effective in dynamic environments. +**Functionality:** GenAI models are programmed and trained to perform a wide range of tasks, from answering FAQs to executing complex commands. These models possess natural language understanding, contextual awareness and can understand and process diverse inputs, providing relevant and accurate responses. -This technology is revolutionising multiple industries like Healthcare, Entertainment, Marketing, Finance by enhancing creativity, productivity, and efficiency. And hence there is the need for its effective testing to ensure that these models meet the desired standards of performance and user engagement. +**Learning Ability:** Through machine learning algorithms, these models have ability to continuously learn from interactions and to improve their responses over time. By analysing vast amounts of data and feedback from users, AI can refine its understanding and prediction capabilities and hence very adaptable to new patterns, trends and user behaviours; making it more accurate, reliable and effective in dynamic environments. -## What all does Gen AI testing involve -**Functionality Testing:** Verifying that all features of the Gen AI model under test work as intended. This includes validating input handling, output accuracy, and error handling capabilities to guarantee that the AI behaves as expected in all scenarios. +This technology is revolutionising multiple industries like Healthcare, Entertainment, Marketing, Finance by enhancing creativity and efficiency. And hence there is the need for its effective testing to ensure that these models meet the desired standards of performance and user engagement. -**Usability Testing:** Assessing how easy and intuitive it is for users to interact with the system. This includes evaluating the user interface, navigation and overall user experience to ensure that the GenAI based application is user-friendly. -**Performance Testing:** Ensuring that a substantial number of queries/inputs can be handled without degradation in performance. This type of testing measures the responsiveness, stability, and speed of the Gen AI model under various loads. +## What all does GenAI testing involve -**Security Testing:** Checking for vulnerabilities that might compromise user data or privacy. This involves a comprehensive evaluation of the AI model's security measures to protect against unauthorised access, data breaches, and other potential threats. +**1. Functionality Testing: Verifying that all features of the Gen AI model under test work as intended.** +This includes validating input handling, output accuracy and error handling capabilities to ensure that the model behaves as expected in all scenarios. + +**2. Usability Testing: Assessing how easy and intuitive it is for users to interact with the model.** +This includes evaluating the user interface, navigation and overall user experience to ensure that the model is user-friendly. + +**3. Performance Testing: Ensuring that a substantial number of queries/inputs/prompts can be handled without degradation in performance.** +This includes measuring the responsiveness, stability and speed of the model under various loads. + +**4. Security Testing: Checking for vulnerabilities that might compromise user data or privacy.** +This involves a comprehensive evaluation of the model's security measures to protect against unauthorised access, data breaches and other potential threats. + +**5. Compatibility Testing: Making sure that the model functions seamlessly across different platforms and devices.** +This involves testing the model on various devices, operating systems, browsers, hardware configurations to ensure consistent performance and functionality. -**Compatibility Testing:** Making sure the system functions seamlessly across different platforms and devices. This involves testing the Gen AI model on various operating systems, browsers, and hardware configurations to ensure consistent performance and functionality. ## Challenges in AI Chatbot Testing From cbda09d1a1b07b56884b0164994ddea128fdba9e Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Mon, 28 Oct 2024 12:23:28 +0000 Subject: [PATCH 25/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- ...024-10-28-Testing-GenerativeAI-Chatbots.md | 114 +++++++++--------- 1 file changed, 56 insertions(+), 58 deletions(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index cee79b29ed..0f1dcea842 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -11,119 +11,114 @@ author: snandal image: "/uploads/Evolving%20with%20AI%20From%20Traditional%20Testing%20to%20Model%20Evaluation%20I.png" --- -## Understanding Gen AI systems +## Understanding GenAI models Generative AI (GenAI) models are designed to create content, recognise patterns and make predictions. In addition, they have an ability to improve over time as they are exposed to more data. -GenAI chat models, such as GPT-4 by OpenAI, can generate human-like text and other forms of content autonomously. They can produce outputs that are remarkably like human-created content, making them useful for a wide range of applications. They leverage artificial intelligence, machine learning and natural language processing to understand, interpret and respond to user queries. +GenAI chatbot models, such as GPT-4 by OpenAI, can generate human-like text and other forms of content autonomously. They can produce outputs that are remarkably like human-created content, making them useful for a wide range of applications. They leverage artificial intelligence, machine learning and natural language processing to understand, interpret and respond to user queries. ## Two key aspects of these models are: -**Functionality:** GenAI models are programmed and trained to perform a wide range of tasks, from answering FAQs to executing complex commands. These models possess natural language understanding, contextual awareness and can understand and process diverse inputs, providing relevant and accurate responses. +**1. Functionality:** GenAI chatbot models are programmed and trained to perform a wide range of tasks, from answering FAQs to executing complex commands. These models possess natural language understanding, contextual awareness and can understand and process diverse inputs, providing relevant and accurate responses. -**Learning Ability:** Through machine learning algorithms, these models have ability to continuously learn from interactions and to improve their responses over time. By analysing vast amounts of data and feedback from users, AI can refine its understanding and prediction capabilities and hence very adaptable to new patterns, trends and user behaviours; making it more accurate, reliable and effective in dynamic environments. +**2. Learning Ability:** Through machine learning algorithms, these models have ability to continuously learn from interactions and to improve their responses over time. By analysing vast amounts of data and feedback from users, AI can refine its understanding and prediction capabilities and hence very adaptable to new patterns, trends and user behaviours; making it more accurate, reliable and effective in dynamic environments. This technology is revolutionising multiple industries like Healthcare, Entertainment, Marketing, Finance by enhancing creativity and efficiency. And hence there is the need for its effective testing to ensure that these models meet the desired standards of performance and user engagement. -## What all does GenAI testing involve +## What all does GenAI chatbot testing involve -**1. Functionality Testing: Verifying that all features of the Gen AI model under test work as intended.** +- **Functionality Testing: Verifying that all features of the Gen AI chatbot model under test work as intended.** This includes validating input handling, output accuracy and error handling capabilities to ensure that the model behaves as expected in all scenarios. -**2. Usability Testing: Assessing how easy and intuitive it is for users to interact with the model.** +- **Usability Testing: Assessing how easy and intuitive it is for users to interact with the model.** This includes evaluating the user interface, navigation and overall user experience to ensure that the model is user-friendly. -**3. Performance Testing: Ensuring that a substantial number of queries/inputs/prompts can be handled without degradation in performance.** +- **Performance Testing: Ensuring that a substantial number of queries/inputs/prompts can be handled without degradation in performance.** This includes measuring the responsiveness, stability and speed of the model under various loads. -**4. Security Testing: Checking for vulnerabilities that might compromise user data or privacy.** +- **Security Testing: Checking for vulnerabilities that might compromise user data or privacy.** This involves a comprehensive evaluation of the model's security measures to protect against unauthorised access, data breaches and other potential threats. -**5. Compatibility Testing: Making sure that the model functions seamlessly across different platforms and devices.** +- **Compatibility Testing: Making sure that the model functions seamlessly across different platforms and devices.** This involves testing the model on various devices, operating systems, browsers, hardware configurations to ensure consistent performance and functionality. -## Challenges in AI Chatbot Testing +## Challenges in GenAI chatbot testing -Testing AI chatbots presents several unique challenges and limitations due to the intricacies of AI algorithms and the complexity of natural language processing. +Testing GenAI chatbots presents several unique challenges and limitations due to the intricacies of AI algorithms and the complexity of natural language processing. Below is an overview of some of the key challenges faced in testing GenAI chatbot models: -**Non-Deterministic Responses:** AI chatbots exhibit non-deterministic behaviour, meaning that they can generate completely different responses to the exact same input, under exact same conditions. These variations in responses are not necessarily errors, but mere deviations from previous behaviour, making it harder to define what constitutes a "correct" and “expected” response. -Hence AI testing require sophisticated techniques such as probabilistic testing, where multiple iterations of the same test case are executed to assess the overall performance of the model. +**1. Non-Deterministic Responses:** GenAI chatbot models exhibit non-deterministic behaviour, meaning that they can generate completely different responses to the exact same input, under exact same conditions. These variations in responses are not necessarily errors, but mere deviations from previous behaviour, making it harder to define what constitutes a "correct" and “expected” response. +Hence GenAI model testing require sophisticated techniques such as probabilistic testing, where multiple iterations of the same test case are executed to assess the overall performance of the model. -**Extensive Test Data Requirements:** AI chatbots require substantial training data to accurately learn and interpret user inputs. For specialised domains, domain-specific test data is necessary. Collecting a diverse and representative dataset can be particularly challenging, especially in niche areas. Insufficient data can lead to reduced accuracy and reliability in the chatbot's responses, especially in context-specific scenarios. +**2. Extensive Test Data Requirements:** GenAI chatbot models require substantial training data to accurately learn and interpret user inputs. For specialised domains, domain-specific test data is necessary. Collecting a diverse and representative dataset can be particularly challenging, especially in niche areas. Insufficient data can lead to reduced accuracy and reliability in the chatbot's responses, especially in context-specific scenarios. -**Managing Complex Conversations:** AI chatbots must maintain context and recognise references to previous messages to effectively manage complex conversations. Natural language is inherently ambiguous and includes variations such as slang and regional dialects. Testing these scenarios necessitates equally complex scripting and scenario planning, which can be particularly challenging and resource intensive. +**3. Managing Complex Conversations:** GenAI chatbots must maintain context and recognise references to previous messages to effectively manage complex conversations. Also, natural language is inherently ambiguous and includes variations such as slang and regional dialects. Testing these scenarios necessitates equally complex scenario planning, which can be particularly challenging and intensive. -**Lack of Standardised Evaluation Metrics:** As AI technology is new, existing evaluation metrics may not fully capture the quality of the user experience. Establishing appropriate and comprehensive evaluation criteria is an ongoing challenge. Developing standardised metrics that accurately reflect chatbot performance and user satisfaction remains a significant hurdle in AI testing. +**4. Lack of Standardised Evaluation Metrics:** As AI technology is quite new, existing evaluation metrics may not fully capture the quality of the user experience. Establishing appropriate and comprehensive evaluation criteria is an ongoing challenge. Developing standardised metrics, that would accurately reflect the chatbot's performance and user satisfaction, remains a significant hurdle in GenAI model testing. -**Continuous learning and Evolution:** AI systems are designed to continuously learn from user interactions and evolve over time. This dynamic nature poses a significant challenge for testing, as it requires ongoing monitoring and testing to ensure consistent performance and to identify and address any unintended issues that arise from continuous learning. +**5. Continuous learning and Evolution:** AI systems are designed to continuously learn from user interactions and evolve over time. This dynamic nature poses a significant challenge for testing, as it requires ongoing monitoring and testing of these models, to ensure consistent performance; and to identify and address any unintended issues that arise from continuous learning. -Despite these challenges; regular updates, monitoring, combined with robust testing strategies can help organisations ensure the reliability, accuracy, and user satisfaction of their AI chatbot systems. +Despite all these challenges, regular monitoring, combined with robust testing strategies can help organisations ensure the accuracy, reliability and user satisfaction of their GenAI chatbot model. -## How to test - -Testing Generative AI (Gen AI) models involves evaluating multiple aspects such as the quality, relevance, coherence, and ethical considerations of the outputs generated by these models. Here is a structured approach to testing Gen AI models, along with some tools and best practices - -**Define Clear Evaluation Metrics** Establish specific metrics to evaluate the performance of the Gen AI model. Common metrics include: -Accuracy: How correct the outputs are based on expected results. -Relevance: The relevance of the generated content to the input prompt. - -Coherence: Logical consistency and fluency of the generated text. +## How to test -Diversity: The variety of responses generated from similar prompts. +Testing GenAI chatbot models involves evaluating multiple aspects, such as the quality, relevance, coherence and ethical considerations of the outputs generated by these models. Here is a structured approach to testing GenAI models, along with some tools and best practices: -Ethical Considerations: Checking for biases, harmful content, or inappropriate responses. +**Define Clear Evaluation Metrics** Establish specific metrics to evaluate the performance of the model. Common metrics include: +- Accuracy: How correct the outputs are based on expected results. +- Relevance: The relevance of the generated content to the input prompt. +- Coherence: Logical consistency and fluency of the generated text. +- Diversity: The variety of responses generated from similar prompts. +- Ethical Considerations: Checking for biases, harmful or inappropriate responses. **Simulate Real User Behaviour** Use diverse and realistic test data to emulate actual user interactions. This involves: -Testing with data that mimics real-world scenarios to uncover practical issues. - -Conducting tests with actual user interactions to gather authentic feedback. - -Establishing feedback loops to continuously collect and act on user feedback for ongoing improvement. +- Testing with data that mimics real-world scenarios to uncover practical issues. +- Conducting tests with actual user interactions to gather authentic feedback. +- Establishing feedback loops to continuously collect and act on user feedback for ongoing improvement. **Focus on Conversation Flow** Ensure the chatbot can maintain context and handle conversations naturally. This includes: -Testing the chatbot’s ability to understand and manage references to previous interactions. - -Evaluating its performance in handling ambiguous language and variations, such as slang and dialects. - -**Monitor Performance** Continuously track performance metrics to ensure the chatbot remains responsive and reliable. Key actions include: +- Testing the chatbot’s ability to understand and manage references to previous interactions. +- Evaluating its performance in handling ambiguous language and variations, such as slang and dialects. -Regularly reviewing accuracy, relevance, and coherence metrics. +**Monitor Performance** Continuously track performance metrics to ensure that the chatbot remains responsive and reliable. Key actions include: -Using monitoring tools to track real-time performance and detect any issues promptly. +- Regularly reviewing accuracy, relevance and coherence metrics. +- Using monitoring tools to track real-time performance and detect any issues promptly. **Incorporate User Feedback**  User feedback is invaluable for refining the chatbot. Best practices include: -Using feedback to update and enhance test cases, ensuring they remain relevant and comprehensive. +- Using feedback to update and enhance test cases, ensuring they remain relevant and comprehensive. +- Iteratively improving the chatbot’s accuracy and relevance based on real user interactions and suggestions. -Iteratively improving the chatbot’s accuracy and relevance based on real user interactions and suggestions. +By following this structured approach, organisations can effectively test and refine their GenAI chatbot models, ensuring they deliver high-quality, relevant and ethically sound outputs. -By following this structured approach, organisations can effectively test and refine their Generative AI models, ensuring they deliver high-quality, relevant, and ethically sound outputs. -## Manual Testing +## Test Approaches -Manual testing plays a vital role in the development and refinement of Gen AI-based models. By incorporating human judgement and real-world variability, manual testing helps ensure that AI systems are not only functionally accurate but also contextually appropriate, robust, emotionally intelligent, culturally sensitive, and ethically sound. +### Manual Testing + +Manual testing plays a vital role in the development and refinement of GenAI chatbot models. By incorporating human judgement and real-world variability, manual testing helps ensure that AI systems are not only functionally accurate but also contextually appropriate, emotionally intelligent, culturally sensitive and ethically sound. **Human Evaluation:** -Given the unpredictable nature of AI, involving human testers to manually evaluate the responses based on predefined criteria can be beneficial. This can help assess aspects that are difficult to quantify, such as naturalness and appropriateness. Human evaluators can also provide nuanced feedback on the tone, style, and clarity of the responses, ensuring that the AI aligns with the desired user experience and communication standards. +Given the unpredictable nature of AI, involving human testers to manually evaluate the responses based on predefined criteria can be beneficial. This can help assess aspects that are difficult to quantify, such as naturalness and appropriateness. Human evaluators can also provide nuanced feedback on the tone and style of the responses, ensuring that the chatbot model aligns with the desired user experience and communication standards. **Simulate Real User Behaviour:** -Human evaluation can easily mimic actual user interactions ensuring the chatbot maintains context and handles conversations naturally. Testers can engage in varied and spontaneous dialogues that reflect real-world use cases, uncovering issues that automated tests might miss. This helps ensure the AI's responses are not only accurate but also contextually relevant and user-friendly. +Human evaluation can easily mimic actual user interactions ensuring the chatbot model maintains context and handles conversations naturally. Testers can engage in varied and spontaneous dialogues that reflect real-world use cases, uncovering issues that automated tests might miss. This helps ensure the model's responses are not only accurate but also contextually relevant and user-friendly. **Test Robustness:** -Human evaluation can also be useful in ensuring that the chatbot can handle a variety of inputs and situations gracefully. Subjection of common typos, incomplete or fragmented sentences dynamically will help evaluate if the chatbot can still understand the user's intent. Also, scenarios where the conversation is interrupted and resumed later will ensure the chatbot is able to maintain or recover the context of the conversation. +Human evaluation can also be useful in ensuring that the chatbot can handle a variety of inputs and situations gracefully. Subjection of common typos, incomplete or fragmented sentences dynamically will help evaluate if the chatbot can still understand the user's intent. Also, scenarios where the conversation is interrupted and resumed later will ensure the chatbot is able to maintain and recover the context of the conversation. -## Automated Testing +### Automated Testing -While manual testing effectively applies human judgement and real-world variability when it comes to testing AI systems, automated testing on the other hand provides consistent and repeatable test results, ensuring that the chatbot performs reliably under various conditions. Automated tests can be run frequently and at scale, allowing quick identification of issues. +While manual testing effectively applies human judgement and real-world variability, when it comes to testing GenAI models, automated testing on the other hand provides consistent and repeatable test results. This is useful in ensuring that the chatbot performs reliably under various conditions. Automated tests can be run frequently and at scale, allowing quick identification of issues. -**Efficiency and speed:** Automated testing significantly reduces the time required to test AI chatbots. Scripts can run multiple tests simultaneously, providing quick feedback and enabling rapid iterations. +**Efficiency and speed:** Automated testing significantly reduces the time required to test GenAI chatbots. Scripts can run multiple tests simultaneously, providing quick feedback and enabling rapid iterations. **Consistency:** Automated tests ensure consistency in testing, eliminating the variability and potential biases introduced by human testers. This consistency is critical for identifying regressions and maintaining the quality of chatbot interactions. @@ -131,9 +126,12 @@ While manual testing effectively applies human judgement and real-world variabil Below are some notable options available that can be used to set up automated testing for AI chatbot outputs, each offering unique features to support comprehensive testing. Please note that these tools are relatively new and still being explored for their full potential and hence are expected to evolve as the field of AI chatbot testing evolves. -## Promptfoo -## Botium -## Testmybot -*At Scott Logic, we have conducted several projects exploring the effectiveness of automated testing for AI chatbots. One of the tools we have used extensively is PromptFoo. This tool has provided us with valuable insights and enhanced our testing processes, ensuring our chatbots deliver high-quality interactions. -Stay tuned for our upcoming blog, where we will provide an in-depth review of PromptFoo, including its features, benefits, and how it can be leveraged to improve AI chatbot testing.* +## Popular automated testing tools for GenAI Chatbot Testing + +### Promptfoo +### Botium +### Testmybot + +*At Scott Logic, we have conducted several projects exploring the effectiveness of automated testing for AI chatbots. One of the tools we have used extensively is **PromptFoo**. This tool has provided us with valuable insights and enhanced our testing processes, ensuring our chatbots deliver high-quality interactions. +Stay tuned for our upcoming blog, where we will provide an in-depth review of **PromptFoo**, including its features, benefits, and how it can be leveraged to improve GenAI chatbot testing.* From f0de37b6015bbc31c99474e4b112378f3f87b087 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Mon, 28 Oct 2024 12:50:57 +0000 Subject: [PATCH 26/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- ...024-10-28-Testing-GenerativeAI-Chatbots.md | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 0f1dcea842..9e8fca6c35 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -130,8 +130,70 @@ Below are some notable options available that can be used to set up automated te ## Popular automated testing tools for GenAI Chatbot Testing ### Promptfoo + +One innovative automated testing tool designed to evaluate and improve the performance of GenAI based chatbot models is **Promptfoo**. It focuses on providing comprehensive testing capabilities to ensure that the chatbot delivers accurate and reliable responses in various scenarios. + +**Key Features of Promptfoo** + +- **Open Source:** Promptfoo is an open source, node.js library designed to improve the testing and development of large language models (LLMs). + +- **Dynamic Prompt Testing:** Promptfoo excels in testing dynamic prompts and responses, ensuring that the chatbot can handle a wide range of user inputs and generate appropriate replies. + +- **Model Comparison:** Promptfoo allows you to compare the outputs of different GenAI chatbot models or versions side-by-side. This helps in selecting the best model for your chatbot and understanding how updates or changes impact performance. + +- **Automated Scenario Testing:** Promptfoo allows for the automation of complex testing scenarios, covering various conversation flows and edge cases. + +- **Real-Time Analytics and Reporting:** Promptfoo provides metrics, analytics and detailed reports on test results that offer quantitative insights into the chatbot's performance. This data-driven approach helps in identifying specific areas that need improvement and measuring the impact of changes. + +- **Integration and Scalability:** With its API integration, Promptfoo can be seamlessly integrated into the development and testing pipelines, allowing for continuous and scalable testing. This is crucial for maintaining the quality of the chatbot as it evolves and handles increasing user interactions. + +- **User-Friendly Interface:** Promptfoo offers an intuitive and user-friendly interface for managing test cases and reviewing results. + +In summary, **Promptfoo** is a powerful and versatile tool designed to enhance the automated testing of GenAI chatbot models. Its dynamic prompt testing, contextual understanding, seamless integration with development pipelines makes it an essential tool for developers who are seeking to maintain and improve the quality and performance of their conversational AI application. + ### Botium + +Another powerful and versatile framework designed specifically for testing conversational GenAI models is **Botium**. It supports a wide range of chatbot platforms and provides extensive functionality to automate and streamline their testing process. + +**Key Features of Botium** + +- **Cross-Platform Support:** Botium is compatible with numerous chatbot platforms, including Facebook Messenger, Microsoft Bot Framework, Google Dialogflow, Amazon Alexa and many more. This makes it a versatile tool for developers working with different chatbot technologies. + +- **Scripted and Automated Testing:** Botium allows for both scripted and automated testing of chatbot interactions. Test scripts can be written to cover specific scenarios and automated test cases can be run to validate chatbot performance over time. + +- **Capture and Replay:** Botium allows you to record the actual conversations and replay them to test the chatbot’s responses, making it easier to identify issues. + +- **Natural Language Understanding (NLU) Testing:** Botium provides robust features for testing the NLU components of the chatbot, ensuring that it correctly interprets and responds to various user inputs. + +- **Integrated Speech Processing for Testing Voice Apps:** Botium supports the testing of voice-enabled applications by processing and validating spoken inputs, ensuring that the chatbot understands and responds accurately. + +- **Performance and Load Testing:** To ensure that the chatbot can handle high volumes of interactions, Botium includes stress and load testing capabilities. Additionally, it can assess other non-functional aspects such as performance, reliability and usability, ensuring that the chatbot meets quality standards beyond just functional requirements. + +- **Security Testing and GDPR Testing:** Botium enables checks for vulnerabilities and ensures that the chatbot is secure against potential threats, protecting user data and maintaining trust. Also, it can verify that the chatbot complies with General Data Protection Regulation (GDPR) requirements, ensuring that user data is handled appropriately and privacy is maintained. + +- **Integration with CI/CD Pipelines:** Botium can be integrated with continuous integration and continuous deployment (CI/CD) pipelines, enabling automated testing as part of the development workflow. + +In summary, **Botium** is a comprehensive tool that provides all the necessary features for effective automated testing of GenAI chatbot outputs. Its cross-platform support, robust testing capabilities, and integration with CI/CD pipelines make it an indispensable tool for developers aiming to maintain and improve the performance of their conversational AI applications. + ### Testmybot +If you are looking for a simpler open-source tool to automate your chatbot testing, **TestMyBot** is another option that provides tools and libraries to automate the testing of conversational bots, ensuring they function correctly and deliver a consistent user experience. **TestMyBot** is built on top of **Botium**, leveraging Botium’s capabilities while providing a simplified interface for developers for basic testing needs. + +**Key Features of TestMyBot** + +- **Focused Scope:** TestMyBot focuses solely on conversation flow testing without the extensive configuration required for load, stress, or security testing. + +- **Scripted and Automated Testing:** TestMyBot allows for both scripted and automated testing of chatbot interactions. It supports comprehensive testing from user input to the chatbot’s response, covering the entire interaction flow. + +- **Framework Integration:** TestMyBot works with various chatbot frameworks, including Botpress, Microsoft Bot Framework, Rasa and more. It can test chatbots deployed on different platforms like Facebook Messenger, Slack and custom web chat interfaces. + +- **Assertion & Validation:** TestMyBot uses assertions to validate specific aspects of the chatbot's responses. + +- **Integration with CI/CD Pipelines:** TestmyBot can be integrated with continuous integration and continuous deployment (CI/CD) pipelines, enabling automated testing as part of the development workflow. + +- **Streamlined Interface:** TestmyBot is very easy to use for beginners and optimised for straightforward testing. + +**TestMyBot** is a powerful tool for automating the testing of chatbots, ensuring they deliver a reliable and consistent user experience. It simplifies basic chatbot testing, making it an excellent choice for quick setup and straightforward tests. + *At Scott Logic, we have conducted several projects exploring the effectiveness of automated testing for AI chatbots. One of the tools we have used extensively is **PromptFoo**. This tool has provided us with valuable insights and enhanced our testing processes, ensuring our chatbots deliver high-quality interactions. Stay tuned for our upcoming blog, where we will provide an in-depth review of **PromptFoo**, including its features, benefits, and how it can be leveraged to improve GenAI chatbot testing.* From 6b08e452c767d8d3b46b331284b1b9a1a66ec0bd Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Mon, 28 Oct 2024 13:54:33 +0000 Subject: [PATCH 27/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 9e8fca6c35..1264c7e7bf 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -11,7 +11,7 @@ author: snandal image: "/uploads/Evolving%20with%20AI%20From%20Traditional%20Testing%20to%20Model%20Evaluation%20I.png" --- -## Understanding GenAI models +# Understanding GenAI models Generative AI (GenAI) models are designed to create content, recognise patterns and make predictions. In addition, they have an ability to improve over time as they are exposed to more data. From 20f7b69963f2b03c1c83f65fdae5ac0bad277e4a Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 09:42:46 +0000 Subject: [PATCH 28/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 1264c7e7bf..50fdf5e433 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -1,5 +1,5 @@ --- -title: Testing GenerativeAI Chatbots +title: Testing GenerativeAI Chatbot Models date: 2024-10-28 10:00:00 Z categories: - Artificial Intelligence @@ -61,7 +61,7 @@ Hence GenAI model testing require sophisticated techniques such as probabilistic **5. Continuous learning and Evolution:** AI systems are designed to continuously learn from user interactions and evolve over time. This dynamic nature poses a significant challenge for testing, as it requires ongoing monitoring and testing of these models, to ensure consistent performance; and to identify and address any unintended issues that arise from continuous learning. -Despite all these challenges, regular monitoring, combined with robust testing strategies can help organisations ensure the accuracy, reliability and user satisfaction of their GenAI chatbot model. +Despite all these challenges, regular monitoring, combined with robust testing strategies can help ensure the accuracy, reliability and user satisfaction of their GenAI chatbot model. ## How to test @@ -129,7 +129,7 @@ Below are some notable options available that can be used to set up automated te ## Popular automated testing tools for GenAI Chatbot Testing -### Promptfoo +### [Promptfoo](https://www.promptfoo.dev/) One innovative automated testing tool designed to evaluate and improve the performance of GenAI based chatbot models is **Promptfoo**. It focuses on providing comprehensive testing capabilities to ensure that the chatbot delivers accurate and reliable responses in various scenarios. @@ -151,7 +151,7 @@ One innovative automated testing tool designed to evaluate and improve the perfo In summary, **Promptfoo** is a powerful and versatile tool designed to enhance the automated testing of GenAI chatbot models. Its dynamic prompt testing, contextual understanding, seamless integration with development pipelines makes it an essential tool for developers who are seeking to maintain and improve the quality and performance of their conversational AI application. -### Botium +### [Botium](https://cyara.com/products/botium/) Another powerful and versatile framework designed specifically for testing conversational GenAI models is **Botium**. It supports a wide range of chatbot platforms and provides extensive functionality to automate and streamline their testing process. @@ -175,7 +175,7 @@ Another powerful and versatile framework designed specifically for testing conve In summary, **Botium** is a comprehensive tool that provides all the necessary features for effective automated testing of GenAI chatbot outputs. Its cross-platform support, robust testing capabilities, and integration with CI/CD pipelines make it an indispensable tool for developers aiming to maintain and improve the performance of their conversational AI applications. -### Testmybot +### [Testmybot/Botium Bindings](https://github.com/codeforequity-at/botium-bindings) If you are looking for a simpler open-source tool to automate your chatbot testing, **TestMyBot** is another option that provides tools and libraries to automate the testing of conversational bots, ensuring they function correctly and deliver a consistent user experience. **TestMyBot** is built on top of **Botium**, leveraging Botium’s capabilities while providing a simplified interface for developers for basic testing needs. From 83e2a5fbf3fb9c4a32eb24c7e5ac742490cd6a14 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 09:51:59 +0000 Subject: [PATCH 29/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md Added links --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 50fdf5e433..64f2bb85cb 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -129,9 +129,9 @@ Below are some notable options available that can be used to set up automated te ## Popular automated testing tools for GenAI Chatbot Testing -### [Promptfoo](https://www.promptfoo.dev/) +### Promptfoo -One innovative automated testing tool designed to evaluate and improve the performance of GenAI based chatbot models is **Promptfoo**. It focuses on providing comprehensive testing capabilities to ensure that the chatbot delivers accurate and reliable responses in various scenarios. +One innovative automated testing tool designed to evaluate and improve the performance of GenAI based chatbot models is **[Promptfoo](https://www.promptfoo.dev/)**. It focuses on providing comprehensive testing capabilities to ensure that the chatbot delivers accurate and reliable responses in various scenarios. **Key Features of Promptfoo** @@ -151,9 +151,9 @@ One innovative automated testing tool designed to evaluate and improve the perfo In summary, **Promptfoo** is a powerful and versatile tool designed to enhance the automated testing of GenAI chatbot models. Its dynamic prompt testing, contextual understanding, seamless integration with development pipelines makes it an essential tool for developers who are seeking to maintain and improve the quality and performance of their conversational AI application. -### [Botium](https://cyara.com/products/botium/) +### Botium -Another powerful and versatile framework designed specifically for testing conversational GenAI models is **Botium**. It supports a wide range of chatbot platforms and provides extensive functionality to automate and streamline their testing process. +Another powerful and versatile framework designed specifically for testing conversational GenAI models is **[Botium](https://cyara.com/products/botium/)**. It supports a wide range of chatbot platforms and provides extensive functionality to automate and streamline their testing process. **Key Features of Botium** @@ -177,7 +177,7 @@ In summary, **Botium** is a comprehensive tool that provides all the necessary f ### [Testmybot/Botium Bindings](https://github.com/codeforequity-at/botium-bindings) -If you are looking for a simpler open-source tool to automate your chatbot testing, **TestMyBot** is another option that provides tools and libraries to automate the testing of conversational bots, ensuring they function correctly and deliver a consistent user experience. **TestMyBot** is built on top of **Botium**, leveraging Botium’s capabilities while providing a simplified interface for developers for basic testing needs. +If you are looking for a simpler open-source tool to automate your chatbot testing, **[TestMyBot](https://github.com/codeforequity-at/botium-bindings)** is another option that provides tools and libraries to automate the testing of conversational bots, ensuring they function correctly and deliver a consistent user experience. **[TestMyBot](https://github.com/codeforequity-at/botium-bindings)** is built on top of **[Botium](https://cyara.com/products/botium/)**, leveraging Botium’s capabilities while providing a simplified interface for developers for basic testing needs. **Key Features of TestMyBot** @@ -195,5 +195,4 @@ If you are looking for a simpler open-source tool to automate your chatbot testi **TestMyBot** is a powerful tool for automating the testing of chatbots, ensuring they deliver a reliable and consistent user experience. It simplifies basic chatbot testing, making it an excellent choice for quick setup and straightforward tests. -*At Scott Logic, we have conducted several projects exploring the effectiveness of automated testing for AI chatbots. One of the tools we have used extensively is **PromptFoo**. This tool has provided us with valuable insights and enhanced our testing processes, ensuring our chatbots deliver high-quality interactions. -Stay tuned for our upcoming blog, where we will provide an in-depth review of **PromptFoo**, including its features, benefits, and how it can be leveraged to improve GenAI chatbot testing.* +*By following a structured approach and using these tools, organisations can effectively test and refine GenAI chatbot models, ensuring high-quality outputs.* From 86885d86016d1d5ac179883f8d50618933cc4dd4 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 10:00:42 +0000 Subject: [PATCH 30/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md Added links to Promptfoo resources --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 64f2bb85cb..2222f42a8a 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -151,6 +151,11 @@ One innovative automated testing tool designed to evaluate and improve the perfo In summary, **Promptfoo** is a powerful and versatile tool designed to enhance the automated testing of GenAI chatbot models. Its dynamic prompt testing, contextual understanding, seamless integration with development pipelines makes it an essential tool for developers who are seeking to maintain and improve the quality and performance of their conversational AI application. +Below are some useful resources on the tool : +- [Promptfoo - Initial Set up](https://www.youtube.com/watch?v=VKyJYgz8IiQ) +- [How to assess an LLM with the Promptfoo](https://www.youtube.com/watch?v=0sk5rb_ErOE) +- [Using Promptfoo to COMPARE Prompts, LLMs, and Providers](https://www.youtube.com/watch?v=KhINc5XwhKs&t=753s) + ### Botium Another powerful and versatile framework designed specifically for testing conversational GenAI models is **[Botium](https://cyara.com/products/botium/)**. It supports a wide range of chatbot platforms and provides extensive functionality to automate and streamline their testing process. @@ -175,6 +180,8 @@ Another powerful and versatile framework designed specifically for testing conve In summary, **Botium** is a comprehensive tool that provides all the necessary features for effective automated testing of GenAI chatbot outputs. Its cross-platform support, robust testing capabilities, and integration with CI/CD pipelines make it an indispensable tool for developers aiming to maintain and improve the performance of their conversational AI applications. +Below are some useful resources on the tool : + ### [Testmybot/Botium Bindings](https://github.com/codeforequity-at/botium-bindings) If you are looking for a simpler open-source tool to automate your chatbot testing, **[TestMyBot](https://github.com/codeforequity-at/botium-bindings)** is another option that provides tools and libraries to automate the testing of conversational bots, ensuring they function correctly and deliver a consistent user experience. **[TestMyBot](https://github.com/codeforequity-at/botium-bindings)** is built on top of **[Botium](https://cyara.com/products/botium/)**, leveraging Botium’s capabilities while providing a simplified interface for developers for basic testing needs. @@ -195,4 +202,6 @@ If you are looking for a simpler open-source tool to automate your chatbot testi **TestMyBot** is a powerful tool for automating the testing of chatbots, ensuring they deliver a reliable and consistent user experience. It simplifies basic chatbot testing, making it an excellent choice for quick setup and straightforward tests. +Below are some useful resources on the tool : + *By following a structured approach and using these tools, organisations can effectively test and refine GenAI chatbot models, ensuring high-quality outputs.* From d8f901e177d21f830788c0b0436f5d4b39df99eb Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 10:45:16 +0000 Subject: [PATCH 31/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- ...024-10-28-Testing-GenerativeAI-Chatbots.md | 21 ------------------- 1 file changed, 21 deletions(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 2222f42a8a..7748ffe455 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -182,26 +182,5 @@ In summary, **Botium** is a comprehensive tool that provides all the necessary f Below are some useful resources on the tool : -### [Testmybot/Botium Bindings](https://github.com/codeforequity-at/botium-bindings) - -If you are looking for a simpler open-source tool to automate your chatbot testing, **[TestMyBot](https://github.com/codeforequity-at/botium-bindings)** is another option that provides tools and libraries to automate the testing of conversational bots, ensuring they function correctly and deliver a consistent user experience. **[TestMyBot](https://github.com/codeforequity-at/botium-bindings)** is built on top of **[Botium](https://cyara.com/products/botium/)**, leveraging Botium’s capabilities while providing a simplified interface for developers for basic testing needs. - -**Key Features of TestMyBot** - -- **Focused Scope:** TestMyBot focuses solely on conversation flow testing without the extensive configuration required for load, stress, or security testing. - -- **Scripted and Automated Testing:** TestMyBot allows for both scripted and automated testing of chatbot interactions. It supports comprehensive testing from user input to the chatbot’s response, covering the entire interaction flow. - -- **Framework Integration:** TestMyBot works with various chatbot frameworks, including Botpress, Microsoft Bot Framework, Rasa and more. It can test chatbots deployed on different platforms like Facebook Messenger, Slack and custom web chat interfaces. - -- **Assertion & Validation:** TestMyBot uses assertions to validate specific aspects of the chatbot's responses. - -- **Integration with CI/CD Pipelines:** TestmyBot can be integrated with continuous integration and continuous deployment (CI/CD) pipelines, enabling automated testing as part of the development workflow. - -- **Streamlined Interface:** TestmyBot is very easy to use for beginners and optimised for straightforward testing. - -**TestMyBot** is a powerful tool for automating the testing of chatbots, ensuring they deliver a reliable and consistent user experience. It simplifies basic chatbot testing, making it an excellent choice for quick setup and straightforward tests. - -Below are some useful resources on the tool : *By following a structured approach and using these tools, organisations can effectively test and refine GenAI chatbot models, ensuring high-quality outputs.* From e0d7eb72c37d9ae6b44124fb72f0736c5ad06ed0 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 10:53:44 +0000 Subject: [PATCH 32/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 7748ffe455..c482c02a4c 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -96,8 +96,6 @@ Testing GenAI chatbot models involves evaluating multiple aspects, such as the q - Using feedback to update and enhance test cases, ensuring they remain relevant and comprehensive. - Iteratively improving the chatbot’s accuracy and relevance based on real user interactions and suggestions. -By following this structured approach, organisations can effectively test and refine their GenAI chatbot models, ensuring they deliver high-quality, relevant and ethically sound outputs. - ## Test Approaches @@ -181,6 +179,6 @@ Another powerful and versatile framework designed specifically for testing conve In summary, **Botium** is a comprehensive tool that provides all the necessary features for effective automated testing of GenAI chatbot outputs. Its cross-platform support, robust testing capabilities, and integration with CI/CD pipelines make it an indispensable tool for developers aiming to maintain and improve the performance of their conversational AI applications. Below are some useful resources on the tool : - +- [Automated testing of a Google Dialogflow chatbot through Botium](https://www.youtube.com/watch?v=b8CgSkJmxCA) *By following a structured approach and using these tools, organisations can effectively test and refine GenAI chatbot models, ensuring high-quality outputs.* From d1a41d42c84eccfc13066e48afa1a6f9e41b826c Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 10:56:02 +0000 Subject: [PATCH 33/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index c482c02a4c..ef955077a4 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -28,7 +28,7 @@ GenAI chatbot models, such as GPT-4 by OpenAI, can generate human-like text and This technology is revolutionising multiple industries like Healthcare, Entertainment, Marketing, Finance by enhancing creativity and efficiency. And hence there is the need for its effective testing to ensure that these models meet the desired standards of performance and user engagement. -## What all does GenAI chatbot testing involve +## What all does GenAI chatbot testing involve? - **Functionality Testing: Verifying that all features of the Gen AI chatbot model under test work as intended.** This includes validating input handling, output accuracy and error handling capabilities to ensure that the model behaves as expected in all scenarios. @@ -64,7 +64,7 @@ Hence GenAI model testing require sophisticated techniques such as probabilistic Despite all these challenges, regular monitoring, combined with robust testing strategies can help ensure the accuracy, reliability and user satisfaction of their GenAI chatbot model. -## How to test +## How to test? Testing GenAI chatbot models involves evaluating multiple aspects, such as the quality, relevance, coherence and ethical considerations of the outputs generated by these models. Here is a structured approach to testing GenAI models, along with some tools and best practices: From ea597d32f397ee552d235eff24d66c37441d960f Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 12:00:03 +0000 Subject: [PATCH 34/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index ef955077a4..0f9ac7d1e4 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -28,6 +28,14 @@ GenAI chatbot models, such as GPT-4 by OpenAI, can generate human-like text and This technology is revolutionising multiple industries like Healthcare, Entertainment, Marketing, Finance by enhancing creativity and efficiency. And hence there is the need for its effective testing to ensure that these models meet the desired standards of performance and user engagement. +## Why is it important to test GenAI chatbots? + +Testing GenAI chatbots is crucial for several reasons. These models operate in dynamic and diverse environments, interacting directly with users who can have very varied expectations, behaviours and cultural contexts. Without thorough testing, a chatbot may fail to deliver relevant or appropriate responses, potentially leading to user frustration or mistrust. Proper testing ensures the chatbot can handle a wide range of user queries, delivering quality and user-friendly interactions. +Moreover, these models are now often used in sensitive industries, such as healthcare and finance. Providing reliable, unbiased and secure information is very crucial in these sectors. Testing helps identify and mitigate risks related to privacy, bias and security vulnerabilities, ensuring user data is protected. +As chatbots continuously learn from interactions, testing also enables consistent performance monitoring, which is important to maintain reliability and prevent unintended behaviours from emerging over time. + +In essence, robust testing safeguards both user experience and brand reputation, making it indispensable for any organisation deploying GenAI chatbots. + ## What all does GenAI chatbot testing involve? - **Functionality Testing: Verifying that all features of the Gen AI chatbot model under test work as intended.** From a04de6673f26e68544489b3c3825e8fd7661f553 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 12:05:35 +0000 Subject: [PATCH 35/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 0f9ac7d1e4..b86e5c80f3 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -31,7 +31,9 @@ This technology is revolutionising multiple industries like Healthcare, Entertai ## Why is it important to test GenAI chatbots? Testing GenAI chatbots is crucial for several reasons. These models operate in dynamic and diverse environments, interacting directly with users who can have very varied expectations, behaviours and cultural contexts. Without thorough testing, a chatbot may fail to deliver relevant or appropriate responses, potentially leading to user frustration or mistrust. Proper testing ensures the chatbot can handle a wide range of user queries, delivering quality and user-friendly interactions. + Moreover, these models are now often used in sensitive industries, such as healthcare and finance. Providing reliable, unbiased and secure information is very crucial in these sectors. Testing helps identify and mitigate risks related to privacy, bias and security vulnerabilities, ensuring user data is protected. + As chatbots continuously learn from interactions, testing also enables consistent performance monitoring, which is important to maintain reliability and prevent unintended behaviours from emerging over time. In essence, robust testing safeguards both user experience and brand reputation, making it indispensable for any organisation deploying GenAI chatbots. From fec3e14aef0d6465a55e23ed28207a5592606d3b Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 12:08:51 +0000 Subject: [PATCH 36/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- ...024-10-28-Testing-GenerativeAI-Chatbots.md | 35 ++++++++++--------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index b86e5c80f3..f9ac21d4e5 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -38,23 +38,6 @@ As chatbots continuously learn from interactions, testing also enables consisten In essence, robust testing safeguards both user experience and brand reputation, making it indispensable for any organisation deploying GenAI chatbots. -## What all does GenAI chatbot testing involve? - -- **Functionality Testing: Verifying that all features of the Gen AI chatbot model under test work as intended.** -This includes validating input handling, output accuracy and error handling capabilities to ensure that the model behaves as expected in all scenarios. - -- **Usability Testing: Assessing how easy and intuitive it is for users to interact with the model.** -This includes evaluating the user interface, navigation and overall user experience to ensure that the model is user-friendly. - -- **Performance Testing: Ensuring that a substantial number of queries/inputs/prompts can be handled without degradation in performance.** -This includes measuring the responsiveness, stability and speed of the model under various loads. - -- **Security Testing: Checking for vulnerabilities that might compromise user data or privacy.** -This involves a comprehensive evaluation of the model's security measures to protect against unauthorised access, data breaches and other potential threats. - -- **Compatibility Testing: Making sure that the model functions seamlessly across different platforms and devices.** -This involves testing the model on various devices, operating systems, browsers, hardware configurations to ensure consistent performance and functionality. - ## Challenges in GenAI chatbot testing @@ -74,6 +57,24 @@ Hence GenAI model testing require sophisticated techniques such as probabilistic Despite all these challenges, regular monitoring, combined with robust testing strategies can help ensure the accuracy, reliability and user satisfaction of their GenAI chatbot model. +## What all does GenAI chatbot testing involve? + +- **Functionality Testing: Verifying that all features of the Gen AI chatbot model under test work as intended.** +This includes validating input handling, output accuracy and error handling capabilities to ensure that the model behaves as expected in all scenarios. + +- **Usability Testing: Assessing how easy and intuitive it is for users to interact with the model.** +This includes evaluating the user interface, navigation and overall user experience to ensure that the model is user-friendly. + +- **Performance Testing: Ensuring that a substantial number of queries/inputs/prompts can be handled without degradation in performance.** +This includes measuring the responsiveness, stability and speed of the model under various loads. + +- **Security Testing: Checking for vulnerabilities that might compromise user data or privacy.** +This involves a comprehensive evaluation of the model's security measures to protect against unauthorised access, data breaches and other potential threats. + +- **Compatibility Testing: Making sure that the model functions seamlessly across different platforms and devices.** +This involves testing the model on various devices, operating systems, browsers, hardware configurations to ensure consistent performance and functionality. + + ## How to test? Testing GenAI chatbot models involves evaluating multiple aspects, such as the quality, relevance, coherence and ethical considerations of the outputs generated by these models. Here is a structured approach to testing GenAI models, along with some tools and best practices: From 9ee0a57ee7f18129dab1915d3e2bcf682f6a1f93 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 13:12:04 +0000 Subject: [PATCH 37/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index f9ac21d4e5..99f025670e 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -36,7 +36,7 @@ Moreover, these models are now often used in sensitive industries, such as healt As chatbots continuously learn from interactions, testing also enables consistent performance monitoring, which is important to maintain reliability and prevent unintended behaviours from emerging over time. -In essence, robust testing safeguards both user experience and brand reputation, making it indispensable for any organisation deploying GenAI chatbots. +In summary, implementing robust testing for GenAI chatbots is essential for any organisation to protect both the user experience and the brand’s reputation. ## Challenges in GenAI chatbot testing From e36bb7b94e740afdd1ef6cbd1f1bca7c894f5d83 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 13:13:19 +0000 Subject: [PATCH 38/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 99f025670e..b8b7546c6d 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -161,6 +161,7 @@ One innovative automated testing tool designed to evaluate and improve the perfo In summary, **Promptfoo** is a powerful and versatile tool designed to enhance the automated testing of GenAI chatbot models. Its dynamic prompt testing, contextual understanding, seamless integration with development pipelines makes it an essential tool for developers who are seeking to maintain and improve the quality and performance of their conversational AI application. Below are some useful resources on the tool : + - [Promptfoo - Initial Set up](https://www.youtube.com/watch?v=VKyJYgz8IiQ) - [How to assess an LLM with the Promptfoo](https://www.youtube.com/watch?v=0sk5rb_ErOE) - [Using Promptfoo to COMPARE Prompts, LLMs, and Providers](https://www.youtube.com/watch?v=KhINc5XwhKs&t=753s) @@ -190,6 +191,7 @@ Another powerful and versatile framework designed specifically for testing conve In summary, **Botium** is a comprehensive tool that provides all the necessary features for effective automated testing of GenAI chatbot outputs. Its cross-platform support, robust testing capabilities, and integration with CI/CD pipelines make it an indispensable tool for developers aiming to maintain and improve the performance of their conversational AI applications. Below are some useful resources on the tool : + - [Automated testing of a Google Dialogflow chatbot through Botium](https://www.youtube.com/watch?v=b8CgSkJmxCA) *By following a structured approach and using these tools, organisations can effectively test and refine GenAI chatbot models, ensuring high-quality outputs.* From 8ded5b6e2cd656bc39dc01665ddd1215981951cf Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 14:42:18 +0000 Subject: [PATCH 39/54] Add files via upload --- _uploads/Testing GenarativeAI Chatbots.png | Bin 0 -> 41811 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 _uploads/Testing GenarativeAI Chatbots.png diff --git a/_uploads/Testing GenarativeAI Chatbots.png b/_uploads/Testing GenarativeAI Chatbots.png new file mode 100644 index 0000000000000000000000000000000000000000..a10fe5fbcae948b2c9da8090cc99665172a727d5 GIT binary patch literal 41811 zcmc%wgOg;<^S}#_ZJRsRj&0l8@yw2GTfI9wwr$%scFY~ywr}q~-`^kb#(N`fM0a$b z>eJ_(%F4>h{AAV-B?U=DI6OEI5D-LZDKQlg5HR2ylmi9|*ixqAlL!0&bykrS0jZiK zI0ha-muy-!CK> zD5xs`zQ90=g(Up@RRk23l;YoMN<#lXpO%e?x+@h`{HyXa*1!VeSu&ZtyM$$rSUZiH zWJFiAGCoT#Dn#|W;!pC|Cwq!I7+dQG4d!Fvg)S)u2_Ts&+ zh9v#Ie`Z6Q|P0C!tKqBgCch8`W7M@m+q5#a84wWYNyP$UlfwZ|6AGZoRweu?$ z9P!f^@jL&}5y61wTJ{&%pR!sKy=qa1db>GnJR*n!Uh&4^?kvgw3E+t+yY;~^5UyhJ z*BhC!HdpV-XQiO%v-zXKF*0=K$h!u;v1X4Qd8lT0#N&lx)z>1Em2 zd4NcXNR=wJI_W2M|IhzPP3?(IVys&YlB_-Q4J>$+=}QoU%12@15sgmdTmJZM2RTg? z*qE+}a=!rE=NO6I7+Rew-m`K}+U>Y^U-*ARfV!hfx{s|FMu2O1po{eOHWHg96IDM< z?K7P&)@yWtEEd%u$HVH&45@56feR1$cX&}`aJm|$#5EnNgA-!vh}WgJeGPK9%+@DD zp^As6iW7dlW5h_ta9^nw2N*hgiuNYI6$Qw2F#K!5*XO1n+HB5duv=@ z6`>{L#LoPxKOHKj(PZneVr8O)#5D=&PTDQdV-X1}R|F z8mwYLEUs29xj?xoin@M^E4bg6%4%*g_!BcBBV*zg6&2OxxG-5oO$}q}^(TW))AV+K zc&T>1nOc>u2J*nulZQao&HlLRT#@|m!o#?aCQb1}$@4#r`7tplx7e4?FL#!- zsznI?!ymJ=v((hobf%&IB(424PbS_ZSLmKIR}nN4eO8RDJkV2n2%vT_+aaTfo!@Nb zX4A?&lM$hqW2E&pHMgp{MvNAl=e_I@bv<1yB;xm)j^x}JQGL8vFV*k#x*NncNU?qn zo5`1)ThKI+|9pSUZXGaOs?zK5*&A`LbkaZ~E_}6j%N?I77`Mw8gdXj>?Xb)HP;waF+fNtzY52Rq!a@+J6MWV8uEk znxZHlyJWB&A?V%geJx|Zl)j0WXwUt8ShYW{9VW_8NWf@&e^@=PY~ILQGVsr|>v}fB zqI6$#gnob8jXwCZ@*^T*mxjskoX7j2f;&0$=*Z$@f=0&h8uZ28{pt0!z9+B{X7d(o z`1C@{2|S4jhdSH$LHp$}+y7`rlGv3fpOTq5wmCBuwwCU%$-w3mz*Kn%i9aFcg{PLV zByYJFY2P%TINX{j2u+AMCJ~7zzUli&7)i*R@oeGq?ek&wq3vRoUOI=} znp7k{r&;r6o_NHQeWlQZP>9hOjSbuK(r&ciB%_EgeI~DaQFZsnA6@c@>)m11Yysb= zw;H4h7eZdQy%uV~?$|kl>Up`(cRqLv~E^ zn_a$^6DiCMmUEO=^ChSezx+Snlb1w$IE*8(7NKnL zhao;|)m(ytLrtgKcUe|Rvw2(@dOdB2TP{{;H&`vuV%hn>T|+bIcciTU0$}&IR_S-9 zJzT7h``^A^cF&n6>q&IKKj263Ckz8cWZ4%2`^MR}_Sa%>=YjVgbdfzrOGm}`)A3C5 zxu@<<+-4F|UH)|aGUe6(#?E_PzsRdGgBTU*l5OhMQ=1jWt)W6Z-0Lmq%5`=yhIOIW zy&xQ4#Je~!m@YfH4}8ABxSiug9edYb_?(luVqv1NAkK*4|pI(=jwLWau+gv;3VCk|ncR~-@t(U6y z_hn*IWp(S#r-&p|SxgBqY1JGExEy+P(+0 z#}t1TtI`tYlqs$Yt5a^I-p7v?4SY{)Z~|ABV;h8OjZO^OxA@awdG;WZ{2@u?8N%aw zGN6&(e%Y0t@RTzm)t^}+hV9>H1y1+tpsM}C*j>}hi^q1OBlY9s)m}f!gREtrtp;J; z`?B-xJ=wr-Taj=nVUD)&aJ}`EPxuFqN*F6x1~&6sn?i zmm0?2PVH7F(_(y-3(0^K`j(4_%S}a@uTH5)Tud#G7u}zJA5{?K{%}Cs&^H)nujd^o zEF;W4uMalj=Oh^Ff;Z9Mw>RL%^8*drF3*s*{J?N805HZTHV@z}>9gj%@zOZ6YUlsd z8)DWnLN&rj3O}`(&Jk|#d-uW;s_uHBMI_`&7@SL>QEfTg0QO+`G?U9QNKmK2qBmP( z>qErB>;7b6B@>!fy=(y3Z~#6WMYyX-q4kW;$VK;u0*YBpDP|1xRahnj4wE)Twmx7L zjey%J1bh(`Y`4-79|a%<4SIJivPJRb%auD!aRve|OUU?XKXN_tCHX&d2M=6wDKfVm zAE?=5-()olYsz(X@^=KZ%f#kTUBBW`LM`Hc;aZl&v_F+i6kwyL zK&ocJE#T32swyysZuVfSnyQn9rMA~rn`~v#sT2xE_hfc}5 zF?%+xl=rKe4Oi++Gr{ElStSl1x`J7iWm;3-*68rS>xeNK3CsPv*H2}T&~#@cnIe9* zkCzR#`*G5Py0bTDo~W*smP~35_AQx;1mIKmL`nsw1U6c(wK%fbZ}2&ng6dksbcOLE zXpq7{pX2E?+2Hzr3q*lMMm7hXtuvL?ce~mW{%EK(>WBK6$0EBc0p~{dw~7t2JqXq3 zEVc+#M}*}!I(KrV_tKV=LxAqiPbR96gQ*LrsFF|TU^5v)xZsD8C7kBJR&x8Z-|E{= zdr2UdEs!|{yNU)hHX#wHn{5v$#EGd2ex$J&PS1n~eP>@jx_<0DR+~2y6)3Ts7uD%i z6`csSE5ydcSmvJ|(8Yo`8ySMMKja02jc!KPLOTjL?1H*1ecnz9uE-=NBn+$!8F-(w zM-lgV)l>=5gYKoxGAn?H*nGS_pGAopPi66Y-#86Ok@^Ngm14!{gOFr`Ofz;Q(rMll zOBCw}DYkdq2v|m-(93eU$^AJG!}%JJ71r*4d2nfXvDP}-QhZv~9u*&F93+c0{M?T$ z=n&0Mur!&;n|W9uoaDUM5=4i72yk9lq;27djeyUC7g6{(YBB@@duLCp1jDcKSqX+aL=L&Rxs#^Z{vWNaP5rqc$Ss0`!#KPF0APj*-Pd}@6O|m*%G7=&C(~(0QQc8aj6R&NX8IH3 z;*RoPyV!RWXZxis&AdL`-a5NntgXyc>G2+J{=#JqTWZ}7!}sXYtkz!(M(KR0;KybU zhDJWPcd_TQu9i$N*&T|^ygdT-xY?7|SNuuh{wn;|Ba`yS}n`2%EvCJk&=TW$*%pJ zqlw4z5s#ihy-baif^j)0MBgUuw7NU1k{<9wr0a{i!dfu0G@sG^OwSI4CjMH`NNY%r ztyV+K8an?PTjjULoiC{l-Mw&&Il|x@OvaTp0y~w4dO?lQoo6l;aZpvl5Uu2b7=FQ_ zFss3`D8r!xY2eUEUnme(2*XKDV+!BK=&JYlG~2LA*J5GV!Xwb7IUP5Sj-ks5g077A z`?UN3n8BEp?w1>D<;b0A%o#|!xm*fo;EYOV=(T3ln*Y`foa^3wIw|f|JObC2%9o4AiypC9xW)Ht9QYctR>|- z{Sg>{wD&kdm?%xkoLG9X!`vE`?*7|<{70PXkG}pou>|RFq_fbvav40XKN;ku5<$jQ zKR@1&Q_Zusa}8rK36C<={>2J=ky9ZF?`O<#W!^N9?B`~T%r)6Eg|@%5i*^m(%f!@p zLX)~ePO7~^-(3I{~5#o!4>LqWY0({qvsg&u=d1JNbpC}4?`?a z7?saWU#ZhM-o=hDUC4yPw)oQRoD)k-Jj>*Vahz(2`aWDk*=gYcSTCXVV#RirCy%m) zLQEndqrgD)zbG<>Qvh8hYF_LH9#+%)>o0xS&FXx;<_-y*m}P?>)c4>E2Fyt!t7_Yu z?BS|&N1^wtMOxNM!zc%>e{`o2+M0)*2VqP{E>a$DQDA}CYZSS4X`}p`? zhlYkRkop@e^>@1A?d$JV5S8EMyHZo7KLP_?Y7Q~gXyzNemHy5uh+ga`HK`T!*`uDG zQJ1}?Y8rqH9U{gEW2aYlR4%|8|BA=lmgB|nmB`C)7pabrTLA;j=59=s9s>fM@=N~w zA|-)c(eI}L`=XJT%a`&J=TsaM!hd29A?tCk>U6;;Bz}h{#D8QB)hLzu=AkXnMS7GVwvV?wjc*03Ugw6d^JEKNJ4G;kbFX`B4Wzib_${yJA3xOvgLnd5R z#x z+>ou?6&rGbpZ;}0f=Hf%$bQ8BT{VEN7YZtR)W0C`|1aY4|1LlI|Kw_d;eBt6fBhK| z*#0acbH8?vE95BV?nIvfvVTZ}0!7knOq?cflJkEJgHkWCf+I;jF$AT_hpXoVsK`&S-zb``L5-=c))x~M_NX9^3KsA| zCUwrgv41YdJ;d@mBAaGt%li3!oZ#rh8fam{0V*u39R0Kl{zFhA%rP}*b`&{Csm_OJ z>yBa1zgbn9Iwf(blK(c$+_=KQbPr^Vr%!}rCUp2UaJw@G)-q6c%uqW|vc_o$N;D;q zALzQh1shY>t&8opw>eb3Ui z_PhO65Lw-dsoax=nq2M7NCSxZ=XH=#I{rpK;Cw6~{D_R>yPSZBmc8N7s-jXd*+Cr;EIY~3_46TEci zR&<=qW~otR(!^@!ZqM+ovCf1z7?#7eUF!n%iOa=u zp5Vxl+MVq3f0qjSwU6YD2N|NylH5Z1g_w+@;N(7Igq5=oZ+~5NT1JbAJ1<6(R#1%1 zlV{0Uf{4ah#A;Be!X;ex?;qO$YCTjrH#rZwC*HN4)%`w9LvJ?}>Y&Rdj z$s~?h;k%tj+bH6H|AnU2Nkftc1v$XRx-=q~>Uc{1K26~2vr1F)hS|}X1DBlMeL(!3 zqd9ZEqKR9lGzJ8&$}5v+lV^oB9lAjEWfUbU5ss0#QiHD5jrhNLsw6hD?3o2h3OO4E z8Sn1^0+|@O^Bf2GpLS8lhNwwVd4Ho^iL2AO{=WegrmlU0d{H&?frHLT7Wepeq|kuTT8iTMY+`E)Q7-O;fIfFS z_Uq^u<@e}Bf>aJT82F+bmexakG}3+Cst}CF9n~QA7Mm~6kkDze5&!F{A?QWxuel#( z?BPl`CY2%n4G}a{mcv0Kf@(iXW>F8&9sw~o2$~{r1%$|E*2F4g2^jzN>=5$on~7$c zRnvSi|GLCS;Kv>@mQ9tihhga0FIkPq#%!JK?~oUipVK*f49a+@Dp}hKQia(_6ya?b zps}PFTC{7r-#Y%AaX+ycq9&fmPu)e%Vxy${oFZa^QenV`1P6HSh_IB+<64qahRNVA z_>k%U??RSJ554<&G9eL)|IqdpdPWdcx>8>;YKAR+GvJHQ&CA=86^ls>h_20nJ0PPW zzZXM`Cu29R12l|mlq+6QR!u-00|h7*PuS8sbW&ZapmE7-b4N0fzi4NK7)8i}XZc%! zK&8f)^bb3~Z%f1UOnrh?VrXJ=v`l00-#a8Ox$I$RbkNp3r6tbfxi^1MlZcG5vM8X2 zCb%;dEe|FeK|DbZ9enb5p#P^&k|b_N1ltAaC?75qQ3!msH+#@nD$Fucq1XyixJlHi zVxBfX;8e)Q>QhQG{t8@D0tYfAi~Vpd_%zTu&{{?JpQNZIK3;AW+uC$EohL~c7$QYP zL?*bIarcAvK8Krlt^{Xie2S?g&tJ;-u`3AouXh?@r#m! zC54N#aQhRVyx=xGP~Wh>(9qDh?2VxExSio-3izhQ#m9SkYH8sB1G6Yv%__M>TJ^l1 zp1AmV`4%o*r^C#qexmC-XFn@Clh%D!`a1x5;)@<#MC*nU#kp4gYJd$4Y}G`qGkUIFd4u zQj{uY3oxF#`S|etPGt>0n9P(l9ZxFk4Te_z3lxbSuu72I-Or2C8sQF%XF{N$RUKli z&VEZ;m=#5;%&Pga`4xp15k+V7ZFrZwk8meI$l{K5vgI^sv3=8fMd zz9TnpWeg7~Z?<3 zbK|3(*Fl~afTc2uKdf}HV*AF=36_7V;xWg3kl}rUW|~&ggg(N=wV2@?UAzjf(ML5e zT6=iRK{|OuSHN}Js<)0UHGgd8L1eHU8&hY#jBp=C4+M+d`+rdOZ<^oSTWxjTFWfe{ zeC~5_SX^LyD+k2ABzOeB!+}MClkJW%>Xn+_9?W-WJD+em^l%?Gn!F;VJuhftU}B29 zwP&84#Cn9I8Cc}_B5v>)c9 zmykbM&!esh|Di%ycEf7(-(f&b6F@UF5&cEsCj*!yGu2gEvnS9xUQpUuy|JK zdVKb~^d$7X-fx!yVA)4g8YrWXs;ki;If=rThU`DakdtS1ABIwC5G#tycmzw*)X3>L zPx5EkO{wNW^I>_PQ;`vEkeE*d@ zo<*f@7~SL4n2bcl`DUJ`?R>JA?J~RDlPGfJ^|DJc*dAXjcSf6&muSSoM&cKau=m5+ zwkbmb*V|KmbgBAy65nMw=YsCoLZQcRWh{wZ5W9WaF9~kjs08|kME6CEie8;N)LB|2do|R;l*{MoDL4^ z?q^_{=Xk|DyKr|n5}JWt6UbF=f6E4yQR-@{t+XS|yB&zK2rxs^If}&=xzdkH9Lq=@ zzYGcQG3UM!f}Qre3+5CX_*@WdN8dXrh=w6Be@L9OoFC!Bl^RwgyQ#p!G>*pnWVZR? z3(Oh-;iaF`Me#Uh^2`Koe5nZJX7?5*^fRUkffiD*5bCg)wy>Or(03@H_o_M@mi)WW zb61`eQ>zR$l+*LS`3R0GpIsPmBmp#uH_nr!$!l_fruKz7C2Q+HtE56Bs?}0x&3B3D z;K1~iQa0D~@vtPVpajWdmPo($C4JJcRF7tj7x=k1%8z5H(k6fb!$_+InfV$ceyg>T zE@wDIg#0=4$(Y|}yK}wxMz@sm$8~PWD)}#n8LIJNkqbHonr)3!ztu>~*tqA@iA_9( zjOz-EVi{&qSguoJWSSaW$o+qs!u_IM4wd62Ft>>>#;7YcFv4f%+;ecD~jc&p^@XeV0d+*`}1@kJ7#HM|B~gqAB8%ky7Vp z+xGRbabAPmjXOk%?^ArX9&EInI@@R&Uz|#0x1_J~>DFzAbSI{<1+;hD3*l)?3PEZa z0;`Q3)uciG7#9U|YS;ULj+gsddxQO^fH^rB>RP@xGM>|LCfi+AuFGoAYRS~SZ;9>c zA_f+ogk&_Ki87=i>`J>kQ#&Ay-8yOg-qH^FQAk?IUh4af=e$1~c^Qo6O` z{;XPb{+{yK{wK_fj-9iNqgYH?XjyxV@7!<(AEhiy@uU?Uu&cT>6`LZHda@zlWQFg` zn?i4KO;wZ z3<98-NNJRR%n%<%Ufb7`Ech=KiPODYWj>Zl=MQ!G(&79h;bc*uQ4SLw%HDrkrJsj8 z;cOOx*-tG_oCc@MD?c|`)yf(V4vikKgx8Zl;dBq5+J|5Il=e#WTfn(0_s?^RK%=v)#@KOXyH;6Kds1o2inDV`n0!{lp~ z#n$({9bvgSJvY@M5{^u~|Lk>dHJ8fj=(00FlU1!z3Aee)x$aTv=(`qh*w-H|AS(F& z)_%)3KxVon=ZLZ;NjhlyLuCrO1ExRCq8WFXj#=em8>f_yCH)WziX*HCi8lS!c~!^b zjQDDoMB!GLJOg{H7kzaYhY#!&dC<1)MCo}1MkU#?TbJ42F0{hFl+@RmM{Z7b{R;&T zpv^x|X?{enC#8%fNzqYPPds7t^F-7!Q_1`;u-d0nt)^>NE#^p82{EW~{3u@F-r;iW zNO7<`>0sSeSq2M1ShSJvYcw1$`a@jR+{RRQLV1$%Oo)Ec;i>2mpOB#TV&98aY!S3y zCiz(9V!EWDu*Be({&{IEBY%I<#e`)jE=DLSmmo>@lR;S@OS$Z0`$Mv`4sB!46BzVQ z2mhXX%KtCm7p~@46vGHexrU!uB|9tyfPogK0f|GfGb4SAy<>ELzAA=?O{_6?k{95x z0^tyRqef~Q@#$*Yq`~%8;ajT{!}*`i(!;Mk{2W$Kxme6Jf${CXSDD*933O!S(YKCi5yVPUhj;Pyot4Cy?2 znp|1$ugKRr*In<>Pozk_p}T?}S1=6vbwQQdK%dMgSvvC%d#PwR$9$Eo?9ZP+U0%a7 zva*D|A1{hc$2I5D8u6cO@v&;XT&-2x-55%3)05QNTo5?XSi3)dXjJI-kJ}NiB^#TV zOjPb6>GH}ZRTX$&_y&!mZ+UnbEM-!52qsla+ik76pDoKPzF(TYxISe2_%hjUrVVvJ z!Zq40tm)G60*r(Sdt6s*j9A;1r*U?8)XOyz)7WhbKALd-kzu(X8F2l(4YyfdE?y1h z@1ybpn>WNF@fLSa8$8F|zBO31zN`)1!{uQw-QP?SmMCQ6t*l3K3uyxxeOQytx`Oqh zjoNTyutIiJFh>rsmgZW3v9p2Qpc?>iNL~W+=is}+glx;z$@BRxSDm&}cj7UnCC7(s zXN#4w{e-G0!rrN5gFms~*sK+53ym{Yg|&yMzDz%?^IG;7AvRb~5%G9Dk(horyWu%< z)!dCcQR`OFvs$QtR4-NI*7*Yqq2mAE?|%0f^xWVv$im|})}kmJLHWDHE%F<|lW#pw zQPeq)gI?dqM~Yzdhnnhh-M)p=W}sTLhV4DpsUl!ir#fo))xa;0b#z!g^Qia!kC&jf zFw^FyUv}|HFR{;NFJag#w>kIrx69?$bPn4;5~qvydBBWJsdd_@={JM*=HF@TPTzbT zf3*g(=;UUR2Vb^iH`#9vjw_i9zJBM^DY(FUfT8$;Q*{W1cru%8AgA@hpHdiUm8S5*^LWI<1j;e)TXki4MhhbW=)1 zlezTCuMweTV93;{`X)>4cgHRGR+Bj(c7ITght0Q4!Y7mX^FRWF@+sZEEed}u1DKQk z<@a{X`EpmE>aw*(x?nhT+vEs4tt25XULzDpa=Lj5b+Sfa zwgGP6nL5;Yi+1UsZgnIn@WJ=~ywpt~7zSm0RzBrq0!rc`@ajVu0h^(?%E54S-aK^0 zE&p=G`tq6$xh9kk8u(STLOk*$dAcUIc_!e4)5##jUpYYOE8aZJ zhWnI(%i`PgBY0KeW)!WRmP`g@Q(bg=7*)cUn);ec(A{{H?182Z5=l%V^Q zaz?>oq?}ssol)$T+Ta@TckjJiRJO-x|M^y*-y+lk02U=N95#Md#z)h zq3@znJ3JrvOa3l*%Z`na5M4;r&Nk4)Gd05eZP#J?^VOI4YNaZWx7CPB*#c>AkddCp z3|L2q>izE%pG2n0kE6u~Gc)v{TN-^*TttKPE4c1Hg&93-%r+&V#HBv)<*DTiqCCoBM)rXr7 zza<5nkdtvQ%FVO9Bdam1td|%GEoVbGTcTRQUOOK@lx?1S`C&M@vye5mLs_2AAG=#7 zw<+M5I9>=G^^_hR1chqWdA20Fl6Cc%5I-K~rn>TmvphCD{dFBZq08RWO?kJ=1=|^Vxi+cE{k020&jY>hadQQZkXaOpow$ zdeQ3yD`fIi68UIHLiFg+;6F4Ls zN2^*WLqTiMg&^2qB;FIksjn4xWmsj zSz#m`?7LgJnq-;JSf*TB_5eG*<@ecN&F23>CXg++K)tIg@cCwKw?>Nf6}d(cK_@V! ze4kF^_F=VTh>V)LyjDm%m)RGnMwZt*m>s^EbHLId7l9AcagCXa_17nPpZhe#vKht0 z%|Ff7l~J4ZxuuS}-CB1ftiV<1vDMcDTAl?tyI^1eP08eTi@3`kokbyt;nYuB2T zv}gM=$77Z34z*@PMi)1pa?p@=#w;xz{x}UuhOt35h2lJUK}v|>NN0w4?bml2(Obg z|H-K7vgZxA1nLSd0l$G!3S9-F+FIuiqnO`V_v$F;y83!72y~eK>EzwUeU@)~G?=M$= zCML#e&-Qz}hp%XlRc%FtjJEQoV;QeEU=EYc2OkVT9ekh77DR=VO?U(!cm7Ge<&g2} z0}m*Z6*gT}b=c3s&}u_`_CE4+4$bx_fsfmgSx*48Wr5|Lmfwzl+$0#hiPF)6E=b!} zBSQ{xXs$k<;hrMx0s6S1HID}m2~f0OGn^l8r~Gp~6;LX&aP9dH-$d7R?%^X2nN`YE z@KCP~1&$ciu2A~HZFdS3Gm)wJ=9yXxSab-sB!cN+FSyRVgiSyq=G-?w$B-QnbwLHE zUNVV2mL)p98jl9CvkEM;6?3YK{0Be_0g*kc>_pQJ#lG*qrzT0%v|ukf4;4l)IbyCNVp^7jvy=8fbWQLPV^LYJp!<&Er_5^F(T6QD4XNFI z7ON^Zp5~q2kWL5Z6^n1V^M0|9G$AqP2ByPkpwIuHjKsfc6!*1SuBlR$%j4T33ats1 znl%&z7S0(Ng<(VL+3k-d=F%~C|B8lRWN(3XrGfw;X^9D{F2TIcZ`n5*LQ$*2Kmqu* zG?btnqPQ3{+)o#OKU07j8!uM0(zo1s_T4~WV@!%}5-*wvbp{lk8py6Gg5TF&T?M<( z-FJFlUF!(1hhd%vzJ}LPr~$!Ivqmq#WWf|9Fx*?^5?ut92@|DM z5f$>P*}(r6eRJyNPcQ9!o+r8hX96p|K9}7*1p#=)(53&^P!8LTv87Q=kH7#(l}{v4 z7;DNClBP|GL}vO7JRA5a6!%qh1e;My8`!aA-e{n=>~fe1tdHVu=T)Q@c$@FcNR1f_ zBQe-+jS(Twr!6KwU2*LVZnp2R+yxDT{sI9K(lZ}DJj2N<$Wctg=KDtsi`zmBWTR$A z$ZpkaKwvAk+iJlpN)pUN>7nC**6lneS41=znq>e(ppiP5;*0VR_<*79kJpR5#|__5 z1~@#vtJ+Ma?i8u&t~Wt9_%<|%svgl5+J@yX0O^x$T+u zO;5Zn@hF0!Ll`-vz=B)uGupY;kf%25k;=Qo7w-i0CTPumS(7YdY+|uuySFP&D#c8* z?HyPx;>hAgqd;}!pb^YvM%{9qrZc6;*(T?r%ChTVi$*wNkZXwMrHAu&PlIJ>G4ThU zE9T)Qgd>0OCD01!Qv_X3K>p0~Inv=X7!3YZ3>qqHk^hKk2<&Q@4iqA*(0gDb%+9x9 zD8HeJ%az%m&i(A+IHH7o*IL!*xO>{cy=3-q}%Vc0A*P zrW3s&=y8Q|#Shv*x2C94uFld_txOxzVVy+pL%fdUp28jOwzzgWbPW^=LRol3S&Pc{ z`$$yvAp5X>RR_eIsmYbXlQId)RMQsI`FHhooDc$)}p27;yCT zKsvJCXuU05xws{ljH-tj+JvhT4v$2O-4WuK)YHxj*#X@Z@9PYor0%K%_oOcf5EL9> zjN&BYbJ+gbE2^Q|<)hu^y-Z@a=c2@hN6-=2Ul?NOh)DFO#>~>l&{k+$V7bkes=L)H z0*)xBMrHf=Yf^gk?o&x`@Qcd5mHP8z9vC9yB;;#2iby}?BY6cRAbMu21BJed0oS9a zKdx{P6c3|gWs`LW$&SECx~FM+c|)Lum~-}Gho28L?+Yhs40^(Seg5!lKwy9AW`NP* zU%WA1YrH6dnUgzMkW3o$w&8uD6lum=j0v9&FbZO?F^jb7AX*F`1SMnOoPOHicR|rP zCjn8)CaJ1!tE->O15;XrHQS&AQGUAYmd!w9hgi#5wc9=o@%s4!u?@EL)!3S(hTTCO zSn&TsFk=CUCEXI*P>6U4Fd3+|JcVCNatPk$gh3cMB>jLLzpLFvPome-F*={=5w98+ zyo0Q9YP4P&JO0L7F=qXI!#??nTC=6y_H!IX8ty(cSm{lxxk(J|@1SiO0Mpacz2 z4xAR!4PHnKl)yvg88|p$3+hgi}wemG6A;fZ-0mwGmY^Kd9N;I);T_q*%dRJLchHb*gi#VXKF`zcHlQ>a7!# zt&MZjh2fEmj{1n96dxBcN_tp!AHvblK;bh<9Mw3=Nx>o8m;0Uf#p33W{u@Zu@t;Qg zOhg8uGyD+INuCQVe&T9aW`NsdJSQVBB@4&X-F6*v$6&@V}FEIVgP(GR+cEhUeco~lGzwHBia?Lu$+90YjbP>QLZtVr>=@^P=S>$QEy zjBI&?O?EYW;*Xx|6W7L)13+4zDJv*VIyfF+q=D=wt=5vkteqSLE_=J(Su;k%YjE9_ z)f2%FfSeTZlE9GM-Q?LgxV9DT%Dad!2_z)piqkEf6<_+jMGtifW=inCLm{?%V?^TJ zqy5f$*NlY&0VubOq3FOkRC(0S)JC-w0TyO&YoPM#g*)3CJ&1ZTp&tt|0k&TTT-7J@ zv%HwoUojuM8!tfSneIO3gNW~OlFl-iFiVzd$`+~;+4m5uaCcfOURzRx`%_X=(ms2f z6dbq2Wih-CNhxWO-HF5!m<)(QIs(H+0kYdOFdfnYp$T4F*tJn(T!a$G!@)GTc9GMn zVn)T0Gq#*OvTpvW|S^?d# zJcv+KPRKx(mzCk8b4~+8PG~$Oz78Y89*}n41-HYGckD=CU%tbnR__6+rf-<{t5gs% z3wZl97GEn6S-nDq+1k)xL0$v6=%`U}M^;UIBf>)w?9X+n$fleKXYLCR@6RoUe!)&I zlk$@aZT=~TWfrsSH3#JFjAD6Y%${otuYhnzqbTm5yqIwBbU93NG2WqBFO2z4DFG-5 zHIofU@G2KMTjHi8Hy0d@f~Nb}MKXg;F@)Cu?at6^4q5ZT4Kb>BQ)4T4BmLwh{@v*% z2HokNAV+_9Cfyy+B`U3m;5NDZWW+pnsmH`@fu4a!(qI9B$Iy2p5h9H-%$;@oh7tu( zZ0HgbYT5c|0HaGUzZry}L$3lKKy3m)t#Z4!n#OnEFLPbFzzhK;c^ByhbvIAfN+FzN zOxqHWyxd+ByBdUuW2AMBLw3(fW>A(9Bjp$;M=iv=%+b0$*BKF%qKp)>pLDqtzo9As z1Lrzg!`2`HW>gY6Vd^@X5kUv42R10 z6CgY1Xs$tmf;t_M<+x}a%Ejoe!J46{@JOob?trV^9P2E?#5xehFZgn8Q!Bmk#e6+P zOGr1pe79BsknHu*I?PR_DhyGNnQ+V0{UMy6m1Ao9M` z_c|Gc=0*&HB7h_-5NN_lZ`D-gEor~=fPJ<>7em{yyHEc*jRk&pBx>R_eEyzUtMt?!P1!l83T| zpuaeT*$4AO)LGyC6d1UhWDyUF^d+XxV84ekyd)BIm=Bhx;PGh-}Y<0+#t=k?<`?Z0jcWD;;;U`jW`XCE$w<( zhK)JEu`>uc97*E)OU4i~Xxge0!s9vhdcVMVY>~&8=bry!-^=oqV-lO@44PkWJ8hg{ zM3BiraC;x_7=1u~4_7;IU%x>tQ)VY@8C@1*$Ps}ST^dwo1U4)9dsXj=5(@P1RNU+a3i3TfYkMLpUzpYXt?JFriJb_{$((%>T$fH zxkHc`h;m4?6vFFQnj z2`khicQo6D{WKDB4aS7Brx2y2~66_`8;0(h{;Td*aJ;UJ;_JZl>?dJdSn<{`LV zuYfOU7kJEUE=PKyJYR25iGAJ9QTUK@tA%Gr*Fxx`uoubZW%sgX`{OV$0H6qx`Kn$Z z0IZg$7&i3IKO|(Xz5%9#FCqcnh15jkE&emCXrlWd5Xka}HQhMDM zxe67cn8cl)2(mFb$CzY_v6IPLWSW;`z-?yFNCmdIeLtf>XtS=*GnNV;j}K}$4EJP_ za%U2xqP+piGzdwh@T*(p1jUfY_ZpPmxVU=K!CoGP%JZEE9(fMMkR@DY2)AU9Z^%hgtra z`u?}ze_BRy4*FMf9u12X&NB}Ydmk1ii0fS!mU7NwIpIu$V1>Q@HJt_=!JRE}pL^|f+nYyGeBeN&&3474qZ!5Tc z-auU87w1QbyY^?xOfxAJJTk$j*r~k%!qTBJyb_muOu&qsoMR67xY3WG81x%Op@hp2 z3?XyKyM_V75E5r@JZL_-Yo*{G5s548 zp^iZjSmcG26pP}LfF~M4{C-3PJH{65q0UFt2pR|~aIKQ*7V;Ri?!mlRu_;b#rLPs_ z%T<1JlA>Xp27qv;BHwY`L{$52+K754VRDcC#LT*TFOmM6jaOpl{2&w2soLg^RH^*L z5h2_oXKDnCy_lt8o*X*7HSO+++}P;I10zqA?%UtZ^&X!%nwNvgn`X7QC(DlXUtRKH zdVyOx<<+~ikvC2j%AK4f&+|mDK9gCyb~ln>^oKT$MRJ(MR_K6vhEo7ffZS=;oev!h z43y*rnbtl?6lmZ)vF{sXdkhJ+GAD@!Gxco>_U6u)7>f~74?|lTgqk8_YV>l8pit#v zoSrV_X60dK2qdq(Z;&WrR2AMD@OV6CN8S>*kDKt%FTM}YOS;YePmXR~dVMX}%s#)O zW<*ww6bN8CB+Z=MuZajW+IZ2@tcI)w(G+u&+zE(w~=ZoOTv zx6=x^Etq(KH8)no{%guPbFQP%29*}y*EqqtY{m2s!oxV>WOs;^sFqhZA~Z;3LAk>D zege-7AtIQ3CMAzDN_uTHtOAyS?+R-e1jn zS{-KZEXmD3Iye|~IQC6?)5|C0!I$vC3SH@8AXe-4%elD815hkCyV%>!oPt<-(mROo z5r(0gA&kSOwYKcX6Y0hy*^CnmL;GuUdqc|Mhr=XLaNDkvqDWpz)ET8FV5Yz3_x2Uw z6D#&O_OQ>DDD!V%(JOgPuxasc>4(JhxCZqAjSSznssZ#d1Vq{>)YpDjiM(mx0vZ9? z00ma-R(YZu<`6CnAnCI16;X%&&>qnkvG0Bj6`$wae3yep#_b>3O5?xl`FfFCiZ4Vk zW#2tTjvxy*T_cLniieNlI;ypV@&{$k;r1vD8w~26UgQVzJTw3v#&>{jyX7P8v6}XB z(_@6F;?Ki`7K?lG% z=&I=Y&}9u8F>e?fnC$a6aIK@N_NI2(KtLNTX@}WwC@lNsZv7~4mn&P}bvS+c@Cafi z1sD}c?cUIzMvh98B|0?_#uWksdbwTs3Oijroe>?X%TtGQUCMb5pi8n!_5Tk|XB`*C z_jhq>=~!T4=|+|gQM#5T1f-Q!x&)+4Qo2C|Tmg|>KvGIUQaS`F0qO4U4xiEQ@A;D# zyR$QMWA6RT`<(N_^NvxF=|V4TI z4*=}@$_s^z~Bp(ab ze8vxoc2aVmzA82+v7gdNhzz`tvS|*|j#nhYcV{eHAi-*Le94SiYKO&U)U4qH{~iv( zS~nwYHw#jVc`S2sOL#Buz@=SLgX=k>#^ogQ%RMa*JR=g%?a@!Rr}9d1qmy3S|P zGfUW>46%8)pcR5Zo0*H>N+OEO@6)m{p{&(X*5r=>BLgRI7K;jg{IC6L7Y=&UZHEds z3$2lu{)gF0{*e!4+q^ffxOHlxzrCZpArqLcG2*>gdUK~A~J zhA8IwpQ%iPFMTksi`S=Vr+m1>ERhirZZ7^MPy3NLn9@7E1|j%|%T>RAKjig3R&iD) zCJ1@G54_AuS!NIpU=;~7oNtuDBze?HxLWjxl0Eclveuirzi!n93#@&*J^o^x1q+RL z(h%J6(~$n5f8>I^B4Z(fssoIJji5QYbYaw{+`^^{WRNm?2>wMQf}70VB!i#v=gzQ= zL^wHz36KbL8KU(_aJo&^3b#!RU3O0lroXJsh4h3*@s*Tu9jWDr(QL79k>d#B{Jy)z zyF?DxF3;f3`BKh$4T5V{HkaGDaM2MP9iGa!ir8^XZ+tV-%_KSqdYL>gP!y41SjB)m zj>UX!jkt0Y=ZS@)pn7csiG2d|d|<`J*7h6Aj=Myh5n~3rBmP-$OTjx*qVyWg*5WYp zl&|7H3HFvd5X+pZZ^fdavDYXUr`dYi{b?$3EqoIO-n~h7mA8kU9jC3HGw3iPBZtE6 zE2b^)XNJB@hu0Hhp9e{v4G23Bmyy-@C}!CgHrGSlpD)aiGf{V0kXnbj1sl?{^DUUP zov~=>to4k>FU!w%^$~`#m9=(;j8LpvlZQj_d}0dUfpuQJ;;g6CY2nFRf|BT4tov1Q z?LK)^_cCk8ZtD*v%E$MCV0TM<0A{Ax!GuZI;iZ)xyxUyeTmhHdI(EKo`k1UF4@dC~ zdOXu3e#OQ4*)LvX(CH~$MP_j9Y^O-&M{>oxZ;{-2Q~hgGK9=e-n$dlGpW^g*FVY#e z{9OlXI%L+<-F5J}7Z@gxvnrrB)_u)NYLCw#9?Zldd`e1TG6QdK+6aFelRG_-CMKs3 z{dR%3*BndlXuy8k04KBZyeX(%R>-}c>THD&5+A0&qW&Hdvrz)6Jl7wl}1F&(q?_{`4tJ&Jmhgsb;!=8aL?D5 ze-OkolgQE~Vgy!~O_4=(1R*D&gI_?%7XJL_h`CP%uqOoJH7?WAjiiJoL$uV@w zxN0W@B}vZFsBk1E3GHGK$k`}_IpMQ9u`KzXY^b$IkfK$dNaS{#4D6aSZMVBcgypG{ zV5Gd`U5xT%B^)wSbWzx7eV+0~2XSjR*C;&BYo*xzpv|+}&4?af9(A1^{H<_XF5QXI zg|ggIMgS$&iyOj=82k49b$Nl|SZu^t{th7n(N;iryyGP&D;rx#IeX1pU)-EGt~feb zi8~Z<*{6>8g@r$q-|sDU%1E1>2~E-ej2UI8xGx9ho+4Nr5e~ zoEbz#0jKJ=YAW;I!->RkrETB3p6KzDxc6)Ja1H02ich0LHM}T=ChDcM^GQ<-6cX3y zLQO5~8Q7-l`RBEwcXQ_6vVS*Ma-n~oxHe%jmL>bqPKFJYE||2=TuWDweN_rjE4{0F zjWzkseZS?M&&#^*w(iV;h$OM^0K~^72j{WAJznYgnfXzW>nfc!4+|c957C)PSNe&V zP#fN>DxE2JGdkU0SN^&#=c$(5hr+!4S=vhzNFI0f{-divg9S)62y;Q2(p0V zfA40;FQgxYU>vR%mzH|cpVJRupVJIrVyTZij^0PkAHc)>B+Gq6n_Gq9Jxer|fkt0$ zg52PjXs1oz{C(qe-hs&{b**Y)uiIbZFFq~Oz4RZK*`ZI?GGfPzEQH9mlL_-@$1hhS=bsl^pTV|AaUs6? zsz*1Kl0x>UEJB4gGa9haMk?8Es&g^cum5&z57a=<*EZiM?IwV5es&Imd)R z*8Cpv&pgQI_rg)&n;zSRKg^jvO*H3Wa7=E8^UY+ zIH!8=uZV8*Ox~*D(F!xF0VE~2&hT_!f%AnVq^L_(jz)aZI0liu%P(Dow6Knz7V!pFICZU(>4kPLVPmv-6T_pWnwz2^*EJ4{TziI4PRe z&@%oXs<8>ogdBPJvWQ`w5KzY?OS}@^M0Vc58w2K`Ri-4{2VX2-X`M37z`KA z9I>2|IeS1ODo`mA*W%_}e|+?D3FF5@qJ$+Dma2!Bku;Aij@txV8n`jw#YUC^$E3l+ z{W4qTQSN7`?3mep=g#2Ja>oEzLc-B{Yiy?*i&Zd^_#_=3qS0Z4fzP=z0Z+rnNw$R7 zJu({>3aFY^BzRKLIhU)KCN&D@Fk25N;j^U*OVoGGOtTltF*+AVrp&Cw*L2Vl# zk6hpxPue{ptg=~|a~D8wy03j}R8Ye;Y$FA9K;kxk@W+@gpsq2TN}r1LHmW1Vi*9IQ z2_X*gF{t27zb!2pS|7vOT6ryAWJW?ittHvsWUSSvM|>^AdZc#b33nLAU5VASG6e_( zK}Z5KAVkzN)2Q|RAehcMJ<{T46BC*m1>e;>pTf4gexWAl z1D;PUQ7<8L7NrvF5_O=A$T223pq>0rZ^gc=fSmQaYzUMe_IjZ>DDw)sexE%cI1HjH z_dffsQqRTbpKc94yhOsfS^~kVGpEu`10=XBsxwM8c})=Tlec<=5M0Av^Imb0NQRfD z3h#Y&SS(VH5-09+#s7q;$w{U-97;X;I3In(ZeQruLx+iGSxCPh|I=E`L_41e0@i82 z>C8rkp^Eh|$%{=^zl@s%hwFdRJ!C*}aRVYea2VSsfTSDoSog7t?gcJ(#)UI4%$=Kw zl?o9&T&&qzokB+V+nx*uy`JeYO9^~oKrn%5IC<+?4E9%d&?8rt^vH|LL{sfMYn|Y5 zRHC}t((}JZJ;u|7Wh|M;VB;i1g*tox&y@4Ga!(S2vnIwWM z1JnvpqnUS6ZwaG%{u~HpFTi_We3DI$o29gfhzQ-znAIa07N-Dt zUn1hh^u)y0P;T`r!W{9Qw+X$@F`?P1_(y^WXL(+lX8vV;EKvw3rB5b^Zrv*C4H~M$ zvRf+|$Y8S><zyf7N3CWpsJ@CGo_7y91%M6Z<@rCUzduhSX6ua=$mGOlR_i-K zbQ_M-UV!_EUlAY)*_smNt1x@~=utkvrws!vA53P}I|_t{rx$wVS@>#~&PkmO>VH6M4Mt@ z@95^&SF_3i2MyTNML-eo*Bh}Dv?1Qi$}cr+)`v@T-FC-oyElgW+w~)O%iX^?Q}iE> z!(@eM4hjoQ^7g;~3GZ%dSU$TG1zvB#?%lhNngMf{Hy>g)oem%b-u}wB+!!fZ-UQ$w zSL&lB+z`;J^cKL&^-q!Lmpmn*yWaXTTC)D+vZkuVP2_dgxqlfT7kL8j^-h4JSh24s z#xwItQ)*m&<_e_*zu!De;o_62eN4DcJeKW3jcuAY|7|n;0GG zOBhXnpCTH3*LF#L4A#jO=hu1O5xgLx)?sXw3~LEqQ&X;r;+K!pUwp2)Q?VfVvmjl9 zQibx~Yq((J#XWvxt*8MxC@Cpvvl*nmSv}r8vhuc|a=VR?PS=55co?6$Lz>T!1~;E0 zXlD_?w#~d|H%D9n*`0_dX?MTrmx>axGQ9#VDd4rWJKzLpL{OSE{1KlzdVx_kApj-RBI2mCvhn%|~!HHJ~jYeeoC`s;9+3n;Ppw@HTkhRmyD ziGLa?7(Bqy4gkb97U!P@#VMiP*67F^1s|5!rc7v964v>^F-IpRpU#%7h%STo4~qB6 zMF=YE-RmI?wN;fo6 zqjE!ne2;etw)W-3Rs28f7EzX#(@!qpKBdHmk;G8~O2}$oZN*ciKc9Vbb^ZAB( zx-2r1knP0mbts0rrxP+B6UAEV9&%OXe@Qph)A?1?43YqE*kN4{PDcY4UVu88* zOz0YGyEYob+E5o!%VK-9g|@;Ta#Qn&(P{s)6? zz&?1DdlN;Ar;ZBDWUI=-@cJ*d{AxrnJA|s8$78y_pD{EE zT@lnD$7BAbfVaNL$sC{Lp3K~5FS3p;C9fIQzB^cUc>djj#ChKzej`SQulk z{h#2_my%WjKd=DeB;5?yMosZLB_DVxmI+qH*qjCLEBBC{o5 zrB~OVL-*K4AEBeFTs2S>Iy;}FO+%PQ+_(zKIB zo1Ir99<^Wjo6ffGGvI|~z5+ijl#i-Y_^JLU;Pu97Wp0r|Xni9CgQN%Q`=3RFco_-- z;MggG3?PZi_8$Cj)(b6#3YFfl%yGZKA7$R~E@e4=5gj{0VS)~{oCy7G^Tmc&Dn5s{mj7V>VC z^8^hyWLq_ou#>(3Fxs!4)1)@bQC-IXc9fgUpLW#9%TfsT@E>{81e27ENm=A)u6!_6 z{GNUSI>XIt@HrnS%CBOq)&<_Q%XCUdKN${uJLiGF6dt5V#^;uV>zEesm68iy$DuqZ3C84?X^FFjyiX zKr}t{TY!Yl5uZb}qIvN0W5bjsQ>Eqv^KTVr_>ChV2XJw&-1akVH=a49w+Q-vw`Sw( zKT0$ig8pnK(-G8~AMLaSHq?^6Wy2d)l+R8XH*{{4pKb8eSvM0s1EXyQ(pQ_qL3m5K zfcQvsqsGvCzN~&McNoD=%zCoDBWD}=azlsOmd_kBME>*he2NFob4Y^exzWR_SChWV zYy)`f!4bbRCPEf=efr;@>`e>me11OGpT8H3L3?>|?C*$Cs`eLq>X!RFU-2Quvu;Tw z{(XtzH|jw}jHbJ_YX;Lu%Wls`$A$i{RzK9JBQd6ZZmZWX>YpoIE>9`I25cM;%u%tthh5ZsUgYqX)_oxm+(kr;SN?f=Y{c z4a#x>{5~_BwYJKh>*{Q~Xoh`+aEs`drToP0`MmB_gR|VY*eHQxM!4S&`NIo<}zn*Ryb+ ze|&^0#k8xtcpEV2f=>;b5%NGE-L(csaPstrroxbDhfkcv;(8479gs>+!U|3eMy_*B z-Y`%X;S69OAqlW*MvV**Q-PRra|L{F1@izwFexWs-1Cqy4t9rRJ2?SlCgrt2kF#YL z8{G}PuC!(qM5EE-tt>oK`MZWA2Mu^%f`-dj8kPQ>U7Ln=WU|!jLzKc)hVUpqx$O(p zVh#^sifTrRmyb{rl9A;`6zjAV;J(>w4OQ6)jn><2I+uH4KtV@e;)#=e7!1OGngqb> zI%+R@3hd;Ulx`lXJkW2xksLRgeqbyA@u8i3hGg~n^0ob3bGiXhG9HRNj5#Y z;}_;sD$C0uUi;gK$1~bug{B4%d?!EOM>=1h8S}}13Xlx_T{3 z56565xz^OV23QzUuVEFR*2#EG zf|J|4YEWF|bRN(;9D8=TE=+0OT1ghvw%oBmoV2m1r-Ot>r#D}nQjOL>xu1zQkt`eO z)%9AH2r7`@t`dKE+Tkp3s=nhlw&m%uvLRetcDAi%mFpBo|A+kb$su&7tgCLp8>5#( zhF{EKqAjb)&JJ*1m7J3;eD;OzbbC7TtzM`kM#*Yld7PyyU?hlC@O<4_!^$wcOm^z< z=Pr~ltps*x|ClUTBuG^{V2NW;kzvUi(+*t0G(aUtSKjzAUiS8ex288o<1@ALXi;u= z{FevdCNA?Qw&*P`L=%g53_oWA7Cc0}kfEtVIyU;qsmm=cV?8jQyuxsWL+8858i z8;4?t`5%_rxT$6X?7l%$a*)V7EM%Nt8tXh&IJNF8PBhCWEcRGaRbWLGhuuHCfq|q1 zq-~*P-?!W{2Z@k5L-?&np^Zwm3#D&VHCf-MrmFHj!H&QskETo1B4opE?Xj&LvXw{u zl}XTRAS*;|PZH*B|AO`e%)<%#rO4pr+Jpo0rqRw#7pH@i< z3c+gGh2LIRH+OVc*;vMkR{)-?RjXKSx%>Q0&-_;)x&W}uV}urek$%6FL!6934u@5i z72hmeK9@wS&?)bb9s&8K@$=@2>(#b7?V0JL-Eu&H)yU7lJAi%NacS{KoEk;Y0pVkw zfOb6Bb*sA1kBNiZo1^>&kRj_}*O*FQVo>1Z)_l zBiwfsK*wJCqqENGV!#w@oZ|Bz!iRGY%>Z%UV*c0-{?TX8e94rOyzm3JIKTJduG{_s3=!s z%}w3jMq@b;=$@+pONIT>Lq2s>GX-Ydw`RVj(I#tPbDPYmCNsFKt2%q(6VsCv$3L94 zjZmO)eM!h4OV0H~w?%P=Zaz3tJS@0eUNNoQcB^p3gz;z;po=iQ+wfa3CniT13Y z%!hDl#$pC};x2)@iH8zeGmg3@hW}#nS1;BeR~}n_C!>XtwUr|z#B@)Q{wza7F2Fu4 zyZ$WHv&dQQPaWl7+_^vqtlx27dzN=$2YK7>zl-@;?5mftRT!9ka>^^idob5-E5qs zq4WUx(@ySp^NR#~UneGZ9w{GV*AS&7)dVc0$=fOzVbtcubDI|j!+!8?OZ9>!8o%G} zL8%om+Q2eO!|W%0g7*sDQW4J0msqny1qJs>$yPz;*yFinc(B-h!Ydv(KPqhQN6S#VbVcxZ{=3j4VWqoT3mr0mrJzDaz2 z#HrbXP}^Suidynbcjxn*ntePSHd)L(2daoz z+~vyBypx}tXMF0OrIZG*MsW5vuoWZq7*c;c9JC>~)~;pp%Rb=2f8Wz=buy#ao=5K; zgxgA)Vxpj6Nk9%_4b@nx;A?q)Jdt}|^+Y{%d8)=SWuzC=%&u)(I-X2e41nzA_nWQ> zyMCAja^@-s^Eka^o!{6?mLNHtZ!(=iq>j5N`*LLXYydfBP zCO(O;bIpu=iD9aPMLs>eGwl`BiE_FiPl-U%%r=mBha!tz_ZxR+sBu7azs|dxd)Wxa zK}*<`)_|L*VvIHTUA6s~8BsEIysyu#uV-yyB$d*ir(iSA5cNvm2kg8OVPX-z3N8`@ z1Qv=}L^xW2(rmKo1ASC%NNN~~!A$)`wHW$KJ9ioOEU+n zgxX?FsDpfqW{YkKl^L+b8cz@=MP~LI>203h?&E`Xlww!H(Nb`liMS2oy$sZn`6=Ml z@hK9J0&}jPlNrVDVSyYIIU!Y>cy-e-#rO_`3mnAM|ALe>^BR* z6Q8Hj1CNCHm0!(*Jw_$YVhAqDI_{7VD<-;tpC4p1JSo6N3Yq;H!GOipThtD2bN&g&Q%RN-UPV1$g*J z%7uNMl!bw>@nTU&9W{Ru(5YEn67jp+Ri?o zLjBmU%EzokY~+KydgICi7E*Jkap%>sapSWA-f8(Oh7bL$wx!z`4f~T8YsM+bZb!ur zY~Mk_y)Z)*rBNsyQY`JEokx5SMnx(hcSyo+cz~}FVck0tF-4!96Pd3OtBB*kez}$W zdD^aBIaiFi%&wnqtg7yzaqyA63P#i;5UWS`9%D$1TwwzQ%1BRdr8;QqGI z2YiVxj`ePWOTJ^}1|I>7@d3|yD)4?(@RvEf)WD5Io5J~^hQSX**Mq(S-~n)|BvM+C zrOP~VR1gYwQ=JW(dso;u(VdV(HsZG|_i5Je%=L9^bx(isJ^af~C^zZRm=swu%(MKu z?eB8qS28fyQt5Kof=2^o3&dSOQ&s}ZP)E~%15JXGn52&jFdp!MqbC_lylvJ?lr(6Y z{hn2iG1*}|(j#NwThBW7_(a39EE;%$qwYoO6NSgzB|XD~-%*WA z;YTmJ<3dvZ@&uEv9gw+K9*ktW%F<`Gys@&-eHDx+X5u7I4@y#d@?@fz^jC zN<;l|`W>hzqYU0K_u^RPviSQR_43Acc%Sb;h&GK`G#e6a(PSSDlwz@?VEYZn6~^am zga>zHuDB@=bQQg^2ao}hiSGQyFt(UscyWTal4>Pq?u93$fv7j7k5C{|2o7~^T=m16 z(f?Z^wNy)>B$U~5qt_#0@j zzo$v4t{MB>#O$>fiPU)bF$OX%VO%ZLK3@9ylyx-ItIy9Ldg3D1Q$msz&E&HUbyf&> zRJ1aKPQmXYCnQe3x{E2`unvwDb~7(zzRF~xta+OftHQ*j@K^hVdA4CUSz&JcN@+<6 z4^3F6eDd(-Aa}6tF7U2%T=q*nA(V`PGCvHb8{uP*VZo1v?YF!%jVbg8C8wO3WCA;$ zbRYD)WW4cD?V+;bSj}T1(q84N?jq)9bxJw!j7*=aM2Du?Mv{1t%;y;rY5SXa zh0rfXiePA^u&}>IBKahUX+Y^b$UXc)HKu&KVX;1{o0`lSlb7BXwTbZ>g~|6|DnH^v zjN^8+xdD5eL}ByyvX;7*4YX3q?DdaJsW6cadZQv38GoFMwD?0An5$-0fY)n)vZLpi z3BAA(XXe;JXsMn2b{8|(b{>@*Rmm{qm&cJqShiycCP$b%)PdH{mh~3vS|s zzG1pcBRIC+h!yLletB|1&`_C9hn`)6%FD$zYk}Fe`m)c@8itK5qH1tYw=1e~yF-u0 z1YSB|?+JBq*I^xM0Q+%I-Z0hyaGaB8Wh>n9Q8ascZVAC}Nqp85GX(BM`XdW>>_d|Z zk=cNCY*F(iuC25Jcfre>ibs+bq#_OVi|+IKH;CToAc_x%SrSGtNdONQ{Y&w3H0o9ug^6KMl-G@fB>yPH-u6k26Y7+svE%1V8j6 z#e(w?cYN%x5l$%O*0ltBtsFg@tWO* zej5J5iMPJ;X7}7@U^}SUZnQ$ySa)or^^*ea#_r<&5RGA9FTH%BsFTsWH8JGOxkLN? z!;}?LiVTYl`qw`zdT$Tc{c9xG8gXm)z65+|$6Un-3_W)KmB%`$c)@ZZ$@dfN4W3fH zis2}h0Fm^%p6xW-u9r1A_2G?|01CDlc1U)BqNrj|-4*=}Y2S=}d9w;TGxKA?IG_85 z2RB2W)Nez}F*iqQC&mUmZbPog;rFBpBX}3k%17@QY6nfs2Lh$B@`3wY~PEq6| z5SiShqsLI;N8dLxQlMbhE#;%)s3iy<6lvz8d2n+~q(8`fOsh}R=r*P^0XPUNCzuL{ zeluFxQ!wx&39&H^I5n>S(^_B)yfKlk6)MG|p(GL3X^(ijl4I@B!s`p3^;oWUn61Js z_ahR$ClT5(d`zSmyg)Nlm@TaQ!j72oXg?5;CyEH%0d#*|UlUbF(+1)#1~swh+6E{p z*gbK*$5`x@Xb-d2WfIeC2hnxUwhBfXku?A?M%$<~cXi`_JpiK^l-)wyF2PO z+recI?Ec$6$}Pyd^Aw7sb@(JElo-Hx&P|;xdnA=m6k0S51Ji816^OcTxKjK&TjGlY z$1lt-!%<5u{97^<;{)^%-zb`(^(H`K=kO$)6ozTJ_oPtUu4GyVot)A5rl$`w#=@sW!HG zm1Ib#gBntMqLv8{7lK2igWAE(?K#$~#2d+Lt+9hbF>Fr;Uqf4l#SpvMdWk`7QBB*5 zuPyLU+)8nqJ-L(P()qFVdA@-J8j3PW83b_+=$x(tf!+AND9|Q z0Jh5mmr%S|m~b5c;KQQXhI1a_nshuN<#NOh^J;k?Z88goFr)awCx@r3k?jMR=U~mN zshmpFDvx}gLAhEDC?B#{W=u%d&}kvvjg&mJ#SWhc?1R==9vG+ZH``#eQCtl*!pvp5L2np_<679nCGLtWv<=FDH7CmW^ z%nSqElogB`6=w?C?G*XrN*RjX1pG0R4;nqF#_c(VYb~5#T=)4 zPf54Uk?cPeT`VV{#1%KUZ<~1*&m}&aGzE83-hDT1mgQ^vN+ep&YEcn5%HF|gA=j;y zzO*cxvn7V0#rl~DViptyhZPoF(4)90YOmVf3mDZ}-NU6xE?nBkz$5RZ{-|31py#QV ziPOQId9-c|#p*u4mkAG6pP+*}sw;U#3$u96TQSI2Nj2x!r25c8r!7X>L&uIsWW%YM z(w=GbUeaHOg)(%HdtQf$S>S^kIoq+YwtF9oouP#X_QSj^aGYaO)M6VM3AnG-;j75N@t8@TEK0)a3hm(5hD^p zmJW%V&jPloE>As4j1(GP?VIx^*OV^K3~4WU>yU!oZEF{QD+oStz$0OeNC`G14_UrA z)O#_Rt}`v{#K9nVch-6x>2@HDg??a*W1bjX(Wl);9K2~e%9@%$jd86qL)*)^N?f8k zTVHK=QPJx0=e1WQP3J>Vn7dKzlak-Z81)zQ9*dC)Ej0^U9o~VvQ#ILVyM`YFSLvdN zk&MG;*f_52&tM{<*!aZhT&2!a@UZ1jPB-~x`t&7CIO#a{Wf?{pBSk73Q3LLDTXjr) zj74Sek#fe!DM@$ykvt>92aa!X)2^c>-9jMp**fT0h5B5#xU9u3w7K20+f(UhTIFZN zlf6nl9`u7ODkufVc-x8<{{0Ne?32Bx zkG@QdR4%mSADqUtE7mwXsx&pG584*nsQEb7WLfLQ8e{&U{gn&Z!p&i{NJK0nwHAtV zOJ;z;6VLPL2SALrE|KT-<+Hz!F=Y;&l$IHgyzoVjb8kC{ZN5Ws~l?JnJnm(TxH^kM0_4XA$0D{1;E zo;&>yDYZ~3woWK6^N0S(Sxl|W4NgSVOL4ZffdB#jr07kt5##$YFr*4dVz7kRu4JT zL>xaf8VA{vxV{x>HY9iyClx93gL^_jRTZB|6Gc8?nu6W(phXBygX6uCFkP57b@vt8&P=t!d* zy^n8+%y7vNhEQywYdFlMDu&vgN8Oq+;>@FoE0`UK&WW}NOrE+hl843f9qj48#1G+h zpsALV?%@4R^VN8hDGJuY7SS02o%M+>vQk&g8H(BR`4Afw64idE;PL@h-i46RV=%&D z>!tZMptVWG{>8`qP>%#2N2^w!E_$3fVUC`TLZ#AZ!-@+ zB<)deSzIkyZN1r|d~UR~;)S2RVy;3Zn=+07X>G`mZEclNj;jJOlp+#6tfCkmL1EeY zq%8ftE%~$}8DgbALE4vb`5ijNI2k#eNYB02;2Jc z6vC@iSRT^7Vjx3Am$(fQoW0Gosa+$XA6>?s@nsS>L3*!)VGEAe$y{*TZ+N9!vMwGQ(LT~MZMOtcH1dtKLhqa%; zUkmN)-V4Wro)(2h#uqwft%$VMEzm8>1_`+ve1{394{kl6*OyVVUM!kTl4y0!aR5Y5 zMln-ck-o8(<+^ngS?U8-)HqsZvHBXd6ingn+Y(ABwr{C0!1+;IGn&TvbRbsC93VZ} zaD{bh#~bO8r0V)cRlY>Mh6oc_UY7Q1p~*~$na&B838n7A=NB-gj|NJRzwlNYK%-e# zBV1iPotlmp2J{3O=4ueunf=`rf`qlljf?SEs;)t=t}QANb}Xvo%;8tdB#nCX7(B_v zaS=Z>{!(RX0{DDl2mmo7b%>BEPZ#rB&-dm)W5kM&eRr#REEo;btcix;iI>N!B7{lz ziAiz~r;@=eHWzeF77PEvMY0lPDrVk6@YHrt(Qt6~rTU_oy1sw04L7MK4L50mMQh2P zG%24-%azhSGzMRS36J`M%%ccwJt#;*2KarP9F8RG=w7LNJ!wlV_U+#*qI#t!KaFG* zr0H8ZoTo{V1+Th7v!t}zP43I(wg-JXm`jGtiN61rMI-jZDcVhtZGs}K#p#uwk65Py z?u<{P@!+h_oLbFhTy#CntPfp7#trsu<2{XFuW-oEOpAYs7HTw?k6ix_8s)r~oWN^o zfF}jmisv1eq5xQ~r6d`aCoaUV zn(mxXa?CoWb-=+6{49X4urPdPE1JI{!m)H>k>T(3+@BL!1 z&2W<0se;dS8O&xHfnq?(**hgv%95AB76ebg%n>ViXVu%O+^OHx*q?p6oOOmeE5{s zzLBpF71g#9x_E|V!S>VnZjG>ZW)fu3wuqp%c5YG%UFlxR&SV&VO4 zs$La11~zt`i$AIuI@i$vS6eJ6{k3QvtA7;@W)uFG!RxcoBc#WWP{2+4v&Bj989KpM zuRy#f)&YfooJZ(uVWA-*ukQ~Qjqm=BES#=5PkEi02b|Vu9l)c zz?@GioM#a?wdU$x1T>RN<+}h$jc8gadqOQfVVC`iARWh|Z@3d)R#OvgsPPFO`E}ut z)>iZ5Uw7u_dh6_LJYE1qt0Rwqfs%DSyZSB=s;pjk6glCj!PyR;zbQE&B|q6Cb~fZs z@5)KusMa7qv>(9_@>SZX*icR(mKpq`^Lw)7y7;wUIP-N>!-?Ia|8{wN_%QY%aZzYk z!Makp%)+Dc|3Z6l<}Gp3{CMqOYx=Y2`9p^OWqkk~?}Aro!qHQ~{`3pwOjn^SXIw$* ze|PJECO^-All%CMk{V-0&P^jzz<=LZkO1q-M!orUp{#mIdrx#fvhu%dq!nyi3g)=V zg2OkTiMn7Ns?kaR=BJ$01u#3c?)nFNGMYQp|Ah`gC`s`y*=IHyzo=s3a|2lT-}8o0 z_Asxi1HEtZw~v{8?Rg^(8P&h+!;l4TSyEBF@9*q3stdwU-*j0n@Gfdt{+(rQ%YkdQ z#KDJz5N?vD^TySABEt9oOa1c3ecQRQEbqP`#kyMlkTE+9m7+ZH|JTMT=g(+M>d!Oc zg2k5m??|whU|UVu8fDD+Jq6#re-oTG1MGIO^ z>3)ZX{yTGgO>s*Mj;Oc5aL6L!mXixC&zJu@WdYG0^msO;Fv7vqeSY1IIxfrq9Ty?K z(__nIW(7VF{yvL6noXh0AR+kg=-QBjAt9xAJq`GIX*iGmI|~%#jUm~Ezo5fY%FzFU zvz5Y=n3TwY^B2qmbN$yW+Bjymsc1q`JSFzLjs~uO{eWB>1r*!P{mfZc|6`bO6b@t< zY)wSj2rvA#?zXJNl^RZF{<6wAt?L(5&-@_=O;*bg{JSKqzb6mInO`orBdbjaL$%?H zL;n}my)REgj+X{CSNVpXq|{XP`0tFqOC%6C7Q!f4!^OF4FaBLW*dNpu74zI3 zlA)ybztHdRc~O2l!EBTC{_lr$RFnSKspu1;?hoCVh(bOjSd^fQvGV_Y{eb*VDP`D5 z&&u!E0A8#uQXD0M`j@Wda+esu)^QL7qBh>b*7n%Kp)P{2Bvlm@e4PL9cdOn}**XaT z^=S=N+D|nhS;orG(=PudQsS`U0k`1mZ6{))Z6-ma0sAX zt#11F;jq|H3w+?7ep9iced6~H@~@3pyeVyofJ|_4riLhP$ykUcSZ1FC`@f!Gf)l(p zV@dWr7x>TS+Pcjrc`)aHmk1RO*LT(a>eu;bs(=5#56`visRSq}G24>H=&MqxfK&es zU#K{PZGbR7_Yci~Nm(IKVXXexL8RW_uTm?r{O?2e7;KqnTwH-<$O#7WQyMare+`y1 zRFS~LL=K91HF#jIHYoG285=>s0k~xeA$4JvBOm{<^_e+F(*@Ge+rFB2$-(7MM#;)vmiHPOAS;nhe>1Mu(CbeJ7!-G9s!`9GwA`Pia zaI>1?2I%fUG67zT|00ZW-Jt4VwXKF`3T4e;oYp+q#D@fJ1X-L3Y*1ZTOWZqxl?ti< z(tg@Na6ob4w46}|53G9h=(Rk=t};(H!-5+_BAW#@dY)AD-YtJ49EAn%&9uu+aHIHd zn5+}=CF5HVFmEy6p-0K?BF#iViL434_DSZ?s`%u{I(3~Ou?fA>?+`Vk#Dv^}|KfMR zY7`7JByX&4EHy5=JzA}!+4MLRgg(5Z)`C)Yvga-7^(3dNu~LcrTcosjWcVJzN0nwluKT}{D&3xe z?4>PV5qV0my{LCU+sT%vjrU87ht>Lvo{Pl4S+}h-cKa(PSYa)4z78JE#hERh{OC0_ z=Is)>Zp_b-M^@^C3IDox8%XQv6vo&JdaOE%vxa8VoAbWrBg3@x*13zar<7_;om_bX z_ulbyKmKo$gyIMoyR>XYd_kgs+xQ`l;uxtSz7KoI8h(PO8p`tb4lcH_Eql;_c!Gnv zI$wOz(cXkCNBWz^#%kBMXYG(mfjT%os2jh_@+6A2ue6HcaIbov;w^#ar6|6r*YPV{ zxsrg}?`4Frw=2kX7nl&!$*j@|^Rp|2&;W$D6nuXR1D=)FC$C zEu`wATy}66XByvOq1oRv5iFd}`>pfWZ^e|I{Ct-SgNY=Cgzzo(<0K1~Uhf%}AJ^yp zITr;f)tQVFc2FP~5HtfM-5IZbUO1(G%uc=eA;$sKAwsx?B;&9w+qnxhI6%oy)+xwU zA;pMySNtp@sGIhXA96JCVyl?KXv%v-fAiu8WsXbhIR5<1xF7k=b$y}LW9`Jwy$B;_ zxA^*_58p4XRA+vmJHw-~E!f2+jiMu2e7$x)jvn@Ej&7!vjaK?Ww?=b$JvV)*WKK7h zzuI3vC5$7~3t>wjWtM;k{>)PO@+VypDA6rgm*MkJD%n33C|Jvw(#?4KW^;u59e0HX zPIvtJejf%JJ{y0%~t zk1zyDh~^9F)J>pJZ+#h}O2kJsV8I>K36x&u>OV+&?WpoU|C+ry^64n?9jQ$HF^<~Q`FHx0Pt7_LQ7E56i zkIYo@4Na6~j$m*6eIfiNm2owDGYZqMcJunBYKEI*sfV_g=hnI;SqPnk(FYYmgrvlI zg$MYABoC77+5~&Dp7I%L`kn3Pem(KH0+BRr7Qsx*qywK8>3`@c`(7IZ7fyKsGmMwx zH7G?`>6tpbq7LFq9#sdXJ$0}mSV^JsnME|e#!Fciph4N-J#rkXjPLB8=>~4cjBsfU z0|ObT9eavyoK4$Z*Ep^TPConU!ut8ymx!ljNOJ*h*r5>wNiNO@2oe7hT3D;IOqUcN(P~Q)@ zz`ParqkiBtALCH))5Pm{4W=)5PZpe~Q(JWv`JhPal+jfkLqIIS?pw)jiQr3PCsTh& zWb9|On8oC;e87~R(PKV}yo-zzVuaV~1u01osSm#Tg#Nq!onz%`!>1-AA^nu~;!jfl zy9M#0FfB8%l8REI4Eu-?A>&GRYO1c4LC7*-XqSz*_dSwHf8^Zsr#?MlPSB7UVN87X z7cR}AlOI`tD|h#BeJ5{EGm5qGCWP3v<52T}R5@JRO-IhO2#Y@U z7N}dHP*YXSHZbvxX7XLY^j+KhGhCJ-^@iE~U_2{uxJQLKA8$V}XwG-}u}O{P^{X@? zbF2n#_x7>@l1;{!V?~OUi8`XYCA25Ilm-z2=?>`>MrwxcaHP9)06{oOA^fY9| zd0LCCp01ElYT%%|gFu&lGR1=$%~HjQ4{bUF5*f(eW4<8y6@K zUUl@QLUXAmpXp~izVsxtrXTgHOhLl-Q+z2TZI$8UtD12F8${pQirC=A@UfhG zw0h(Do~B^CcGw6uXEJXwPl|a5>i+7&JpJrsyCF#43sM6iUZi~TE&jfkzPs!@{NAq+V-*FpN`##a-Gfk1NX*r+RU%)#3 zhd#PBd$ttIbQ&epuMVwW(vyLr`})3HT#?e84UT&%HGZ48=*h4{O_`FL)Fhtx?K6_g z%7x0!2e_PF*E{-!Qo!@6$?wM480)$Tup%M>`^XQ!IpB)%WG5GD6r3j;K`i^0Pbetq zeud6Yqedf@5|hkfS4kRKl6_OzW@^6^xhWKERz{G2AIk_$Go@c_m~-WiUNR4FjYTpP z;U2V@N7tja4q$i7Uj6!f4;L0f+g=pI!M?sv+tAybI&gUkn!v1cW0nGH475WcpeN3sK11bM`V62P-XA}no;mpft0Dk4uW`Suu zhzhcc->-s1;@WSvaP1cxu({1zQ^OK$)I!Z%i6<`^rToIoo}N65rIpFvDY#977u9Mf2n&lQlU^E;sJE}P@ZD&06Efq@s8ZC~nlZTJE|&G^ovJjx zI*H7It&G&uPLHQ<lxre|J7nwntVo zLz$Y2p{VPdKEdLx+i0>)Q?ClfH|eiM7u*G}%2CAhlCM<~4Y17v&&BB_yc2&VjRxAx zi#3e4o%ifGW|f;4^AJjL9>l}qkrTDGYIF*=ZRP83T%Z&luk*0e?{qxgAN5~%JnNRa z_&C(U4)8By%L0fm!pv}cSxa5{I5<-wP80zv5Zt9(;)O4DHZPcYY_V?+16IThBn`~( zDZ-z?0P`&DhX2j|H28*q055kq#YZINNxkjZ7jA9a-0^WK!mZ1-4MQ%bpH${=3SOvZ z9Q){hcbSeW%UHp`v9)cUtsvMt%RJ2~APO@!}iQuA!mcdvoLC z1=)WOtjz)wf-`P2PQ)qzv0;V`5OW2ZKHDEcVNA7cEC$py{S`9MYtZ2`^;z)hd5C=s zVb&Nxp-^^LMm1LAum)h(0f(5T3x%;#@Zm9|#yOp_IbzrqcD_3cF)bLldSzQRyq4VF znCcP622>^awR0d<^FLtj_>oCaq4kcdWP4ofizgNuqyZ0-d@q9$b#i zl?ntAnLmt*qCdu2Ef>mJNXSxed(=T`!d?ogP={Dh%{9ysrC1CLU|w-Y#$>HffMvPB zz!r41K5wbPWrd&28o`|GPUHbLVk^+GD)kIcXnOT5{NbUyn7;W)Vf9a@^xGR3Pp1(FlO9G4&2tKRryq)Poobn3|A$d z0S-)J_hgbu=FI>q@I=84)_;_gdk?30H&}#vCqh>Fr*nkbw^z$n!U|^eZ>``05`0?5YQ0a)eDA!(EFB40voCDoaG||2`x<-i_5X&L@~Hs zEC|`HD5gvAfuwm)jfk}6rwM9n-;Wn-LJ%1)d74FUmaVr&8|e3BJfmP_k_DH5Yp(UN zE>Z|MXn!FN2WMOLi4R7Tf_Ngu@oi$rtkPy`$@9BNvnwg5A(kaNO{B1A>{nyG%E&#R zMY+y{rpRt}6RU@&L2J6F!z~Ag5=+@H8<4(|)K6SDG#@4!^Hj_PBEG6#l2p=#~#wChk&-r$JRxLAMzIN-hC{X(;uQfb4RhGx6wi%q%5ivcpwvYqdN zXD?~Wx~Sr?o!f@#c~-qVo5BQbJ~`A`k2n`oyy5$w%!)xry{@?EE4QK4^IvN#z|Odg zk@prvP&iolpq90aqc3N}sSiUr^HCQ#M20Qz?+iQKQ76bxS{W@kOgY zreW`oHZ7}3e=>&x)CvTd?AI;vueYBlQv%WOTVbhZ3Pp#7)moJ-Awim|@N=g*?GcA6 zxIvw0rn>NY6n67y zPq0hYrq7$FoU~q9B_R`2?WbDB&)WAT%5?fj()7MHU)%S;8OfD6+XnO2A51y@Nkx!& zwXe9BL0UxOXZ4kU$bh=_)76TcEt+~WE3147z04{%Vs@2`^lv)7)0F|D#ek>UFqNC8 zN!&thvd`{BG!P@D8m+dZTc(vjQ+jf>);Qa$0d^PINI*jI{mtfL@>vSlrG$at--_RI zhcZLyO58>bZ(~^d+nXdbA9TcAPZb#Zcm?ZgM|Ek7b4-P2ep<+55Wq)MoA3;o{HtEU z7&YNk3Jg*7n!3T?<)4?{TY9N9r#!H(_V%u{lpZBl*!E$#Q4gZPT1MvseGwRT70FEdef&zu#0V6CTxLR{7Vnq!124Y2AN>t#t2J*m%4&ck+_{ zd!M1lVR)#zt}kO8JqE*8ssL+TsEEfw-3s*QT*3fuNd@iONv0vQZlR0h5sPWzE>4Qx zsL2Lc1cpQ+<|NC4PmpNxXC3Q+m=Mo6nH-HZ9)^E!TZ0u^h)q^&Yt7KHuo6aw)e!@{ z=UTNm`)XlAJj;)YC=NLiA`@VLG}BH^GxR(Wl$XI$W)36?A^q~N!rFAPfQ*4Y-7^QL0;Ar+3n+fQ*s z)=sDhcpTnlQefr1{Y+66{&>FEpzMzSz`TLIW1+~@ZTP3NR4MALjmLzqD=WHS23d1D zG+0&Yav!`7W;rS-#-zFVGWh8Y0X}hlHz)-h0Qxe5J{|&2^kiy$`>zs{mg$IgWkU3L z%r)jNo~F%3oTQMR^nVcj9m3b@_2JNV)xWjG!UGR07_<99hP`+^ZN!s4uOf3P@-Y9` zv@YLgQTF_&4-sb{p&*h#oc*f#-f+mp4`}Mw3jXDZptt}!g{%^2X7?O!XC@wN%+H^) z&LKd;2%~jqw5}q(q-)6aVj$sTxf5h6)eyYv{WcH40|DieetW)@tc~Bh;)-xgU+~@A zx(BiTEBMW2Abpy^IVVrodknJ=%ogo8>&6TM(8FB0(+F|tYzm@y(oIG{>6yuw@;0;v zx5cwPv)MdInA@;*#zgIOjcUAPXH8$r$E#Hkas5#m*dP~u_k@S57aLv7({FbiJ=EtZ zi5vCC)iXqDp!+$9`yyw1ph0)ES&JHd7b9Z zcQ+Q^xn?NgExI#a@~tKUplzlN5r^)1Soj=`C6VllYPat(YaeBiJf*Ogj^bZW6c=M7 zD?V1^^ib!teABsE@mmhf9&=t#$=PW*r9ogsEnWGhkZ>5UQm}33QF7K^d2Q{yJ zk1HGLWzf+RY6{iycJ!AQL8r`0DvHQ|013kOiU02dn%x(%G!TCynwDELzbmN!-;T&j zQ6cnGduZqNBON#x>VLdd41|&4qvvE5oYh_0&nR`<3SV;Th7|*b5Q!v= zBb^gOZ96>Mvz55FKmfVtuLy%*hg?j><2Wmu!*ebhx6?qVIuOYI8YMqo zc6W6lp($o+`1tseTtgukn48J8F8PSN>llq3>FyoE?J<3@NkBV}vwUqc;gMZ4;T0Jo zkNwpBjwP@RS2FtuZ5fyn;=3##VgCHGLF|ITwRx!){)o@iMMKo24<<^cMp>>`QP_wI zjtRcEsJdq$0U=D{~fy?dBLI%%fk0ZYX$ilx``(Bw{^s6S z5Vng+HmDE(^66WFp|jY8_4p&)nTK@0DGOx1>ms2|OH20psM&7ffiKN6XiLLivdz!( z!yk?|JJJ3_VXTGE>!5}Am5ss}5i?koYM;EXzs=$|^x`*^E@d=3u&1YC~xRu3Bw zLdCs5jn)^`Ap?A@tP#>Te<3s?o~aPUP{4z_5Fl!ky!ZYn)tBiO=XLIjz>Yqh?M&4? zxVf8ixk_E?nX7T4Jve}kNDu4_qJz^&alNo zEvfM&r#@U9F|uqHWi!$>rmOF{e3E*%mO^)Uf8&DCzB*pB{e{bT=#CD)yyUN)+x5I0 zWiHcckX&%v$ZC~hCiFWt?4b3|B{eS7mRn<^ z+pCXjt#9h=*It(ClnMYL>Bs(VZf@(m+1f>phH7dcxbgKFd~5qhQQ$Rx<-(89;A<$Q zl9JL`yv(g4XL8ljKODq;ukKlDdpC$bj9Sn)aSCUDV!R! z!+dV0%1$8SJR_UZ;G}ZU_9_mu!RG#$bFxf3W_$XZg0RD6Vd#730;Mvy`@5S14v`nh z-PjK^nPoqNAL!9O!g{ij@IY}2SlpXiHaU61JJVw!!MC?FJG(*a&_p9arhZ+j{!! z_Ur-$$h+CLP_Vl^DrFf>kNfMd3*JkcePlf~#l>~yzobj@VpT4KEo;W=jdfZ-L-7x!N3+*N)R2-wBJMuVIf2z6GN^sw`SuA}Z z#Q%HUIR(kGj^0Tl1#Jdu_q%pGp~(s*-$6P)p+UAUVCy_W$T_vJmUMJFbGC z2^Ml`AB(B#6`mfQh%X);SL2%~HT+5=>iC*bJ-rKI>OBjc_-+TeBPQ{z{Ayq}Rl^dK z1BR=)xirYE55qaDHs!6&d_xz5Pt=?9K~$7Rn@2cuYsAn0D!!qpkkxCyIa0CDKVK;A zgigptr#LiUH<=0sEN(B5;QdZWsiJ=9L=dD5kvl#ko3$*XXDH6mx zmOfeT=U3OGp6RUG^`hQ>F%v0>ECr53nVzRZqNrSl0Mmi6Z>XWh(P8>mehb6fKU{ZR zaT~z-uZ_2>V=2|sU&;aAcU~lI$Nil1Csi7;>0Gupx2I%;^)M=K9T=@rrSbQ|=K5_d zMEk7)-jvR?tc&sQVI?xhVCiKoimcst7OlfM5q^ROKr>r<(tXwEj#Id_=gA%m7CmJt z>^WciUtnm#hxu-7nsk~ZjAs8`l!t-9k=-zDzSK?&Q)U%Y13o`!xh(XXUR!gM(5u%I z)CJ+%XEefttXI|p$=qC6OUIbi!{j3R6j(%IKX;}oLCovk3EmHG>}N_6G_fL=i6t+B z5Awrlu#jQhF@vo8av6E%g`Y#ndyVVWD7@2*LOBl{10@OdyvfQMaC3e&MSYggw8X&9 zOM;k`;v=XBsu}K<5n5mUFt=|$(>gpcZeKUa!A-T7}zYBdW(4R1lT`l%YK!L8tU}dc+1Q_i4{r-ob&E`6-b79sy zq_)Awp4Q7}3&M`iG9rCxbC(C;;TDA`<=Do>bmz0=LqP#ewqGpFpq zp-hFc5tAFy7mmzBQ{yC0OQ738PI{^%S*bY5s#~583SW^APh4Wq6`m=>mRln?N)?@t zY#k&wG>O&v%4g?5f3)^+!(AUVEK;F zDJ`r1EBpj(%WnCB@3wX|5vYolu9x4dU*AG>^eQFaw^&NK6Fi6TrA@uq_-`&JmxP)1 zl8(}Fmq)0=Q12p-UnpKF{D^TAmS_?BZu#+Q3!BwIJ)wrBQH-aBXbcObn*9!%n(0XG zU%R{=xR+dpE1xTwGqB9QFMq(3Wg9g?r9gEg^WldT*NI^=3wPpDnBcmfc=B?~o$h2bZdc$|0~c%0+uaMQ?bmCkgmgts_8uT0&5r-m&l{BP(5_=QODasCZh{_n2; zE=UNX@BoMmT!#bA;{WDou<}Bp{x{L|-^2gkeGO~u9%CtRs$S>HqV@ssQB~4V{3dS^ F_J4OMIh_Ci literal 0 HcmV?d00001 From 831af7a385af12fcc1ba74337a96f9bf728f061d Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Tue, 29 Oct 2024 14:46:07 +0000 Subject: [PATCH 40/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md Added image reference --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index b8b7546c6d..a1033f0c59 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -8,7 +8,7 @@ tags: - Artificial Intelligence summary: In the fast-changing world of digital technology, GenAI systems have emerged as revolutionary tools for businesses and individuals. As these intelligent systems become a bigger part of our lives, it is important to understand their functionality and to ensure their effectiveness. In this blog post, we will discuss how we can make sure that our Gen AI-powered systems are not only working properly but are also efficient and easy to use.  author: snandal -image: "/uploads/Evolving%20with%20AI%20From%20Traditional%20Testing%20to%20Model%20Evaluation%20I.png" +image: "/uploads/Testing%20GenarativeAI%20Chatbots.png" --- # Understanding GenAI models From 58e8794db07e5ec72d52ee5077efbe8546c444ef Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Wed, 30 Oct 2024 09:20:07 +0000 Subject: [PATCH 41/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index a1033f0c59..0354fefe7c 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -192,6 +192,7 @@ In summary, **Botium** is a comprehensive tool that provides all the necessary f Below are some useful resources on the tool : +- [Testing Conversational AI ](https://www.youtube.com/watch?v=1Z1a482_ZlA) - [Automated testing of a Google Dialogflow chatbot through Botium](https://www.youtube.com/watch?v=b8CgSkJmxCA) *By following a structured approach and using these tools, organisations can effectively test and refine GenAI chatbot models, ensuring high-quality outputs.* From bbd4eb0c9c2ed2f568a9bdc3a1cb5674dc2e1afe Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Wed, 30 Oct 2024 09:25:39 +0000 Subject: [PATCH 42/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 0354fefe7c..0bd0e2364d 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -192,7 +192,7 @@ In summary, **Botium** is a comprehensive tool that provides all the necessary f Below are some useful resources on the tool : -- [Testing Conversational AI ](https://www.youtube.com/watch?v=1Z1a482_ZlA) +- [Testing Conversational AI ](https://www.youtube.com/watch?v=1Z1a482_ZlA) *(for Botium part please skip to 21:00 min)* - [Automated testing of a Google Dialogflow chatbot through Botium](https://www.youtube.com/watch?v=b8CgSkJmxCA) *By following a structured approach and using these tools, organisations can effectively test and refine GenAI chatbot models, ensuring high-quality outputs.* From 7d369c5e502fc2613b6e6ca9d8bfa477c4143eb6 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Wed, 30 Oct 2024 09:50:37 +0000 Subject: [PATCH 43/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md removed image --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 0bd0e2364d..815fdb0450 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -8,7 +8,6 @@ tags: - Artificial Intelligence summary: In the fast-changing world of digital technology, GenAI systems have emerged as revolutionary tools for businesses and individuals. As these intelligent systems become a bigger part of our lives, it is important to understand their functionality and to ensure their effectiveness. In this blog post, we will discuss how we can make sure that our Gen AI-powered systems are not only working properly but are also efficient and easy to use.  author: snandal -image: "/uploads/Testing%20GenarativeAI%20Chatbots.png" --- # Understanding GenAI models From e922ea05abd004b85bba7e15ac12091fabb1dad2 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Wed, 30 Oct 2024 11:43:34 +0000 Subject: [PATCH 44/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md Implemented review comments provided by Sonali --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 815fdb0450..1c9cc4a43c 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -51,7 +51,7 @@ Hence GenAI model testing require sophisticated techniques such as probabilistic **4. Lack of Standardised Evaluation Metrics:** As AI technology is quite new, existing evaluation metrics may not fully capture the quality of the user experience. Establishing appropriate and comprehensive evaluation criteria is an ongoing challenge. Developing standardised metrics, that would accurately reflect the chatbot's performance and user satisfaction, remains a significant hurdle in GenAI model testing. -**5. Continuous learning and Evolution:** AI systems are designed to continuously learn from user interactions and evolve over time. This dynamic nature poses a significant challenge for testing, as it requires ongoing monitoring and testing of these models, to ensure consistent performance; and to identify and address any unintended issues that arise from continuous learning. +**5. Continuous learning and Evolution:** AI systems are designed to continuously learn from user interactions and evolve over time. This dynamic nature poses a significant challenge for testing. It requires ongoing monitoring and testing of these models to ensure consistent performance and to identify any unintended issues that may arise from continuous learning. Despite all these challenges, regular monitoring, combined with robust testing strategies can help ensure the accuracy, reliability and user satisfaction of their GenAI chatbot model. @@ -79,6 +79,7 @@ This involves testing the model on various devices, operating systems, browsers, Testing GenAI chatbot models involves evaluating multiple aspects, such as the quality, relevance, coherence and ethical considerations of the outputs generated by these models. Here is a structured approach to testing GenAI models, along with some tools and best practices: **Define Clear Evaluation Metrics** Establish specific metrics to evaluate the performance of the model. Common metrics include: + - Accuracy: How correct the outputs are based on expected results. - Relevance: The relevance of the generated content to the input prompt. - Coherence: Logical consistency and fluency of the generated text. From 7c59a83b7ef1b42a1fe57407ce32f432c4db7731 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Wed, 30 Oct 2024 16:20:34 +0000 Subject: [PATCH 45/54] Update _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md Co-authored-by: lhancock-scottlogic <110816543+lhancock-scottlogic@users.noreply.github.com> --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 1c9cc4a43c..3fc26ede78 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -21,7 +21,7 @@ GenAI chatbot models, such as GPT-4 by OpenAI, can generate human-like text and **1. Functionality:** GenAI chatbot models are programmed and trained to perform a wide range of tasks, from answering FAQs to executing complex commands. These models possess natural language understanding, contextual awareness and can understand and process diverse inputs, providing relevant and accurate responses. -**2. Learning Ability:** Through machine learning algorithms, these models have ability to continuously learn from interactions and to improve their responses over time. By analysing vast amounts of data and feedback from users, AI can refine its understanding and prediction capabilities and hence very adaptable to new patterns, trends and user behaviours; making it more accurate, reliable and effective in dynamic environments. +**2. Learning Ability:** Through machine learning algorithms, these models have ability to continuously learn from interactions and to improve their responses over time. By analysing vast amounts of data and feedback from users, AI can refine its understanding and prediction capabilities and is hence very adaptable to new patterns, trends and user behaviours; making it more accurate, reliable and effective in dynamic environments. This technology is revolutionising multiple industries like Healthcare, Entertainment, Marketing, Finance by enhancing creativity and efficiency. And hence there is the need for its effective testing to ensure that these models meet the desired standards of performance and user engagement. From c865c44865f0d1d570ac0aa0342db3bc05dd38b1 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Wed, 30 Oct 2024 16:21:09 +0000 Subject: [PATCH 46/54] Update _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md Co-authored-by: lhancock-scottlogic <110816543+lhancock-scottlogic@users.noreply.github.com> --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 3fc26ede78..7c78b717eb 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -24,7 +24,7 @@ GenAI chatbot models, such as GPT-4 by OpenAI, can generate human-like text and **2. Learning Ability:** Through machine learning algorithms, these models have ability to continuously learn from interactions and to improve their responses over time. By analysing vast amounts of data and feedback from users, AI can refine its understanding and prediction capabilities and is hence very adaptable to new patterns, trends and user behaviours; making it more accurate, reliable and effective in dynamic environments. -This technology is revolutionising multiple industries like Healthcare, Entertainment, Marketing, Finance by enhancing creativity and efficiency. And hence there is the need for its effective testing to ensure that these models meet the desired standards of performance and user engagement. +This technology is revolutionising multiple industries like Healthcare, Entertainment, Marketing, and Finance by enhancing creativity and efficiency. And hence there is the need for its effective testing to ensure that these models meet the desired standards of performance and user engagement. ## Why is it important to test GenAI chatbots? From 2550fe012517221cce579492ae11c6fe1f8bc605 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Wed, 30 Oct 2024 16:21:22 +0000 Subject: [PATCH 47/54] Update _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md Co-authored-by: lhancock-scottlogic <110816543+lhancock-scottlogic@users.noreply.github.com> --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 7c78b717eb..ececa9aa00 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -43,7 +43,7 @@ In summary, implementing robust testing for GenAI chatbots is essential for any Testing GenAI chatbots presents several unique challenges and limitations due to the intricacies of AI algorithms and the complexity of natural language processing. Below is an overview of some of the key challenges faced in testing GenAI chatbot models: **1. Non-Deterministic Responses:** GenAI chatbot models exhibit non-deterministic behaviour, meaning that they can generate completely different responses to the exact same input, under exact same conditions. These variations in responses are not necessarily errors, but mere deviations from previous behaviour, making it harder to define what constitutes a "correct" and “expected” response. -Hence GenAI model testing require sophisticated techniques such as probabilistic testing, where multiple iterations of the same test case are executed to assess the overall performance of the model. +Hence GenAI model testing requires sophisticated techniques such as probabilistic testing, where multiple iterations of the same test case are executed to assess the overall performance of the model. **2. Extensive Test Data Requirements:** GenAI chatbot models require substantial training data to accurately learn and interpret user inputs. For specialised domains, domain-specific test data is necessary. Collecting a diverse and representative dataset can be particularly challenging, especially in niche areas. Insufficient data can lead to reduced accuracy and reliability in the chatbot's responses, especially in context-specific scenarios. From 4f442b4c37a8b38a2d2115af32489b7057cb609d Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Wed, 30 Oct 2024 16:26:51 +0000 Subject: [PATCH 48/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md Implemented review comments from L Hancock --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index ececa9aa00..e29897ae23 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -195,4 +195,4 @@ Below are some useful resources on the tool : - [Testing Conversational AI ](https://www.youtube.com/watch?v=1Z1a482_ZlA) *(for Botium part please skip to 21:00 min)* - [Automated testing of a Google Dialogflow chatbot through Botium](https://www.youtube.com/watch?v=b8CgSkJmxCA) -*By following a structured approach and using these tools, organisations can effectively test and refine GenAI chatbot models, ensuring high-quality outputs.* +By following a structured approach and using these tools, organisations can effectively test and refine GenAI chatbot models, ensuring high-quality outputs. From 13b18bd73d676169446e99d81412245abe087ed7 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Thu, 31 Oct 2024 08:44:52 +0000 Subject: [PATCH 49/54] Delete _posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md --- ...aditional-Testing-to-Model-Evaluation-I.md | 154 ------------------ 1 file changed, 154 deletions(-) delete mode 100644 _posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md diff --git a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md b/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md deleted file mode 100644 index cbc1a032be..0000000000 --- a/_posts/2024-09-01-Evolving-with-AI-From-Traditional-Testing-to-Model-Evaluation-I.md +++ /dev/null @@ -1,154 +0,0 @@ ---- -title: Evolving with AI from Traditional Testing to Model Evaluation I -date: 2024-09-13 10:00:00 Z -categories: -- Artificial Intelligence -tags: -- Machine Learning -- Testing -- Model Evaluation -- Artificial Intelligence -summary: Having worked on developing Machine Learning skill definitions and L&D pathway recently, in this - blog post I have tried to explore the evolving role of test engineers in the era of machine learning, highlighting the key challenges ML brings, strategies for effective model evaluation and finally tried to outline a roadmap for developing the necessary skills to excel in ML model testing. -author: snandal -image: "/uploads/Evolving%20with%20AI%20From%20Traditional%20Testing%20to%20Model%20Evaluation%20I.png" ---- - -Machine learning (ML) is no longer just a concept for the future, it is now a vital part of many everyday applications. From personalised recommendations on streaming services and self-driving cars to detecting fraud in banking and advancements in healthcare, ML is changing the way industries work. - -For a test engineer, the increasing use of ML brings new challenges and opportunities. As ML technology advances, the role of a test engineer also changes. Unlike traditional software, which gives consistent and predictable results, ML models work with probabilities and patterns, making testing and evaluation more complex. - -In this blog, we will look at how ML has developed, how it might affect our job as test engineers and the important strategies, considerations and skills needed to effectively evaluate and test ML models. -We will also draw a roadmap to help us grow professionally in this area. - -## How Machine Learning Has Evolved - -Machine learning has significantly advanced since its early days. Initially, ML models were fairly simple, handling tasks like basic predictions or classifications. However, with improvements in computing power, access to large amounts of data and more complex algorithms, ML has grown into a powerful tool capable of handling sophisticated tasks like natural language processing and image recognition. - -This progress has made ML widespread across industries, leading businesses to increasingly rely on data-driven decisions. As companies embrace ML, the need for thorough and reliable testing of these models has become more important, which directly affects the role of test engineers. - -Before we get into the testing process, it is important to understand what makes ML models different and **what impact they can have on our job as a test engineer.** - -At their core, ML models learn from data. They are trained on large datasets to recognise patterns and make predictions or decisions based on new information. Unlike traditional software, where rules are clearly programmed, ML models develop their logic through training, making their behaviour less predictable and sometimes harder to understand. - -Traditionally, a test engineer's job involved testing systems where a given input always produced the same output, making results predictable. However, with ML, this approach changes. ML models are probabilistic, meaning their outputs can vary depending on the training data and some built-in randomness. This shift has significantly changed how software testing is done. - -We now need to do more than just check whether a system works as expected. We must also evaluate how well a model performs, whether it's reliable and fair. This requires combining traditional testing skills with additional knowledge specific to ML. - -## Looking at the unique challenges that testing ML models brings - -**Unpredictable Outputs:** Unlike regular software, where the same input always gives the same result, ML models can give different results because they work with probabilities. This makes testing harder because instead of just checking if the output is correct, we need to evaluate how accurate and reliable the model’s predictions are. - -**Dependence on Data:** The quality of an ML model depends a lot on the data it was trained on. Testing ML models means we need to deeply understand the training data and its weaknesses because any bias or missing information in the data can lead to incorrect results. - -**Performance Metrics:** Traditional testing metrics like pass/fail don’t apply directly to ML models. Instead, we use statistical metrics like accuracy, precision, recall, F1 score and AUC-ROC to evaluate model performance. Understanding and choosing the right metrics is key to effective testing. - -**Model Degradation Over Time:** As new data becomes available, an ML model’s performance can degrade causing a problem known as model drift. This means continuous monitoring and retraining are equally important, adding complexity to the testing process. - -## Model Evaluation Vs. Model Testing - -Before we dive deep into discussing possible strategies for ML model testing, it is important to understand the difference between Model evaluation and Model testing. - -Model evaluation and model testing might sound similar, but there is a key difference between the two. Evaluation focuses on assessing the model’s performance in a controlled environment, while model testing checks how well the model works in real-world conditions. - -Let's look at it with a simple example of a spam detection model that filters out unwanted emails. - -During the **model evaluation** phase (validation mode), we will use a labelled dataset of emails to calculate metrics like accuracy, precision and recall. For example, the model might successfully identify 95% of spam emails in this dataset. - -After evaluation, during **model testing** phase (prediction mode) the model will be subjected to real incoming emails to see how well it handles a variety of messages, if it can correctly filter out spam in different languages, formats, or tricky emails that look legitimate. Model testing makes sure the model works well in the real environment where it will be used, way beyond the controlled conditions of evaluation. - -## Strategies for Testing ML Models - -To effectively address the challenges that ML brings, test engineers can benefit from the following strategies: - -**1. Define Clear Objectives:** Start by defining what success looks like for your ML model. Are you aiming for high accuracy? or is it more important to avoid mistakes like false positives? Understanding the business objectives and how they translate into model performance metrics is essential. - -**2. Validate the Data:** Ensure that the training, validation and testing datasets are represent real-world situations the model will face. Check for issues like missing values, duplicates and biases. Proper data validation is very important to prevent garbage-in, garbage-out scenarios. - -**3. Apply Cross-Validation:** Implement cross-validation techniques to check that the model’s performance is consistent across different subsets of the data. This helps in identifying overfitting, where the model performs well on training data but poorly on new unseen data. - -**4. Analyse Performance Metrics:** Use a combination of metrics to get a comprehensive view of the model’s performance. For classification models, confusion matrices can be useful in providing insights into where the model is making errors. For regression models, metrics like Mean Squared Error (MSE) or R-squared can offer valuable information. - -**5. Perform A/B Testing:** Compare the ML model’s performance with existing solutions (if available) using A/B testing. This helps in understanding whether the new model provides a significant improvement over the current system. - -**6. Perform Stress Testing:** Subject the model to edge cases and scenarios with outliers to see how it handles unexpected inputs. This can uncover vulnerabilities and help in making the model more robust. - -**7. Continuous Monitoring and Feedback Loop:** After deployment, continuously monitor the model’s performance. Set up a feedback loop where new data is used to retrain and improve the model. Tools like MLflow can help in managing this process efficiently. - -### Let’s look at the practical application of the above strategies with a case study of an ML model predicting patient re-admission after discharge - -Consider a hospital implementing an ML model to predict the likelihood of patients being readmitted within 30 days of discharge. - -**Defining Objectives:** The goal here is to identify the most high-risk patients so that the hospital can take proactive measures to reduce readmission rates and improve patient care. - -**Data Validation:** Further start by validating the dataset, ensuring that it includes a balanced representation of various patient, medical history, treatment plans etc. Use cross-validation to train the model and select the most effective algorithm for predicting readmissions. - -**Metric Selection:** Choose sensitivity/recall and specificity as key metrics. High sensitivity ensures that most high-risk patients are identified, while specificity ensures that the model doesn't over-predict readmissions, leading to unnecessary interventions. - -**Stress Testing:** Test the model with scenarios where patient data is incomplete, such as missing medical history or inconsistent treatment records. Assess how the model performs when handling these edge cases, which are common in real-world medical data. - -**A/B Testing:** Implement the ML model alongside the existing manual risk assessment process. Compare the model's predictions with the manual predictions in real-time, focusing on how well the model improves patient outcomes. - -**Post-Deployment Monitoring:** After deployment, continuously monitor the model’s performance, looking for signs of model drift as new patient data becomes available. Implement a feedback loop where the model is retrained regularly with the latest data to maintain accuracy and relevance. - -## Skills Needed for ML Model Testing - -To effectively perform model evaluation, a test engineer would need a combination of technical, analytical and domain-specific skills: - -**1. Understanding of Machine Learning Concepts** - -* Understanding fundamental ML concepts such as model learning algorithms – supervised learning, unsupervised learning and reinforcement learning, model performance issues like overfitting and underfitting and the types of models (e.g., regression, classification, clustering). -* Familiarity and understanding of model evaluation metrics such as accuracy, precision, recall, F1 score, AUC-ROC, mean squared error, R-squared, etc. and knowing when and how to apply them. - -**2. Statistical Knowledge** - -* Understanding statistical measures - mean, median, variance, standard deviation and concepts like hypothesis testing, p-values, confidence intervals and correlation. -* Knowledge of probability distributions and how they apply to model outputs and uncertainties. - -**3. Programming Skills** - -* Proficiency in one of the scripting languages - Python or R, which are commonly used for ML model evaluation. Familiarity with libraries like Scikit-learn, TensorFlow or PyTorch is advantageous. -* Ability to write scripts to automate the evaluation of models, including running tests across different datasets and tracking performance over time. - -**4. Data Analysis and Validation** - -* Skills in data cleaning, handling missing data, outliers, data normalisation and feature engineering. -* Ability to validate datasets for biases, errors and representativeness, ensuring that the training data is of high quality and relevant to the model objectives. -* Proficiency in using tools like Python’s Pandas, Matplotlib and Seaborn for EDA to understand the dataset before feeding it into an ML model. - -**5. Familiarity with common ML Tools and Platforms** - -* Experience with tools like Scikit-learn for traditional ML, or TensorFlow/PyTorch for deep learning models. -* Familiarity with tools like MLflow, which assist in tracking experiments, managing models and automating the retraining process. -* Knowledge of using version control systems like Git for managing code and model versions. - -By combining these skills along with domain knowledge of the business context in which the model will be deployed, it can be effectively evaluate and tested, ensuring that it meets both technical and business requirements while remaining reliable, accurate and fair. - -## Roadmap for Developing ML Model Testing Skills - -Here’s a step-by-step roadmap for a test engineer to develop ML model testing skills: - -**1. Start with Basic ML Theory:** Learn the fundamental concepts of ML, focusing on the types of learning, model evaluation metrics and common challenges. - -**2. Build Statistical Knowledge:** Study key statistical principles and how they relate to ML. This is essential for understanding model performance. - -**3. Learn Data Analysis:** Practice data preprocessing, cleaning and validation using tools like Python’s Pandas or R. Data quality is critical for effective ML models. - -**4. Develop Programming Skills:** Get comfortable with Python or R. Start by running simple ML models using libraries like Scikit-learn and gradually move to more complex tasks. - -**5. Explore ML Tools:** Familiarise yourself with popular ML frameworks like TensorFlow, Keras, or PyTorch and model management tools like MLflow. - -**6. Practice Model Evaluation:** Apply your skills by testing real ML models. Use cross-validation, stress testing and performance metrics analysis to assess models. - -**7. Continuous Learning:** Stay updated with the latest trends in ML and software testing. Engage in online courses, webinars and forums to keep your knowledge fresh. - -By following this roadmap, test engineers can successfully transition into ML model testing, ensuring they stay relevant in an evolving field. - -## In Conclusion - -Evaluating and testing ML models requires a mix of traditional testing skills and an understanding of statistical methods. With the right testing strategies and by following a structured approach to testing, incorporating data validation, metric analysis, stress testing and continuous monitoring, we can ensure that our ML model provides real value and meets business objectives. - -Additionally, staying informed about the latest trends and tools can help handle the challenges of ML testing effectively and make a significant impact on your organisation’s success. -Machine learning is complex but also an exciting opportunity to explore new dimensions of software quality. Let’s embrace these challenges as a chance to grow and expand our skills in the field of software testing. - -*Hoping to publish my next blog soon, where I will dive deeper into evaluation metrics and how to choose the right ones for assessing machine learning models effectively.* From 9525fe9d3d7c0f0e3dd70b0c1c51cdbd3b36761a Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Thu, 31 Oct 2024 08:46:47 +0000 Subject: [PATCH 50/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index e29897ae23..68cf55bc64 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -42,7 +42,7 @@ In summary, implementing robust testing for GenAI chatbots is essential for any Testing GenAI chatbots presents several unique challenges and limitations due to the intricacies of AI algorithms and the complexity of natural language processing. Below is an overview of some of the key challenges faced in testing GenAI chatbot models: -**1. Non-Deterministic Responses:** GenAI chatbot models exhibit non-deterministic behaviour, meaning that they can generate completely different responses to the exact same input, under exact same conditions. These variations in responses are not necessarily errors, but mere deviations from previous behaviour, making it harder to define what constitutes a "correct" and “expected” response. +**1. Non-Deterministic Responses:** GenAI chatbot models exhibit non-deterministic behaviour, meaning that they can generate completely different responses to the exact same input, under the exact same conditions. These variations in responses are not necessarily errors, but mere deviations from previous behaviour, making it harder to define what constitutes a "correct" and “expected” response. Hence GenAI model testing requires sophisticated techniques such as probabilistic testing, where multiple iterations of the same test case are executed to assess the overall performance of the model. **2. Extensive Test Data Requirements:** GenAI chatbot models require substantial training data to accurately learn and interpret user inputs. For specialised domains, domain-specific test data is necessary. Collecting a diverse and representative dataset can be particularly challenging, especially in niche areas. Insufficient data can lead to reduced accuracy and reliability in the chatbot's responses, especially in context-specific scenarios. From 1cb24d096d3d68085dfea32a8eb30355add663bf Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Thu, 31 Oct 2024 12:47:11 +0000 Subject: [PATCH 51/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 68cf55bc64..65d20989eb 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -78,7 +78,7 @@ This involves testing the model on various devices, operating systems, browsers, Testing GenAI chatbot models involves evaluating multiple aspects, such as the quality, relevance, coherence and ethical considerations of the outputs generated by these models. Here is a structured approach to testing GenAI models, along with some tools and best practices: -**Define Clear Evaluation Metrics** Establish specific metrics to evaluate the performance of the model. Common metrics include: +**Define Clear Evaluation Metrics -** Establish specific metrics to evaluate the performance of the model. Common metrics include: - Accuracy: How correct the outputs are based on expected results. - Relevance: The relevance of the generated content to the input prompt. @@ -86,23 +86,23 @@ Testing GenAI chatbot models involves evaluating multiple aspects, such as the q - Diversity: The variety of responses generated from similar prompts. - Ethical Considerations: Checking for biases, harmful or inappropriate responses. -**Simulate Real User Behaviour** Use diverse and realistic test data to emulate actual user interactions. This involves: +**Simulate Real User Behaviour -** Use diverse and realistic test data to emulate actual user interactions. This involves: - Testing with data that mimics real-world scenarios to uncover practical issues. - Conducting tests with actual user interactions to gather authentic feedback. - Establishing feedback loops to continuously collect and act on user feedback for ongoing improvement. -**Focus on Conversation Flow** Ensure the chatbot can maintain context and handle conversations naturally. This includes: +**Focus on Conversation Flow -** Ensure the chatbot can maintain context and handle conversations naturally. This includes: - Testing the chatbot’s ability to understand and manage references to previous interactions. - Evaluating its performance in handling ambiguous language and variations, such as slang and dialects. -**Monitor Performance** Continuously track performance metrics to ensure that the chatbot remains responsive and reliable. Key actions include: +**Monitor Performance -** Continuously track performance metrics to ensure that the chatbot remains responsive and reliable. Key actions include: - Regularly reviewing accuracy, relevance and coherence metrics. - Using monitoring tools to track real-time performance and detect any issues promptly. -**Incorporate User Feedback**  User feedback is invaluable for refining the chatbot. Best practices include: +**Incorporate User Feedback -**  User feedback is invaluable for refining the chatbot. Best practices include: - Using feedback to update and enhance test cases, ensuring they remain relevant and comprehensive. - Iteratively improving the chatbot’s accuracy and relevance based on real user interactions and suggestions. From fb476dfedd37408c560eb80e52da9c62e528b3bf Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Fri, 1 Nov 2024 10:40:17 +0000 Subject: [PATCH 52/54] Update 2024-10-28-Testing-GenerativeAI-Chatbots.md Added image --- _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md index 65d20989eb..c6a145c1d6 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md @@ -8,6 +8,7 @@ tags: - Artificial Intelligence summary: In the fast-changing world of digital technology, GenAI systems have emerged as revolutionary tools for businesses and individuals. As these intelligent systems become a bigger part of our lives, it is important to understand their functionality and to ensure their effectiveness. In this blog post, we will discuss how we can make sure that our Gen AI-powered systems are not only working properly but are also efficient and easy to use.  author: snandal +image: "/uploads/Testing%20GenarativeAI%20Chatbots.png" --- # Understanding GenAI models From f4c21b853469a822baf546c3a787edeacf5d08ee Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Fri, 1 Nov 2024 10:42:51 +0000 Subject: [PATCH 53/54] Update and rename 2024-10-28-Testing-GenerativeAI-Chatbots.md to 2024-11-01-Testing-GenerativeAI-Chatbots.md Updated date in the file and the filename --- ...-Chatbots.md => 2024-11-01-Testing-GenerativeAI-Chatbots.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename _posts/{2024-10-28-Testing-GenerativeAI-Chatbots.md => 2024-11-01-Testing-GenerativeAI-Chatbots.md} (99%) diff --git a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md b/_posts/2024-11-01-Testing-GenerativeAI-Chatbots.md similarity index 99% rename from _posts/2024-10-28-Testing-GenerativeAI-Chatbots.md rename to _posts/2024-11-01-Testing-GenerativeAI-Chatbots.md index c6a145c1d6..996028aea2 100644 --- a/_posts/2024-10-28-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-11-01-Testing-GenerativeAI-Chatbots.md @@ -1,6 +1,6 @@ --- title: Testing GenerativeAI Chatbot Models -date: 2024-10-28 10:00:00 Z +date: 2024-11-01 10:00:00 Z categories: - Artificial Intelligence tags: From acdacd4cf79851c8572b2b10d79cd8d8ac603a67 Mon Sep 17 00:00:00 2001 From: Shikha-Nandal <141818785+Shikha-Nandal@users.noreply.github.com> Date: Fri, 1 Nov 2024 11:28:40 +0000 Subject: [PATCH 54/54] Update 2024-11-01-Testing-GenerativeAI-Chatbots.md Added placeholder in the image metadata url --- _posts/2024-11-01-Testing-GenerativeAI-Chatbots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2024-11-01-Testing-GenerativeAI-Chatbots.md b/_posts/2024-11-01-Testing-GenerativeAI-Chatbots.md index 996028aea2..bafc61b9b4 100644 --- a/_posts/2024-11-01-Testing-GenerativeAI-Chatbots.md +++ b/_posts/2024-11-01-Testing-GenerativeAI-Chatbots.md @@ -8,7 +8,7 @@ tags: - Artificial Intelligence summary: In the fast-changing world of digital technology, GenAI systems have emerged as revolutionary tools for businesses and individuals. As these intelligent systems become a bigger part of our lives, it is important to understand their functionality and to ensure their effectiveness. In this blog post, we will discuss how we can make sure that our Gen AI-powered systems are not only working properly but are also efficient and easy to use.  author: snandal -image: "/uploads/Testing%20GenarativeAI%20Chatbots.png" +image: "{{ site.github.url }}/uploads/Testing%20GenarativeAI%20Chatbots.png" --- # Understanding GenAI models