|
| 1 | +--- |
| 2 | +layout: docs |
| 3 | +header: true |
| 4 | +seotitle: LangTest - Deliver Safe and Effective Language Models | John Snow Labs |
| 5 | +title: LangTest Release Notes |
| 6 | +permalink: /docs/pages/docs/langtest_versions/release_notes_1_10_0 |
| 7 | +key: docs-release-notes |
| 8 | +modify_date: 2023-10-17 |
| 9 | +--- |
| 10 | + |
| 11 | +<div class="h3-box" markdown="1"> |
| 12 | + |
| 13 | +## 1.10.0 |
| 14 | + |
| 15 | +## 📢 Highlights |
| 16 | + |
| 17 | + |
| 18 | +🌟 **LangTest 1.10.0 Release by John Snow Labs** |
| 19 | + |
| 20 | +We're thrilled to announce the latest release of LangTest, introducing remarkable features that elevate its capabilities and user-friendliness. This update brings a host of enhancements: |
| 21 | + |
| 22 | +- **Evaluating RAG with LlamaIndex and Langtest**: LangTest seamlessly integrates LlamaIndex for constructing a RAG and employs LangtestRetrieverEvaluator, measuring retriever precision (Hit Rate) and accuracy (MRR) with both standard and perturbed queries, ensuring robust real-world performance assessment. |
| 23 | + |
| 24 | +- **Grammar Testing for NLP Model Evaluation:** This approach entails creating test cases through the paraphrasing of original sentences. The purpose is to evaluate a language model's proficiency in understanding and interpreting the nuanced meaning of the text, enhancing our understanding of its contextual comprehension capabilities. |
| 25 | + |
| 26 | + |
| 27 | +- **Saving and Loading the Checkpoints:** LangTest now supports the seamless saving and loading of checkpoints, providing users with the ability to manage task progress, recover from interruptions, and ensure data integrity. |
| 28 | + |
| 29 | +- **Extended Support for Medical Datasets:** LangTest adds support for additional medical datasets, including LiveQA, MedicationQA, and HealthSearchQA. These datasets enable a comprehensive evaluation of language models in diverse medical scenarios, covering consumer health, medication-related queries, and closed-domain question-answering tasks. |
| 30 | + |
| 31 | + |
| 32 | +- **Direct Integration with Hugging Face Models:** Users can effortlessly pass any Hugging Face model object into the LangTest harness and run a variety of tasks. This feature streamlines the process of evaluating and comparing different models, making it easier for users to leverage LangTest's comprehensive suite of tools with the wide array of models available on Hugging Face. |
| 33 | + |
| 34 | + |
| 35 | +</div><div class="h3-box" markdown="1"> |
| 36 | + |
| 37 | +## 🔥 Key Enhancements: |
| 38 | + |
| 39 | +### 🚀Implementing and Evaluating RAG with LlamaIndex and Langtest |
| 40 | + [](https://github.com/JohnSnowLabs/langtest/blob/main/demo/tutorials/RAG/RAG_OpenAI.ipynb) |
| 41 | + |
| 42 | +LangTest seamlessly integrates LlamaIndex, focusing on two main aspects: constructing the RAG with LlamaIndex and evaluating its performance. The integration involves utilizing LlamaIndex's generate_question_context_pairs module to create relevant question and context pairs, forming the foundation for retrieval and response evaluation in the RAG system. |
| 43 | + |
| 44 | +To assess the retriever's effectiveness, LangTest introduces LangtestRetrieverEvaluator, employing key metrics such as Hit Rate and Mean Reciprocal Rank (MRR). Hit Rate gauges the precision by assessing the percentage of queries with the correct answer in the top-k retrieved documents. MRR evaluates the accuracy by considering the rank of the highest-placed relevant document across all queries. This comprehensive evaluation, using both standard and perturbed queries generated through LangTest, ensures a thorough understanding of the retriever's robustness and adaptability under various conditions, reflecting its real-world performance. |
| 45 | + |
| 46 | +``` |
| 47 | +from langtest.evaluation import LangtestRetrieverEvaluator |
| 48 | +
|
| 49 | +retriever_evaluator = LangtestRetrieverEvaluator.from_metric_names( |
| 50 | + ["mrr", "hit_rate"], retriever=retriever |
| 51 | +) |
| 52 | + |
| 53 | +retriever_evaluator.setPerturbations("add_typo","dyslexia_word_swap", "add_ocr_typo") |
| 54 | +
|
| 55 | +# Evaluate |
| 56 | +eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset) |
| 57 | +
|
| 58 | +retriever_evaluator.display_results() |
| 59 | +
|
| 60 | +``` |
| 61 | + |
| 62 | +### 📚Grammar Testing in Evaluating and Enhancing NLP Models |
| 63 | + [](https://github.com/JohnSnowLabs/langtest/blob/main/demo/tutorials/test-specific-notebooks/Grammar_Demo.ipynb) |
| 64 | + |
| 65 | +Grammar Testing is a key feature in LangTest's suite of evaluation strategies, emphasizing the assessment of a language model's proficiency in contextual understanding and nuance interpretation. By creating test cases that paraphrase original sentences, the goal is to gauge the model's ability to comprehend and interpret text, thereby enriching insights into its contextual mastery. |
| 66 | + |
| 67 | +{:.table3} |
| 68 | +| Category | Test Type | Original | Test Case | Expected Result | Actual Result | Pass | |
| 69 | +|----------|------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------:|------------------|---------------|-------| |
| 70 | +| grammar | paraphrase | This program was on for a brief period when I was a kid, I remember watching it whilst eating fish and chips.<br /><br />Riding on the back of the Tron hype this series was much in the style of streethawk, manimal and the like, except more computery. There was a geeky kid who's computer somehow created this guy - automan. He'd go around solving crimes and the lot.<br /><br />All I really remember was his fancy car and the little flashy cursor thing that used to draw the car and help him out generally.<br /><br />When I mention it to anyone they can remember very little too. Was it real or maybe a dream? | I remember watching a show from my youth that had a Tron theme, with a nerdy kid driving around with a little flashy cursor and solving everyday problems. Was it a genuine story or a mere dream come true? | NEGATIVE | POSITIVE | false | |
| 71 | + |
| 72 | +### 🔥 Saving and Loading the Checkpoints |
| 73 | + [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/Saving_Checkpoints.ipynb) |
| 74 | +Introducing a robust checkpointing system in LangTest! The `run` method in the `Harness` class now supports checkpointing, allowing users to save intermediate results, manage batch processing, and specify a directory for storing checkpoints and results. This feature ensures data integrity, providing a mechanism for recovering progress in case of interruptions or task failures. |
| 75 | +``` |
| 76 | +harness.run(checkpoint=True, batch_size=20,save_checkpoints_dir="imdb-checkpoint") |
| 77 | +``` |
| 78 | +The `load_checkpoints` method facilitates the direct loading of saved checkpoints and data, providing a convenient mechanism to resume testing tasks from the point where they were previously interrupted, even in the event of runtime failures or errors. |
| 79 | +``` |
| 80 | +harness = Harness.load_checkpoints(save_checkpoints_dir="imdb-checkpoint", |
| 81 | + task="text-classification", |
| 82 | + model = {"model": "lvwerra/distilbert-imdb" , "hub":"huggingface"}, ) |
| 83 | +``` |
| 84 | + |
| 85 | +### 🏥 Added Support for More Medical Datasets |
| 86 | + |
| 87 | +#### LiveQA |
| 88 | +The LiveQA'17 medical task focuses on consumer health question answering. It consists of constructed medical question-answer pairs for training and testing, with additional annotations. LangTest now supports LiveQA for comprehensive medical evaluation. |
| 89 | + |
| 90 | +##### How the dataset looks: |
| 91 | + |
| 92 | +{:.table3} |
| 93 | +| category | test_type | original_question | perturbed_question | expected_result | actual_result | eval_score | pass | |
| 94 | +|------------|-----------|------------------------------------------------------|-----------------------------------------------------------|-----------------------------------------------------------------|------------------------------------------------------------|------------|------| |
| 95 | +| robustness | uppercase | Do amphetamine salts 20mg tablets contain gluten? | DO AMPHETAMINE SALTS 20MG TABLETS CONTAIN GLUTEN? | No, amphetamine salts 20mg tablets do not contain gluten. | No, Amphetamine Salts 20mg Tablets do not contain gluten. | 1.0 | true | |
| 96 | + |
| 97 | +#### MedicationQA |
| 98 | +The MedicationQA dataset consists of commonly asked consumer questions about medications. It includes annotations corresponding to drug focus and interactions. LangTest now integrates MedicationQA for thorough evaluation of models in medication-related scenarios. |
| 99 | + |
| 100 | +##### How the dataset looks: |
| 101 | + |
| 102 | +{:.table3} |
| 103 | +| category | test_type | original_question | perturbed_question | expected_result | actual_result | eval_score | pass | |
| 104 | +|------------|-----------|-----------------------------------------------------|--------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|------| |
| 105 | +| robustness | uppercase | how does rivatigmine and otc sleep medicine interact | HOW DOES RIVATIGMINE AND OTC SLEEP MEDICINE INTERACT | Rivastigmine is a cholinesterase inhibitor and OTC (over-the-counter) sleep medicine is a sedative. There is a risk of excessive sedation when taking both of these medications together. Patients should consult their doctor before taking both of these medications together. | Rivastigmine and OTC sleep medicines may interact. Rivastigmine can increase the effects of sedatives, including OTC sleep medicines, resulting in increased drowsiness. It's important to talk to your doctor before taking OTC sleep medicines while taking Rivastigmine. | 0.9365371414708803 | true | |
| 106 | + |
| 107 | + |
| 108 | +#### HealthSearchQA |
| 109 | +HealthSearchQA, a new free-response dataset of medical questions sought online, with six existing open-question answering datasets covering professional medical exams, research, and consumer queries. The HealthsearchQA dataset consists of 3375 frequently-asked consumer questions and was curated using seed medical diagnoses and their related symptoms. |
| 110 | + |
| 111 | +##### How the dataset looks: |
| 112 | + |
| 113 | +{:.table3} |
| 114 | +| category | test_type | original_question | perturbed_question | expected_result | actual_result | eval_score | pass | |
| 115 | +|------------|-----------|------------------------------------------|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------|---------------------|------| |
| 116 | +| robustness | uppercase | Are dental abscesses serious? | ARE DENTAL ABSCESSES SERIOUS? | Yes, dental abscesses are serious and can lead to infection and damage to the surrounding tissue if left untreated. Treatment typically involves antibiotics and/or draining the abscess. If left untreated, the infection can spread to other parts of the body. | Dental abscesses can be serious and require prompt medical attention. Left untreated, they can cause swelling, spreading infections, and damage to the surrounding teeth and bone. | 0.9457038739103363 | true | |
| 117 | + |
| 118 | + |
| 119 | + |
| 120 | +### 🚀Direct Integration with Hugging Face Models |
| 121 | + |
| 122 | +Users can effortlessly pass any Hugging Face model object into the LangTest harness and run a variety of tasks. This feature streamlines the process of evaluating and comparing different models, making it easier for users to leverage LangTest's comprehensive suite of tools with the wide array of models available on Hugging Face. |
| 123 | + |
| 124 | + |
| 125 | + |
| 126 | + |
| 127 | +## 🚀 New LangTest Blogs: |
| 128 | + |
| 129 | +{:.table2} |
| 130 | +| Blog | Description | |
| 131 | +| --- | --- | |
| 132 | +| [LangTest: A Secret Weapon for Improving the Robustness of Your Transformers Language Models](https://www.johnsnowlabs.com/langtest-a-secret-weapon-for-improving-the-robustness-of-your-transformers-language-models/) | Explore the robustness of Transformers Language Models with LangTest Insights. | |
| 133 | +| [Testing the Robustness of LSTM-Based Sentiment Analysis Models](https://medium.com/john-snow-labs/testing-the-robustness-of-lstm-based-sentiment-analysis-models-67ed84e42997) | Explore the robustness of custom models with LangTest Insights. | |
| 134 | + |
| 135 | +## 🐛 Bug Fixes |
| 136 | + |
| 137 | +- Fixed LangTestCallback errors |
| 138 | +- Fixed QA, Default Config, and Transformer Model for QA |
| 139 | +- Fixed multi-model evaluation |
| 140 | +- Fixed datasets format |
| 141 | + |
| 142 | +## ⚒️ Previous Versions |
| 143 | + |
| 144 | +</div> |
| 145 | +{%- include docs-langtest-pagination.html -%} |
0 commit comments