Skip to content

Commit 015ce74

Browse files
ArshaanNazirArshaanNazir
authored
Merge pull request #698 from JohnSnowLabs/release/1.2.0
Release/1.2.0
2 parents cc12acf + 687f23c commit 015ce74

96 files changed

Lines changed: 20574 additions & 2894 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/release.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
name: Release
2+
on:
3+
release:
4+
types:
5+
- published
6+
7+
jobs:
8+
publish:
9+
strategy:
10+
fail-fast: false
11+
matrix:
12+
python-version: [3.9]
13+
poetry-version: [1.3.1]
14+
os: [ubuntu-latest]
15+
runs-on: ${{ matrix.os }}
16+
steps:
17+
- uses: actions/checkout@v3
18+
- uses: actions/setup-python@v3
19+
with:
20+
python-version: ${{ matrix.python-version }}
21+
- uses: snok/install-poetry@v1
22+
with:
23+
version: ${{ matrix.poetry-version }}
24+
virtualenvs-create: true
25+
virtualenvs-in-project: true
26+
installer-parallel: true
27+
- name: Publish langtest ${{ github.event.release.name }} to pypi
28+
env:
29+
PYPI_SECRET: ${{ secrets.PYPI_SECRET }}
30+
run: |
31+
poetry config pypi-token.pypi $PYPI_SECRET
32+
poetry publish --build

‎README.md‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ Take a look at our official page for user documentation and examples: [langtest.
4040

4141
```python
4242
# Install langtest
43-
!pip install langtest transformers==4.28.1
43+
!pip install langtest[transformers]
4444

4545
# Import and create a Harness object
4646
from langtest import Harness
@@ -67,6 +67,27 @@ We propose here an early stage open-source community project that aims to fill t
6767

6868
[John Snow Labs](www.johnsnowlabs.com) has a full development team allocated to the project and is committed to improving the library for years, as we do with other open-source libraries. Expect frequent releases with new test types, tasks, languages, and platforms to be added regularly. We look forward to working together to make safe, reliable, and responsible NLP an everyday reality.
6969

70+
## Comparing Benchmark Datasets: Use Cases and Evaluations
71+
72+
Langtest comes with different datasets to test your models, covering a wide range of use cases and evaluation scenarios.
73+
74+
| Dataset | Use Case | Notebook |
75+
|---------------|----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|
76+
| [**BoolQ**](https://aclanthology.org/N19-1300/) | Evaluate the ability of your model to answer boolean questions (yes/no) based on a given passage or context. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/BoolQ_dataset.ipynb) |
77+
| [**NQ-open**](https://aclanthology.org/Q19-1026/) | Evaluate the ability of your model to answer open-ended questions based on a given passage or context. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/NQ_open_dataset.ipynb) |
78+
| [**TruthfulQA**](https://aclanthology.org/2022.acl-long.229/) | Evaluate the model's capability to answer questions accurately and truthfully based on the provided information. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/TruthfulQA_dataset.ipynb) |
79+
| [**MMLU**](https://arxiv.org/abs/2009.03300) | Evaluate language understanding models' performance in different domains. It covers 57 subjects across STEM, the humanities, the social sciences, and more. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/mmlu_dataset.ipynb) |
80+
| [**NarrativeQA**](https://aclanthology.org/Q18-1023/) | Evaluate your model's ability to comprehend and answer questions about long and complex narratives, such as stories or articles. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/NarrativeQA_Question_Answering.ipynb) |
81+
| [**HellaSwag**](https://aclanthology.org/P19-1472/) | Evaluate your model's ability in completions of sentences. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/HellaSwag_Question_Answering.ipynb) |
82+
| [**Quac**](https://aclanthology.org/D18-1241/) | Evaluate your model's ability to answer questions given a conversational context, focusing on dialogue-based question-answering. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/quac_dataset.ipynb) |
83+
| [**OpenBookQA**](https://allenai.org/data/open-book-qa)| Evaluate your model's ability to answer questions that require complex reasoning and inference based on general knowledge, similar to an "open-book" exam. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/quac_dataset.ipynb) |
84+
| [**BBQ**](https://arxiv.org/abs/2110.08193) | Evaluate how your model responds to questions in the presence of social biases against protected classes across various social dimensions. Assess biases in model outputs with both under-informative and adequately informative contexts, aiming to promote fair and unbiased question-answering models. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/BBQ_dataset.ipynb) |
85+
|[**XSum**](https://aclanthology.org/D18-1206/) | Evaluate your model's ability to generate concise and informative summaries for long articles with the XSum dataset. It consists of articles and corresponding one-sentence summaries, offering a valuable benchmark for text summarization models. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/XSum_dataset.ipynb)|
86+
|[**Real Toxicity Prompts**](https://aclanthology.org/2020.findings-emnlp.301/) | Evaluate your model's accuracy in recognizing and handling toxic language with the Real Toxicity Prompts dataset. It contains real-world prompts from online platforms, ensuring robustness in NLP models to maintain safe environments. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/OpenAI_QA_Testing_Notebook.ipynb)
87+
88+
> **Note**
89+
> For usage and documentation, head over to [langtest.org](https://langtest.org/docs/pages/docs/data#question-answering)
90+
7091
## Contributing
7192

7293
We welcome all sorts of contributions:

‎demo/tutorials/end-to-end-notebooks/HuggingFace_Real_World_Notebook.ipynb‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1412,7 +1412,11 @@
14121412
}
14131413
],
14141414
"source": [
1415-
"h.augment(input_path=\"conll03.conll\", output_path=\"augmented_conll03.conll\", inplace=False)"
1415+
"data_kwargs = {\n",
1416+
" \"data_source\" : \"conll03.conll\",\n",
1417+
" }\n",
1418+
"\n",
1419+
"h.augment(training_data=data_kwargs, save_data_path=\"augmented_conll03.conll\", export_mode=\"add\")"
14161420
]
14171421
},
14181422
{

‎demo/tutorials/end-to-end-notebooks/JohnSnowLabs_RealWorld_Custom_Pipeline_Notebook.ipynb‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1368,7 +1368,11 @@
13681368
}
13691369
],
13701370
"source": [
1371-
"harness.augment(\"conll03.conll\", \"augmented_train.conll\", inplace=False)"
1371+
"data_kwargs = {\n",
1372+
" \"data_source\" : \"conll03.conll\",\n",
1373+
" }\n",
1374+
"\n",
1375+
"harness.augment(training_data=data_kwargs, save_data_path=\"augmented_train.conll\", export_mode=\"add\")"
13721376
]
13731377
},
13741378
{

‎demo/tutorials/end-to-end-notebooks/JohnSnowLabs_RealWorld_Notebook.ipynb‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1203,7 +1203,11 @@
12031203
}
12041204
],
12051205
"source": [
1206-
"harness.augment(\"conll03.conll\", \"augmented.conll\", inplace=False)"
1206+
"data_kwargs = {\n",
1207+
" \"data_source\" : \"conll03.conll\",\n",
1208+
" }\n",
1209+
"\n",
1210+
"harness.augment(training_data=data_kwargs, save_data_path=\"augmented.conll\", export_mode=\"add\")"
12071211
]
12081212
},
12091213
{

‎demo/tutorials/end-to-end-notebooks/Spacy_Real_World_Notebook.ipynb‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1382,7 +1382,11 @@
13821382
}
13831383
],
13841384
"source": [
1385-
"h.augment(\"conll03.conll\", \"augmented.conll\", inplace=False)"
1385+
"data_kwargs = {\n",
1386+
" \"data_source\" : \"conll03.conll\",\n",
1387+
" }\n",
1388+
"\n",
1389+
"h.augment(training_data=data_kwargs, save_data_path=\"augmented.conll\", export_mode=\"add\")"
13861390
]
13871391
},
13881392
{

‎demo/tutorials/llm_notebooks/AI21_QA_Summarization_Testing_Notebook.ipynb‎

Lines changed: 3 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
"id": "3o5sAOfwL5qd"
1818
},
1919
"source": [
20-
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/AI21_QA_Testing_Notebook.ipynb)"
20+
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/AI21_QA_Summarization_Testing_Notebook.ipynb)"
2121
]
2222
},
2323
{
@@ -39,7 +39,7 @@
3939
"id": "26qXWhCYhHAt"
4040
},
4141
"source": [
42-
"# Getting started with LangTest on John Snow Labs"
42+
"# Getting started with LangTest"
4343
]
4444
},
4545
{
@@ -54,23 +54,7 @@
5454
},
5555
"outputs": [],
5656
"source": [
57-
"!pip install langtest"
58-
]
59-
},
60-
{
61-
"cell_type": "markdown",
62-
"metadata": {},
63-
"source": [
64-
"## Installing required dependencies"
65-
]
66-
},
67-
{
68-
"cell_type": "code",
69-
"execution_count": null,
70-
"metadata": {},
71-
"outputs": [],
72-
"source": [
73-
"!pip install \"langtest[evaluate,ai21,langchain]\" transformers==4.28.1"
57+
"!pip install \"langtest[evaluate,ai21,langchain,transformers]\" "
7458
]
7559
},
7660
{

‎demo/tutorials/llm_notebooks/Azure_OpenAI_QA_Summarization_Testing_Notebook.ipynb‎

Lines changed: 3 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
"id": "3o5sAOfwL5qd"
1818
},
1919
"source": [
20-
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Azure_OpenAI_QA_Testing_Notebook.ipynb)"
20+
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Azure_OpenAI_QA_Summarization_Testing_Notebook.ipynb)"
2121
]
2222
},
2323
{
@@ -39,29 +39,7 @@
3939
"id": "26qXWhCYhHAt"
4040
},
4141
"source": [
42-
"# Getting started with LangTest on John Snow Labs"
43-
]
44-
},
45-
{
46-
"cell_type": "code",
47-
"execution_count": null,
48-
"metadata": {
49-
"colab": {
50-
"base_uri": "https://localhost:8080/"
51-
},
52-
"id": "azUb114QhOsY",
53-
"outputId": "45a0a692-5076-4fcf-ceb4-b5eb914d9aed"
54-
},
55-
"outputs": [],
56-
"source": [
57-
"!pip install langtest"
58-
]
59-
},
60-
{
61-
"cell_type": "markdown",
62-
"metadata": {},
63-
"source": [
64-
"## Installing required dependencies"
42+
"# Getting started with LangTest"
6543
]
6644
},
6745
{
@@ -70,7 +48,7 @@
7048
"metadata": {},
7149
"outputs": [],
7250
"source": [
73-
"!pip install \"langtest[evaluate,langchain,openai]\" transformers==4.28.1"
51+
"!pip install \"langtest[evaluate,langchain,openai,transformers]\" "
7452
]
7553
},
7654
{

‎demo/tutorials/llm_notebooks/Cohere_QA_Summarization_Testing_Notebook.ipynb‎

Lines changed: 3 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
"id": "3o5sAOfwL5qd"
1818
},
1919
"source": [
20-
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Cohere_QA_Testing_Notebook.ipynb)"
20+
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Cohere_QA_Summarization_Testing_Notebook.ipynb)"
2121
]
2222
},
2323
{
@@ -39,29 +39,7 @@
3939
"id": "26qXWhCYhHAt"
4040
},
4141
"source": [
42-
"# Getting started with LangTest on John Snow Labs"
43-
]
44-
},
45-
{
46-
"cell_type": "code",
47-
"execution_count": null,
48-
"metadata": {
49-
"colab": {
50-
"base_uri": "https://localhost:8080/"
51-
},
52-
"id": "azUb114QhOsY",
53-
"outputId": "45a0a692-5076-4fcf-ceb4-b5eb914d9aed"
54-
},
55-
"outputs": [],
56-
"source": [
57-
"!pip install langtest"
58-
]
59-
},
60-
{
61-
"cell_type": "markdown",
62-
"metadata": {},
63-
"source": [
64-
"## Installing required dependencies"
42+
"# Getting started with LangTest "
6543
]
6644
},
6745
{
@@ -70,7 +48,7 @@
7048
"metadata": {},
7149
"outputs": [],
7250
"source": [
73-
"!pip install \"langtest[evaluate,cohere,langchain]\" transformers==4.28.1"
51+
"!pip install \"langtest[evaluate,cohere,langchain,transformers]\" "
7452
]
7553
},
7654
{

‎demo/tutorials/llm_notebooks/HuggingFaceHub_QA_Summarization_Testing_Notebook.ipynb‎

Lines changed: 3 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,5 @@
11
{
22
"cells": [
3-
{
4-
"attachments": {},
5-
"cell_type": "markdown",
6-
"metadata": {
7-
"id": "yR6kjOaiheKN"
8-
},
9-
"source": [
10-
"# Harness and Its Parameters\n",
11-
"\n",
12-
"The Harness class is a testing class for Natural Language Processing (NLP) models. It evaluates the performance of a NLP model on a given task using test data and generates a report with test results.Harness can be imported from the LangTest library in the following way."
13-
]
14-
},
153
{
164
"attachments": {},
175
"cell_type": "markdown",
@@ -29,7 +17,7 @@
2917
"id": "3o5sAOfwL5qd"
3018
},
3119
"source": [
32-
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/HuggingFaceHub_QA_Testing_Notebook.ipynb)"
20+
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/HuggingFaceHub_QA_Summarization_Testing_Notebook.ipynb)"
3321
]
3422
},
3523
{
@@ -51,7 +39,7 @@
5139
"id": "26qXWhCYhHAt"
5240
},
5341
"source": [
54-
"# Getting started with LangTest on John Snow Labs"
42+
"# Getting started with LangTest"
5543
]
5644
},
5745
{
@@ -66,23 +54,7 @@
6654
},
6755
"outputs": [],
6856
"source": [
69-
"!pip install \"langtest[evaluate,langchain]\" transformers==4.28.1"
70-
]
71-
},
72-
{
73-
"cell_type": "markdown",
74-
"metadata": {},
75-
"source": [
76-
"## Installing required dependencies"
77-
]
78-
},
79-
{
80-
"cell_type": "code",
81-
"execution_count": null,
82-
"metadata": {},
83-
"outputs": [],
84-
"source": [
85-
"!pip install \"langtest[transformers,johnsnowlabs]\""
57+
"!pip install \"langtest[evaluate,langchain,transformers]\" "
8658
]
8759
},
8860
{

0 commit comments

Comments
 (0)