Skip to content

Commit 375e6fd

Browse files
ArshaanNazirArshaanNazir
authored
Merge pull request #717 from JohnSnowLabs/release/1.3.0
Release/1.3.0
2 parents 015ce74 + e52191d commit 375e6fd

89 files changed

Lines changed: 39515 additions & 66357 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/llm_tests_build.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: Manual Tests
2+
3+
# This allows you to run the workflow manually from the GitHub Actions UI. We use it for running llm-tests.
4+
on:
5+
workflow_dispatch:
6+
7+
jobs:
8+
manual_tests:
9+
runs-on: ubuntu-latest
10+
strategy:
11+
fail-fast: false
12+
matrix:
13+
python-version: [ "3.8", "3.9", "3.10" ]
14+
15+
steps:
16+
- uses: actions/checkout@v3
17+
- name: Set up Python ${{ matrix.python-version }}
18+
uses: actions/setup-python@v3
19+
with:
20+
python-version: ${{ matrix.python-version }}
21+
22+
- uses: snok/install-poetry@v1
23+
with:
24+
version: 1.3.1
25+
virtualenvs-create: true
26+
virtualenvs-in-project: true
27+
installer-parallel: true
28+
29+
- name: Load cached venv
30+
id: cached-poetry-dependencies
31+
uses: actions/cache@v3
32+
with:
33+
path: .venv
34+
key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
35+
36+
- name: Install dependencies
37+
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
38+
run: |
39+
poetry install --with dev --all-extras
40+
41+
- name: Run manual_tests
42+
env:
43+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
44+
run: |
45+
poetry run pytest manual_tests/

‎CONTRIBUTING.md‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
# Contributing to LangTest
2+
3+
Thank you for your interest in contributing to LangTest! We value the contributions of our community members and are excited to have you on board. . To make the development journey smooth and collaborative, please adhere to the following steps:
4+
- Thoroughly review the guidelines provided below.
5+
- If you're eager to add a new feature to the LangTest library, navigate to the [documentation](https://langtest.org/docs/pages/docs/contribute) for a clear understanding.
6+
- To illustrate the process, we've included a simple example in the documentation to guide you.
7+
8+
## Finding an Issue
9+
10+
1. **Explore Our Issues:**
11+
Browse through our list of open issues on the [Issues tab](https://github.com/YourUsername/LangTest/issues). These issues cover a wide range of topics and difficulties. You might find something that aligns with your skills and interests.
12+
13+
2. **Filter and Search:**
14+
Use the provided labels and filters to narrow down the list of issues. You can filter by tags like `feature`, `bug`, `fix`, etc. Also, utilize the GitHub search bar to look for keywords related to your expertise.
15+
16+
3. **Read and Understand:**
17+
Once you've found an issue that captures your attention, take a moment to thoroughly read and understand it. Make sure you comprehend the problem, the expected solution, and any existing discussion around it.
18+
19+
4. **Clarify Doubts:**
20+
If you have any questions or need further clarification, don't hesitate to comment on the issue. Our community is friendly and always willing to help. It's better to seek clarity before diving in.
21+
22+
5. **Claim the Issue:**
23+
If you feel confident about tackling the issue, leave a comment expressing your intention to work on it. This prevents duplication of efforts and allows us to provide guidance if needed.
24+
25+
6. **Understand Contribution Guidelines:**
26+
Familiarize yourself with our [Contribution Guidelines](https://github.com/JohnSnowLabs/langtest/wiki). This document outlines coding standards, pull request procedures, and other important details.
27+
28+
## Contribution Process
29+
30+
<img align="right" width="300" src="https://github.com/RakshitKhajuria/first-contributions/assets/71117423/be22d54d-5b62-4a23-b213-0268ed195021" alt="fork this repository" />
31+
32+
1. ### Fork this repository
33+
34+
Fork this repository by clicking on the fork button on the top of this page.
35+
This will create a copy of this repository in your account.
36+
37+
2. ### Clone the repository
38+
39+
<img align="right" width="300" src="https://github.com/RakshitKhajuria/test/assets/71117423/b07c7d60-ab80-4b7f-b972-58fcd1f50741" alt="clone this repository" />
40+
41+
Now clone the forked repository to your machine. Go to your GitHub account, open the forked repository, click on the code button and then click the _copy to clipboard_ icon.
42+
43+
Open a terminal and run the following git command:
44+
45+
```
46+
git clone "url you just copied"
47+
```
48+
49+
where "url you just copied" (without the quotation marks) is the url to this repository (your fork of this project). See the previous steps to obtain the url.
50+
51+
<img align="right" width="300" src="https://github.com/RakshitKhajuria/test/assets/71117423/8eef9f61-283d-4bad-9e7f-8a526e33615e" alt="copy URL to clipboard" />
52+
53+
For example:
54+
55+
```
56+
git clone https://github.com/username/langtest.git
57+
```
58+
59+
where `username` is your GitHub username.
60+
61+
3. ### Create a branch
62+
63+
Change to the repository directory on your computer (if you are not already there):
64+
65+
```
66+
cd langtest
67+
```
68+
69+
Now create a branch using the `git checkout -b` command:
70+
71+
```
72+
git checkout -b your-branch-name
73+
```
74+
75+
4. **Set Up Environment**: Create a virtual environment for LangTest using your preferred method. For example, you can use `venv` or `conda`.
76+
77+
5. ### Install Poetry
78+
79+
Poetry is the package manager used for this project. If you don't have Poetry installed, run the following command:
80+
81+
```bash
82+
pip install poetry
83+
```
84+
85+
6. ### Install Dependencies
86+
87+
Use Poetry to install the project dependencies:
88+
89+
```bash
90+
poetry install --with dev
91+
```
92+
93+
7. ### Make necessary changes and commit those changes
94+
95+
Add those changes to the branch you just created using the `git add` command:
96+
97+
```
98+
git add .
99+
```
100+
Now commit those changes using the `git commit` command with a meaningful message:
101+
102+
```
103+
git commit -m "Add feature: feature-name"
104+
```
105+
106+
8. ### Push changes to GitHub
107+
108+
Push your changes using the command `git push`:
109+
110+
```
111+
git push -u origin your-branch-name
112+
```
113+
114+
replacing `your-branch-name` with the name of the branch you created earlier.
115+
116+
9. ### Submit your changes for review
117+
118+
If you go to your repository on GitHub, you'll see a `Compare & pull request` button. Click on that button.
119+
120+
<img style="float: right;" src="https://github.com/RakshitKhajuria/test/assets/71117423/86cb56af-1dad-467e-afc1-f4594615783b" alt="create a pull request" />
121+
122+
10. ### Create a pull request (PR)
123+
124+
Create a pull request (PR) from your forked repository to the main LangTest repository.
125+
- Clearly describe the changes you've made and the issue it addresses in the PR description.
126+
- If the PR is related to an existing issue, mention it using the syntax "Fixes #issue_number."
127+
> For pull request template click [here](https://github.com/JohnSnowLabs/langtest/blob/main/PULL_REQUEST_TEMPLATE.md)
128+
129+
<img style="float: right;" src="https://github.com/RakshitKhajuria/test/assets/71117423/5c629508-53a7-4444-b036-15f694df675c" alt="submit pull request" />
130+
131+
Our team will review your pull request, provide feedback, and work with you to ensure that your contribution meets the project's guidelines and quality standards
132+
133+
## Proposing a New Issue
134+
135+
1. If you have an idea for a new feature, improvement, or bug fix, you can propose a new issue.
136+
2. Go to the [Issue Tracker](https://github.com/JohnSnowLabs/langtest/issues) and click on "New Issue" or just create a new issue from [here](https://github.com/JohnSnowLabs/langtest/issues/new)
137+
3. Choose the appropriate issue lables (feature, fix, bug, etc.).
138+
4. Provide a clear and concise title and description of the proposed issue.
139+
5. Add labels and assignees, if applicable, to help categorize and track the issue.
140+
141+
We're excited to have you on board with us! Together, we can make NLP models safer, more reliable, and fairer for everyone.
142+
143+
### **Happy contributing!** 💫
144+
145+
## Where to go from here?
146+
147+
Congrats! You just completed the standard fork -> clone -> edit -> pull request workflow that you'll often encounter as a contributor!
148+
149+
Celebrate your contribution and share it with your friends and followers by going to [LinkedIn](www.linkedin.com).
150+
151+
You can join our community team if you need any help or have any questions.
152+
153+
- [Slack](https://www.johnsnowlabs.com/slack-redirect/) For live discussion with the LangTest community, join the `#langtest` channel
154+
- [GitHub](https://github.com/JohnSnowLabs/langtest/tree/main) For bug reports, feature requests, and contributions
155+
- [Discussions](https://github.com/JohnSnowLabs/langtest/discussions) To engage with other community members, share ideas, and show off how you use LangTest!
156+
157+
## Contributors
158+
159+
We would like to acknowledge all contributors of this open-source community project.
160+
161+
<a href="https://github.com/johnsnowlabs/langtest/graphs/contributors">
162+
<img src="https://contrib.rocks/image?repo=johnsnowlabs/langtest" />
163+
</a>
164+
165+
## License
166+
167+
LangTest is released under the [Apache License 2.0](https://github.com/JohnSnowLabs/langtest/blob/main/LICENSE), which guarantees commercial use, modification, distribution, patent use, private use and sets limitations on trademark use, liability and warranty.

‎demo/tutorials/end-to-end-notebooks/HuggingFace_Real_World_Notebook.ipynb‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -111,13 +111,14 @@
111111
"<br/>\n",
112112
"\n",
113113
"\n",
114-
"| Parameter | Description | \n",
115-
"| - | - | \n",
116-
"|**task** |Task for which the model is to be evaluated (text-classification or ner)|\n",
117-
"|**model** |PipelineModel or path to a saved model or pretrained pipeline/model from hub.\n",
118-
"|**data** |Path to the data that is to be used for evaluation. Can be .csv or .conll file in the CoNLL format \n",
119-
"|**config** |Configuration for the tests to be performed, specified in form of a YAML file.\n",
120-
"|**hub** |model hub to load from the path. Required if model param is passed as path.|\n",
114+
"\n",
115+
"| Parameter | Description |\n",
116+
"| ------------- | ----------- |\n",
117+
"| **task** | Task for which the model is to be evaluated (text-classification or ner) |\n",
118+
"| **model** | Specifies the model(s) to be evaluated. Can be a dictionary or a list of dictionaries. Each dictionary should contain 'model' and 'hub' keys. If a path is specified, the dictionary must contain 'model' and 'hub' keys. |\n",
119+
"| **data** | The data to be used for evaluation. A dictionary providing flexibility and options for data sources. It should include the following keys: <ul><li>data_source (mandatory): The source of the data.</li><li>subset (optional): The subset of the data.</li><li>feature_column (optional): The column containing the features.</li><li>target_column (optional): The column containing the target labels.</li><li>split (optional): The data split to be used.</li></ul> |\n",
120+
"| **config** | Configuration for the tests to be performed, specified in the form of a YAML file. |\n",
121+
"\n",
121122
"\n",
122123
"<br/>\n",
123124
"<br/>"
@@ -549,7 +550,8 @@
549550
},
550551
"outputs": [],
551552
"source": [
552-
"h = Harness(task=\"ner\", model=\"trained_model\", hub=\"huggingface\", data=\"sample.conll\")"
553+
"\n",
554+
"h = Harness(task=\"ner\", model={\"model\": \"trained_model\", \"hub\": \"huggingface\"}, data={\"data_source\" :\"sample.conll\"})"
553555
]
554556
},
555557
{

‎demo/tutorials/end-to-end-notebooks/JohnSnowLabs_RealWorld_Custom_Pipeline_Notebook.ipynb‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -109,13 +109,14 @@
109109
"<br/>\n",
110110
"\n",
111111
"\n",
112-
"| Parameter | Description |\n",
113-
"| - | - |\n",
114-
"|**task** |Task for which the model is to be evaluated (text-classification or ner)|\n",
115-
"|**model** |PipelineModel or path to a saved model or pretrained pipeline/model from hub.\n",
116-
"|**data** |Path to the data that is to be used for evaluation. Can be .csv or .conll file in the CoNLL format\n",
117-
"|**config** |Configuration for the tests to be performed, specified in form of a YAML file.\n",
118-
"|**hub** |model hub to load from the path. Required if model param is passed as path.|\n",
112+
"\n",
113+
"| Parameter | Description |\n",
114+
"| ------------- | ----------- |\n",
115+
"| **task** | Task for which the model is to be evaluated (text-classification or ner) |\n",
116+
"| **model** | Specifies the model(s) to be evaluated. Can be a dictionary or a list of dictionaries. Each dictionary should contain 'model' and 'hub' keys. If a path is specified, the dictionary must contain 'model' and 'hub' keys. |\n",
117+
"| **data** | The data to be used for evaluation. A dictionary providing flexibility and options for data sources. It should include the following keys: <ul><li>data_source (mandatory): The source of the data.</li><li>subset (optional): The subset of the data.</li><li>feature_column (optional): The column containing the features.</li><li>target_column (optional): The column containing the target labels.</li><li>split (optional): The data split to be used.</li></ul> |\n",
118+
"| **config** | Configuration for the tests to be performed, specified in the form of a YAML file. |\n",
119+
"\n",
119120
"\n",
120121
"<br/>\n",
121122
"<br/>"
@@ -409,7 +410,7 @@
409410
},
410411
"outputs": [],
411412
"source": [
412-
"harness = Harness(task=\"ner\", model=ner_model, data=\"sample.conll\")"
413+
"harness = Harness(task=\"ner\", model={\"model\": ner_model, \"hub\": \"johnsnowlabs\"}, data={\"data_source\" :\"sample.conll\"})"
413414
]
414415
},
415416
{

‎demo/tutorials/end-to-end-notebooks/JohnSnowLabs_RealWorld_Notebook.ipynb‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -109,13 +109,14 @@
109109
"<br/>\n",
110110
"\n",
111111
"\n",
112-
"| Parameter | Description | \n",
113-
"| - | - | \n",
114-
"|**task** |Task for which the model is to be evaluated (text-classification or ner)|\n",
115-
"|**model** |PipelineModel or path to a saved model or pretrained pipeline/model from hub.\n",
116-
"|**data** |Path to the data that is to be used for evaluation. Can be .csv or .conll file in the CoNLL format \n",
117-
"|**config** |Configuration for the tests to be performed, specified in form of a YAML file.\n",
118-
"|**hub** |model hub to load from the path. Required if model param is passed as path.|\n",
112+
"\n",
113+
"| Parameter | Description |\n",
114+
"| ------------- | ----------- |\n",
115+
"| **task** | Task for which the model is to be evaluated (text-classification or ner) |\n",
116+
"| **model** | Specifies the model(s) to be evaluated. Can be a dictionary or a list of dictionaries. Each dictionary should contain 'model' and 'hub' keys. If a path is specified, the dictionary must contain 'model' and 'hub' keys. |\n",
117+
"| **data** | The data to be used for evaluation. A dictionary providing flexibility and options for data sources. It should include the following keys: <ul><li>data_source (mandatory): The source of the data.</li><li>subset (optional): The subset of the data.</li><li>feature_column (optional): The column containing the features.</li><li>target_column (optional): The column containing the target labels.</li><li>split (optional): The data split to be used.</li></ul> |\n",
118+
"| **config** | Configuration for the tests to be performed, specified in the form of a YAML file. |\n",
119+
"\n",
119120
"\n",
120121
"<br/>\n",
121122
"<br/>"
@@ -242,7 +243,7 @@
242243
},
243244
"outputs": [],
244245
"source": [
245-
"harness = Harness(task=\"ner\", model=ner_model, data=\"sample.conll\", hub=\"johnsnowlabs\")"
246+
"harness = Harness(task=\"ner\", model={\"model\": ner_model, \"hub\": \"johnsnowlabs\"}, data={\"data_source\" :\"sample.conll\"})"
246247
]
247248
},
248249
{
@@ -1284,7 +1285,7 @@
12841285
}
12851286
],
12861287
"source": [
1287-
"harness = Harness.load(\"saved_test_configurations\",model=augmented_ner_model)"
1288+
"harness = Harness.load(\"saved_test_configurations\",model=augmented_ner_model,task=\"ner\")"
12881289
]
12891290
},
12901291
{

‎demo/tutorials/end-to-end-notebooks/Spacy_Real_World_Notebook.ipynb‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -89,13 +89,14 @@
8989
"<br/>\n",
9090
"\n",
9191
"\n",
92-
"| Parameter | Description | \n",
93-
"| - | - | \n",
94-
"|**task** |Task for which the model is to be evaluated (text-classification or ner)|\n",
95-
"|**model** |PipelineModel or path to a saved model or pretrained pipeline/model from hub.\n",
96-
"|**data** |Path to the data that is to be used for evaluation. Can be .csv or .conll file in the CoNLL format \n",
97-
"|**config** |Configuration for the tests to be performed, specified in form of a YAML file.\n",
98-
"|**hub** |model hub to load from the path. Required if model param is passed as path.|\n",
92+
"\n",
93+
"| Parameter | Description |\n",
94+
"| ------------- | ----------- |\n",
95+
"| **task** | Task for which the model is to be evaluated (text-classification or ner) |\n",
96+
"| **model** | Specifies the model(s) to be evaluated. Can be a dictionary or a list of dictionaries. Each dictionary should contain 'model' and 'hub' keys. If a path is specified, the dictionary must contain 'model' and 'hub' keys. |\n",
97+
"| **data** | The data to be used for evaluation. A dictionary providing flexibility and options for data sources. It should include the following keys: <ul><li>data_source (mandatory): The source of the data.</li><li>subset (optional): The subset of the data.</li><li>feature_column (optional): The column containing the features.</li><li>target_column (optional): The column containing the target labels.</li><li>split (optional): The data split to be used.</li></ul> |\n",
98+
"| **config** | Configuration for the tests to be performed, specified in the form of a YAML file. |\n",
99+
"\n",
99100
"\n",
100101
"<br/>\n",
101102
"<br/>"
@@ -416,7 +417,7 @@
416417
},
417418
"outputs": [],
418419
"source": [
419-
"h = Harness(task=\"ner\",model=spacy_model, data=\"/content/sample.conll\")"
420+
"h = Harness(task=\"ner\",model={\"model\": spacy_model, \"hub\": \"spacy\"}, data={\"data_source\": \"/content/sample.conll\"})"
420421
]
421422
},
422423
{

0 commit comments

Comments
 (0)