Lab5 fixes #53
Open
Lab5 fixes #53
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sometimes, and article inside `train_data` has more than one paragraph.
In this situation, data["paragraph"][0] does not use the second one.
The code `set([len(article["paragraphs"]) for article in train_data])` outputs `{1, 2}` both for `train_data` and `dev_data`
Fixed the code to utilize every paragraph in the data souce.
Changes:
In the cell right below header "Ładowanie danych":
Train data articles: 8553 -> 11624
Dev data articles: 1402 -> 1453
Train questions: 41577 -> 56618
Dev questions: 6809 -> 7060
In the cell calculating `all_contexts`:
`len(all_contexts)`: 9955 -> 13077
In the cell which is changing the dataset to a shape of Pytanie: Kontekst:
Total count in train/dev: 75605/12372 -> 102805/12824
Positive count in train/dev: 34028/5563 -> 46187/5764
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First commit fixes some typos
Second commit fixes not using every paragraph in
train_dataanddev_datadatasets, details below.Sometimes an article inside
train_dataordev_datahas more than one paragraph.In this situation, data["paragraph"][0] only uses the first one.
The code
set([len(article["paragraphs"]) for article in train_data])outputs{1, 2}both for
train_dataanddev_dataFixed the code to utilize every paragraph in the data souce. Changes:
In the cell right below header "Ładowanie danych", line 1376:
Train data articles: 8553 -> 11624
Dev data articles: 1402 -> 1453
Train questions: 41577 -> 56618
Dev questions: 6809 -> 7060
In the cell calculating
all_contexts, line 1415:len(all_contexts): 9955 -> 13077In the cell which is changing the dataset to a shape of Pytanie: Kontekst:, line 1466:
Total count in train/dev: 75605/12372 -> 102805/12824
Positive count in train/dev: 34028/5563 -> 46187/5764