align definition of duplicate with its behavior #950
+2
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for jovial-piroshki-65ff4d ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
Related slack thread: https://sodadata.slack.com/archives/CAXRR8SS3/p1738086720079429 |
adkinsty
requested changes
Mar 4, 2025
| | `duplicate_count` | The number of rows that contain duplicate values.<br> Include one column in the argument to compare values relative to that one column. <br/>Include more than one column in the argument to search for duplicate pairs multiple columns. Be sure to add a space between the comma-separated values in the list of column names. <br />See also: [Duplicate check]({% link soda/quick-start-sodacl.md %}#duplicate-check)| number<br /> text<br /> time | all | | ||
| | `duplicate_percent` | The percentage of rows in a dataset that contain duplicate values.<br> Include one column in the argument to compare values relative to that one column. Include more than one column in the argument to search for duplicate pairs in multiple columns. Be sure to add a space between the comma-separated values in the list of column names. <br />See also: [Duplicate check]({% link soda/quick-start-sodacl.md %}#duplicate-check)| number<br /> text<br /> time | all | | ||
| | `duplicate_count` | The count of distinct values that have duplicates.<br> Include one column in the argument to compare values relative to that one column. <br/>Include more than one column in the argument to search for duplicate pairs multiple columns. Be sure to add a space between the comma-separated values in the list of column names. <br />See also: [Duplicate check]({% link soda/quick-start-sodacl.md %}#duplicate-check)| number<br /> text<br /> time | all | | ||
| | `duplicate_percent` | The percentage of rows in a dataset that have dupliates.<br> Include one column in the argument to compare values relative to that one column. Include more than one column in the argument to search for duplicate pairs in multiple columns. Be sure to add a space between the comma-separated values in the list of column names. <br />See also: [Duplicate check]({% link soda/quick-start-sodacl.md %}#duplicate-check)| number<br /> text<br /> time | all | |
There was a problem hiding this comment.
"dupliates" should be "duplicates"?
There was a problem hiding this comment.
Still I'm not sure if this accurately captures the meaning of the duplicate percent metric...hmm...I think it's literally duplicate_count divided by row_count times 100. Do you think this matches the proposed description?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
align definition of duplicate_chck duplicate_percent with its behavior
"The behavior of duplicate checks is a bit unintuitive and doesn't match our docs, and I am wondering if this is a bug. The value of a duplicate count check is equal to the count of distinct values that have duplicates. But in the docs it says a duplicate_count is "The number of rows that contain duplicate values". There is a similar issue for duplicate percent checks, for which the value is the count of distinct values that have duplicates divided by the total number of rows, but the docs say duplicate_percent is "The percentage of rows in a dataset that contain duplicate values"."