Readme changes regarding attributions, new import functionality, todo…

… updates.

readme.md

@@ -58,9 +58,43 @@ Change `savePageAsIndexOnRightClick` to which index your save as is on your brow
 
 The rest is fine if you don't have a really old computer.
 
-## Credits
+## Dependencies
 
-I'd like to personally thank all the people that have developed\/maintained\/poured their hearts into the packages\/tools this *thing* is using and the artists that pushed me to keep a copy of their music in my library.
+### Binaries
+
+[Python ~=3.11](https://www.python.org/) - Core language. Licensed under PSFL license.
+
+[FFmpeg](https://ffmpeg.org/) - Used in conversion of the files. Licensed under LGPLv2.1 license.
+
+### Python packages
+
+[beautifulsoup4](https://www.crummy.com/software/BeautifulSoup/) - Used in parsing the user's likes page HTML. Licensed under MIT license.
+
+[eyed3](https://github.com/nicfit/eyeD3) - Used in tagging the files. Licensed under GPL-3.0 license.
+
+[ffmpeg-python](https://github.com/kkroening/ffmpeg-python) - Used in converting files to desired format(s) as a wrapper. Licensed under Apache-2.0 license.
+
+[mutagen](https://github.com/quodlibet/mutagen)* - Subdependency, planned to replace eyed3. Licensed under GPL-2.0 license.
+
+[pandas](https://github.com/pandas-dev/pandas) - Used in CSV/JSON helper utilities. Licensed under BSD-3-Clause license.
+
+[pillow](https://github.com/python-pillow/Pillow) - Used in modification of covers. Licensed under MIT-CMU license.
+
+[pyautogui](https://github.com/asweigart/pyautogui) - Used in fetcher to get the user's likes page. Licensed under BSD-3-Clause license.
+
+[pyinstaller](https://github.com/pyinstaller/pyinstaller) - Used in compilation. Licensed under a GPLv2 license.
+
+[pyyaml](https://github.com/yaml/pyyaml) - Used in handling user's config files. Licensed under MIT license.
+
+[yt-dlp](https://github.com/yt-dlp/yt-dlp) - Used in downloading user's library. Licensed under Unlicense license.
+
+[fuzzywuzzy](https://github.com/seatgeek/fuzzywuzzy) - Used in comparison utilities for migration for fuzzy matching. Licensed under GPLv2 license.
+
+[python-levenshtein](https://github.com/rapidfuzz/python-Levenshtein)* - Subdependency for fuzzywuzzy. Licensed under GPL-2.0 license.
+
+[keyboard](https://github.com/boppreh/keyboard) - Used in comparison utilities for migration to handle keypresses. Licensed under MIT license.
+
+[prettytable](https://github.com/prettytable/prettytable) - Used in comparison utilities as a part of UX. Licensed under a custom license.
 
 ## Disclaimer
 

todo.md

@@ -9,9 +9,11 @@
 
 ## I Highest Importance
 
-- [ ] CSV "import" currently overwrites the file :facepalm:
-  - [ ] Convert the provided CSV to JSON. check if key exists, if not add it yada yada
-  - [ ] Move it to intermediates
+- [ ] Allow the library file to have no artist & no title specified
+  - Make a function to fetch missing metadata?
+    - InnerTube
+    - yt-dlp
+      - `yt-dlp --get-filename -o "%(uploader)s hsfOqb5r8m0VbV31 %(title)s" VIDEO_URL` for null artists, which is caused by ViMusic's search function returning empty
 
 - [ ] Make an executable for Linux aswell
   - [x] Need the user to download ffmpeg binaries as `ffmpeg-python` is just a wrapper, should mention that in the readme
@@ -20,74 +22,83 @@
   - [x] Write a utility script to check if there are missing pairs
   - [x] `matcher.py`, helper script to make the switch from your old archive.
   - [ ] Script to migrate to youtubes metadata using previously created user metadata.
+    - [ ] InnerTube
     - [ ] Use a headless browser to get links under Songs if it doesn't exist as a Song fallback to Video
       - `https://music.youtube.com/search?q=example+song`
   - [ ] Script to remove duplicate entries that only differ with the key.
 
 - [ ] Provide the user with a fallback json that consists of replacement keys \(for the ID's that fail or preference\)
+  - [ ] Write a sanitazation function to remove fallbacks and replace them
+    - [ ] Make sure every import (fetcher, rimusic, csv) does not add the keys to the library
   - [ ] Use the dbtools utility to be written to prompt the user for the replacement key
 
 - [x] Log unavailable videos into a file.
   - [ ] Could be better
 
-- [ ] Exit and notify the user somehow when `yt-dlp` outputs `Sign in to confirm you’re not a bot. This helps protect our community.`
-  - [x] Break download loop
-  - [ ] Use an alternative?
-    - Piped
-    - [Invidious](https://github.com/grqz/yt-dlp-invidious)
-
 ## II Requires Research
 
+- [ ] Find out what InnerTube API can do
+  - [ ] for [python](https://github.com/tombulled/innertube)
+  - [ ] for [javascript](https://github.com/LuanRT/YouTube.js/)
+
 - [ ] Find out how to launch gui without a console
 
 - [ ] Find out if there is a better way to structure the CLI logic or make it more maintainable in general [click](https://click.palletsprojects.com/en/8.1.x/)?
 
-- [ ] Do async downloading
-
 - [ ] Find out if this output from `yt-dlp` affects anything `WARNING: [youtube] Failed to download m3u8 information: HTTP Error 429: Too Many Requests`
 
-- [ ] Fetch lyrics and add them into `library.json`
-  - Sources that RiMusic is using seems pretty solid.
+- [ ] Fetch lyrics and tag the songs with it
+  - Unsynced?
+    - Boring
+  - Synced?
+    - Sources that RiMusic is using seems pretty solid.
+    - Not always available
+
+- [ ] Do async downloading
 
 - [ ] Write tests
 
-- [ ] InnerTube
+- [ ] Some languages have characters that are indexed inside the latin alphabet and by default I think Python just looks for its unicode index, so fixing this
 
 ## III Lowest Importance
 
-- [ ] Make settings interactable (e.g import the file named x)
+- [x] Do attributions for dependencies
+  - Add links for the licenses
+
+- [ ] Add a function to delete files in `downloads` that aren't in `library.json`
+
+- [ ] Add a function to delete entries in `library.json` that aren't in `downloads`
+
+- [ ] Remove quirky file finding method for the `temp` directory in `download` function as it's not required anymore?
 
 - [ ] Log bad thumbnails
   - [ ] `hqdefault`
   - [ ] Ones that are available as `hqdefault` but aren't a square (`720x720`)
     - OpenCV hsl similiarity? the fills in those files aren't just 1 color last I tried & checked
 
-- [ ] Remove quirky file finding method for the `temp` directory in `download` function as it's not required anymore?
-
-- [ ] Allow the library file to have no artist & no title specified
-  - Tell `yt-dlp` to automatically tag it
-  - Make a function to fetch missing metadata?
-
-- [x] Handle cover and music seperately in `downloader.py`
-  - [ ] Better `return` values
-
 - [ ] Replace `eyed3` with `mutagen`
 
-- [ ] Rename `key` to `video_id`
+- [ ] Replace `fuzzywuzzy` with [`TheFuzz`](https://github.com/seatgeek/thefuzz)
 
-- [ ] Write a better scrape algorithm, get album names properly this time.
+- [ ] Make settings interactable (e.g import the file named x)
 
-- [ ] Add a function to delete files in `downloads` that aren't in `library.json`
+- [x] Handle cover and music seperately in `downloader.py`
+  - [ ] Better `return` values
 
-- [ ] Add a function to delete entries in `library.json` that aren't in `downloads`
+- [ ] Rename `key` to `watch_id`
 
-- [ ] Fetch metadata from YouTube instead of using the existing data
+- [ ] Exit and notify the user somehow when `yt-dlp` outputs `Sign in to confirm you’re not a bot. This helps protect our community.`
+  - [x] Break download loop and go to the next task
+  - [ ] Use an alternative?
+    - Piped
+    - [Invidious](https://github.com/grqz/yt-dlp-invidious)
 
-- [ ] Implement `yt-dlp --get-filename -o "%(uploader)s hsfOqb5r8m0VbV31 %(title)s" VIDEO_URL` for null artists, which is caused by ViMusic's search function returning empty
+- [ ] Write a better scrape algorithm, get album names properly.
+  - What to do with songs that doesn't belong to an album?
 
 - [ ] Write `PermissionError` exceptions
 
-- [ ] Some languages have characters that are indexed inside the latin alphabet and by default I think Python just looks for its unicode index, so fixing this
+- [ ] Write new GUI with [ImGui](https://github.com/hoffstadt/DearPyGui)
 
 ## IV Doubts and Head Scratchers
 
@@ -107,6 +118,10 @@
 
 NOTE: These will be removed after a while.
 
+- [x] CSV "import" currently overwrites the file :facepalm:
+  - [x] Convert the provided CSV to JSON. check if key exists, if not add it yada yada
+  - [x] Move it to intermediates
+
 - [x] Record any manual changes to songs ~~\[did do this, but it's instead checking if json has been manually edited\]~~
   - why did i do it for the json? probably i hadn't done the export import at that time and was doing something wacky
   - [x] Check the downloads directory if any of the song tags has been manually changed

ytmasc/intermediates.py

@@ -5,6 +5,8 @@
 from shutil import rmtree
 
 from eyed3 import load as loadmp3
+from json import dump
+from pandas import read_csv
 
 from ytmasc.converter import convert_bulk
 from ytmasc.downloader import download_bulk
@@ -141,3 +143,37 @@ def create_config():
             },
         }
         update_yaml(yaml_config, default_config)
+
+
+def import_csv(csv_file: str, json_file: str, overwrite=True):
+    df = read_csv(csv_file)
+    df.fillna("", inplace=True)
+    json_data = read_json(json_file)
+
+    for index, row in df.iterrows():
+        key = row.iloc[0]
+        value1 = row.iloc[1]
+        value2 = row.iloc[2]
+
+        if key in json_data:
+            logger.info(f"Key {key} is already in the library.")
+            if (
+                (json_data[key]["artist"] != value1)
+                or (json_data[key]["title"] != value2)
+            ) and overwrite:
+                logger.info(
+                    f"Values don't match, updating with:\n"
+                    f"artist: {json_data[key]['artist']} -> {row.iloc[1]}\n"
+                    f"title: {json_data[key]['title']} -> {row.iloc[2]}"
+                )
+                json_data[key] = {"artist": value1, "title": value2}
+        else:
+            logger.info(
+                f"Key {key} is not in library, adding it with values:\n"
+                f"artist: {row.iloc[1]}\n"
+                f"title: {row.iloc[2]}"
+            )
+            json_data[key] = {"artist": value1, "title": value2}
+
+    with open(json_file, "w") as f:
+        dump(json_data, f, indent=2)

ytmasc/intermediates_cli.py

@@ -4,7 +4,11 @@
 
 from ytmasc.dbtools.comparison import compare
 from ytmasc.dbtools.find_unpaired import find_unpaired_files
-from ytmasc.intermediates import update_library_with_manual_changes_on_files, run_tasks
+from ytmasc.intermediates import (
+    import_csv,
+    update_library_with_manual_changes_on_files,
+    run_tasks,
+)
 from ytmasc.tk_gui import create_gui
 from ytmasc.parser import parse_library_page, parse_ri_music_db
 from ytmasc.utility import (
@@ -54,7 +58,15 @@ def get_cli_args():
     parser.add_argument(
         "--import_csv_to_library",
         action="store_true",
-        help="Imports a CSV of 3 columns [ID, artist, title]",
+        help="Imports a CSV of 3 columns [ID, artist, title]. If keys exist but their values are different they will be updated with the tags from the CSV",
+    )
+    parser.add_argument(
+        "--import_csv_to_library_no_overwrite",
+        action="store_true",
+        help="Imports a CSV of 3 columns [ID, artist, title]. If keys exist but their values are different they will NOT be updated with the tags from the CSV",
+    )
+    parser.add_argument(
+        "--direct_import", action="store_true", help="Completely overwrites the library"
     )
     parser.add_argument(
         "--db_compare",
@@ -90,8 +102,10 @@ def handle_cli(args: classmethod):
             args.update_library_with_manual_changes_on_files
             or args.export_library_as_csv
             or args.import_csv_to_library
-            or args.update_tags
+            or args.import_csv_to_library_no_overwrite
+            or args.direct_import
             or args.db_compare
+            or args.db_find_unpaired
         ):
             create_gui()
 
@@ -101,9 +115,14 @@ def handle_cli(args: classmethod):
     if args.export_library_as_csv:
         convert_json_to_csv(library_data_path, csv_library_data_path)
 
+    # modify this later on and get a filename
     if args.import_csv_to_library:
+        import_csv(csv_library_data_path, library_data_path)
+    elif args.import_csv_to_library_no_overwrite:
+        import_csv(csv_library_data_path, library_data_path, overwrite=False)
+
+    if args.direct_import:
         convert_csv_to_json(csv_library_data_path, library_data_path)
-        # should work properly now with fillna()
 
     if args.db_compare:
         compare()