Question: Why "Creating directories."?

[LuminairPrime]

I just completed a full run of TwitchClipper. I see that the console output says "creating directories", and then a bunch of empty directories are created. Some are named after clips, many are duplicates with "#" at the end, and some are named nonsensically like "2BEO51~O". It looks like the ones ending with "#" have clip videos in them. But there are also many clip videos in the root directory.

Is any of this a bug? I expected to have just ONE directory with all clips.

I'm running the app over SMB on a NAS. Not sure if that would cause this. I can test locally next time.

[mortenmoulder]

I can explain the weird names, by sending you this link: https://superuser.com/questions/458995/files-folders-get-weird-names-and-become-inaccessible-on-samba-share

It's called file name mangling and it's an SMB issue. I've had the same issues on my end as well. I can't do anything about that, unfortunately.

I don't understand why it would create a bunch of empty directories. It creates a folder dependent on your custom save expression, but it should only create that folder if there's going to be a clip saved in that folder as well. What is your custom save expression and what parameters did you input (streamer, start/end)?

[LuminairPrime]

1. I will try to isolate the cause of the extra folders and report back. Maybe they are related to running the .exe over the network. I did this because I couldn't find a way to run the .exe locally and choose the save location. Would I do that in the config file? Like //share/location/{broadcaster_name}/file.mp4? Or Z:/location/{broadcaster_name}/file.mp4? Alternatively, the design of the download location options could be changed to: read from the config file for the filename, and read from the command line for the download location.

1b) The pattern I used is below. I just tried "/{id}.mp4" quickly, and I don't see any folders created, so that must mean a bug in my pattern:
"Download": { "SavePathExpression": "/{broadcaster_name}/{yyyy}{MM}{dd} - {viewcount} - {title} - {broadcaster_name} {id}.mp4", "Locale": "en-US" }

2. I will research the topic of samba "mangled names=no" for my Synology NAS. I wonder if this is a workaround rather than the cause of the problem. I can suggest to Synology that they enable this by default in any case.

3. Yt-dlp has filename options which may help prevent errors. Not suggesting we use this yet, but it's an option, and I don't think much would be lost if the filenames had strange characters stripped out.

--restrict-filenames Restrict filenames to only ASCII characters, and avoid "&" and spaces in filenames --no-restrict-filenames Allow Unicode characters, "&" and spaces in filenames (default) --windows-filenames Force filenames to be Windows-compatible --no-windows-filenames Make filenames Windows-compatible only if using Windows (default)

So if necessary, as a last resort, one could use --restrict-filenames and --windows-filenames all the time no matter what.

Thank you again for your responsiveness!

[mortenmoulder]

@LuminairPrime I create the folders in C# not via yt-dlp. yt-dlp is ONLY used to save the files in the folders created by C#.

The save expression you listed should create 1 folder (the broadcaster's name) and then 1 file per clip inside of that folder. Which other folders are created? Except those Samba errors.

Unfortunately there's no way to write an absolute path save expression. It will always save in the same directory as the executable. Honestly it saves me a lot of trouble, and it's a lot more clean to be honest.

[LuminairPrime]

I ran a quick test on the channel xQc and ctrl-c-cancelled after a few files were downloaded.

Examples of folder names on NAS:

Examples of folder names on windows computer, mounted on T:, in the clips\xQc folder:

[mortenmoulder]

@LuminairPrime Okay so that problem you see there is because of illegal characters. I've tried to fix illegal characters before, by replacing the forbidden characters with an empty space, but there are literally so many that it's impossible. Try creating a file that contains : for example.

This problem does not exist on Linux though, where you don't have that many limitations.

[LuminairPrime]

That's where the filename options with yt-dlp come in! Try one or both of these in your command calls to prevent yt-dlp from ever exposing weird characters: --restrict-filenames and --windows-filenames

[mortenmoulder]

@LuminairPrime Okay I see the problem now. One of the clips had an / in it, which I replace with a \ on Windows, which creates a folder. However, I shouldn't do that on the custom save expression, because that's entirely up to you if you want it that way. I moved the logic a bit around, and I also made yt-dlp create the folders (something youtube-dl could NOT do).

Since / is illegal in Windows, it will now be replaced with nothing (as per the yt-dlp documentation).

I've created a new release - try it out! :)

[LuminairPrime]

Ah ha, good detective work! The old "different slash for a different platform \ /" trick, which then compounded into also creating a folder, lol! 😄

Just tried the new version and got this error:
Downloading clips made by xqcSystem.ArgumentNullException: Value cannot be null. (Parameter 'path2') at System.IO.Path.Combine(String path1, String path2) at TwitchClipper.Services.HostService.GetDownloaderExecutablePath() at TwitchClipper.Services.DownloaderService.CheckExecutableExists() at TwitchClipper.Application.Run(Options options) at TwitchClipper.Program.Main(String[] args)

[mortenmoulder]

@LuminairPrime That could be because of appsettings.json. I have changed "YouTubeDL" to "Downloader" as well as the new paths. You need to replace appsettings.json as well as the .exe file (or just copy the top part with the new yt-dlp settings).

[LuminairPrime]

Good news! That seems to work! I'm going to run a full download and I'll update you after!! 🥳

[LuminairPrime]

Update: The download PARTIALLY worked. I'll explain it here unless you want me to start another issue ticket.

Here's the output of what I ran with a different streamer:

Clips found: 13711 - Page 333/334
Found a total of 13711 clips and 0 already exists. Downloading 13711 clips.
Is your locale supported? Check https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c and scroll down. This is case sensitive.

That's it. But when I check the output folder, only 2446 clips were downloaded. So I suspect this "locale supported" message at the end hints that the download algorithm ended early for some reason. I checked the log, and the last result appears normal. Maybe the next clip after that had a weird name that caused an error? I don't know how to check for that.

I'm gonna run some more tests...

Edit: I ran a download test with the date range set to include the last clip from the above failed download, and hundreds on either side. This test completed successfully without that "locale supported" error. So I'm not sure what could have caused it to happen.

PS: I see an off-by-one error in both the "Page 333/334" line, and in the final output. In this last test I ran, the output says +1 more clips than were actually downloaded.