Library Auto Updater in loop

[bananabob]

Using the Library Auto Updater add-on) and it is fine when updating the audio library but when it updates the video library it seems to go into a loop.
I have run it with debug on and pasted the results here https://paste.osmc.tv/kigadokita 1
The loop constantly displays messages on the screen (when the add-on is run manually) Saying Compressing database, and searching for data on the TVDB database.
The log shows these messages when running using the “RUN” option on the Settings>Add-on browser>Library Auto Update screen and repeats them for ever.

2021-10-27 11:26:58.064 T:866 INFO : VideoInfoScanner: Starting scan …
2021-10-27 11:26:58.266 T:866 INFO : VideoInfoScanner: Finished scan. Scanning for video info took 00:00

When I invoke an update of the library from the context menu the update runs as expected with no issues

2021-10-27 11:08:27.154 T:729 INFO : VideoInfoScanner: Starting scan …
2021-10-27 11:08:27.491 T:729 INFO : VideoInfoScanner: Finished scan. Scanning for video info took 00:00
2021-10-27 11:08:40.554 T:729 INFO : VideoInfoScanner: Starting scan …
2021-10-27 11:08:40.740 T:729 INFO : VideoInfoScanner: Finished scan. Scanning for video info took 00:00
2021-10-27 11:08:43.160 T:739 INFO : initializing python engine.
2021-10-27 11:08:45.765 T:739 INFO : CPythonInvoker(15, /home/osmc/.kodi/addons/script.skinshortcuts/default.py): script successfully run
2021-10-27 11:08:46.380 T:739 INFO : Python interpreter stopped

Kodi (OSMC) is running on a Raspberry Pi The files for the library are held on another Raspberry Pi acting as a file server. I have posted this on both the OSMC and KODI forums but they have been of no help.

Thank you.

[robweber]

What timers do you have setup as part of the schedule and what is the interval? The log file says it's only creating one scheduled timer for the Music Library. Is that correct?

[bananabob]

I have created a series of screenshots that, hopefully, give you the information you require, and demonstrate the issue.
Note that although the screenshot for the video library settings show that it not to run, the problem occurs even when the settings allow it to run. In other words, a video library update runs even if the settings say don't.
Here is the dropbox link for the screenshots.

[night4awk]

I had the same issue (LibreELEC-RPi4.arm-10.0.1.img - Kodi 19.3)

Workaround: I downgraded the "Library Auto Update"-Addon to Version 1.2.1

[bananabob]

Where do I find 1.2.1?

[night4awk]

Here u go:

https://mirrors.kodi.tv/addons/matrix/service.libraryautoupdate/

[bananabob]

Reverted to 1.2.1 and problem gone away. Thank you

[night4awk]

Ur welcome ;)

[robweber]

I'm really at a loss for what could be causing this - especially given that a downgrade seems to fix it. Comparing the actual changes between those two versions there is very little code that is actually different. It's mostly changes to be compatible with Kodi library changes. I've been unable to duplicate on any of my main or test systems.

matrix-1.2.1...matrix-1.2.3

[peterjanes]

I had the same issue with 1.2.3 and disabled it until I could investigate. Tried turning it on a few days ago using the same basic cron schedule and without downgrading, to see if I could get you a debug log, but it's been working without issue.

Is it possible it was crashing on a bad file or other error? I have done some filesystem cleanup in the last couple of weeks, and had noticed when I ran manual updates that they'd also crash, sometimes taking Kodi down with them.

[bananabob]

I have included the log from my 1.2.1 run last night. As you can see there are some errors relating to files in the music database and the video database.
In the video folders:
There is no file The Complete Truth About De-Evolution (2003) etc
There is no directory /mnt/helium/audio/TV Shows/DWTV/
What is /home/osmc/.kodi/media/Fonts/?

In the music folders you can see that there is a couple of huge SQL errors.

Could either of these be causing the issues with 1.2.3?
How do I remove the issues?

video.txt
music.txt

[robweber]

I don't think these would hurt the addon since the errors are related to Kodi itself. They're definitely not helping the scan complete efficiently though. Just based off what is in the log files here is my two cents:

It seems that /mnt/helium is just not available during the scan, or at least those directories aren't. Is it possible they no longer exist or the source is offline?

The /home/osmc/.kodi/media/Fonts directory is a Kodi directory that holds font files. Not sure why this would be part of your scan. It might be worth just going through your video sources and seeing what things you've set to scan content. It could be you set some higher level directory (like /home/osmc/.kodi/media?) and this is getting picked up too. You could just remove the Set Content flag from those directories and only add the ones that actually contain video files. If you have a way of accessing the database directly you can run a query to show all paths that have content set. This might help narrow down which ones you can remove. The query you'd need is:

SELECT * FROM `path` where strContent != ''

The SQL error is a unique constraint being violated. The field album.strMusicBrainzAlbumID must be unique per entry and the value being set (0aad28b0-91c3-48b9-8a09-a196dd7b9763) must already exist somewhere else in the database.

[bananabob]

@robweber Thank you that gives me a lot to consider.
I can confirm that those directories no longer exist.
Also I note that last nights log contained the following:

2021-12-03 01:02:17.667 T:14527 ERROR : GetDirectory - Error getting /mnt/helium/audio/Music/Collection/B/Bad Plus/
2021-12-03 01:02:17.668 T:14527 ERROR : GetDirectory - Error getting /mnt/helium/audio/Music/Collection/B/Bad Plus/Prog/
2021-12-03 01:02:17.668 T:14527 ERROR : GetDirectory - Error getting /mnt/helium/audio/Music/Collection/B/Bad Plus/These Are the Vistas/

Which also appeared in the previous days log. Am I right in thinking that the "clean" option is not working correctly, or are all my databases corrupt in some way?

[bananabob]

Extra comment following my last post. I ran the "clean" option from the "settings" options in the KODI menu and the errors in my last comment have gone. So I assume I am correct in thinking that the "clean" option is not working correctly in the Library Auto Updater.

[bananabob]

@robweber Ignore my meanderings. The clean option is working as it should.

[DaveTiff]

I had the same issue, running Libreelec 10.0.1 with Confluence theme on a Pi4. Installed Library Auto Update from the repository.
Manually started an update and it just kept looping. So uninstalled it. Found this and installed version 1.2.1, now working as expected.
Re Dave

[robweber]

I'm really a loss for what could be different between the two versions that causes a problem like that. Doing a compare from 1.2.1 to 1.2.3 all the differences are purely cosmetic (settings updates, language issues, etc). The only code changes were to tweak arguments related to Kodi API changes.

Personally I've never been able to recreate it but if I ever do I'll revisit.

[markouustalu]

I have 2 different devices, a philips android tv with 19.3 kodi on it and 1.2.3 Library auto update on it and RPi4 with 19.4 kodi (librelec 10) and with 1.2.3 Library auto update. BOTH loop IF i choose manual Library update AND library cleaning is set. If i do not set library cleaning then looping does not occur. I waited for automatic triggering as well and it appeared to loop for a while and then stop with no library cleaning happening although it was set. I hope this helps. Both devices are still in this config and i am happy to provide more information and do some further testing if needed.

[robweber]

I'll be honest - I still can't duplicate this but I do have a theory as to why it might be happening. The way the manual trigger works is to "trick" the system into thinking it's time to update by resetting the last update time. I'm going to have to trace out the flow but I'm thinking this may result in some weird conditions that cause looping. If that's the case I can re-work the manual trigger to work differently.

[markouustalu]

Well, i could let you in so you could see it and try to debug it?

…

On Fri, 15 Jul 2022, 23:34 Rob, ***@***.***> wrote: I'll be honest - I still can't duplicate this but I do have a theory as to why it might be happening. The way the manual trigger works is to "trick" the system into thinking it's time to update by resetting the last update time. I'm going to have to trace out the flow but I'm thinking this may result in some weird conditions that cause looping. If that's the case I can re-work the manual trigger to work differently. — Reply to this email directly, view it on GitHub <#69 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALW7HKZABZEWR3CN6MML5MDVUHDO5ANCNFSM5HAXR5XA> . You are receiving this because you commented.Message ID: ***@***.***>

[benyjr]

Out of curiosity, do the devices that have looping issue "go to sleep" so to speak? I have the latest version of the Library Auto Updater installed on a Ubuntu desktop that is always on and has Kodi always running. I also have it installed on an Nvidia Shield Pro which goes to sleep when not in use. The Ubuntu computer has no problem, but the Nvidia Shield did experience this issue. Could the addon be "banking" library scans? For example if while the shield is sleeping, it misses three scheduled scans, when it wakes up is it processing all the missed scans?

[markouustalu]

No. I installed on clean devices and they started looping right out of the box. After that i tried reboots and waiting all night. Did not change the behaviour.

[timkgh]

Same problem here since I updated the Google Chromecast to Android 12. On a 2018 FireTV stick 4K it works fine, same Kodi 19.4 and latest addon version. The Android version/env might have something to do with it.

On the Chromecast I tried turning off automatic updates of the Video and Music libraries and just leaving the Clean function enabled so it runs after I manually update my library but it still causes the updates to be stuck in a loop, even when I start a manual update from the video library, not via the addon.

It's definitely a problem with the addon, when I disable it the manual update is well behaved, no loop.

[timkgh]

It actually also does it on a brand new Fire TV stick 4K Max, fresh install of Kodi 19.4

[BadIDMan]

Latest update 1.2.4 is not respecting cron input.
I have set to run video library update every day at 4:30:
30 4 * * *

but in fact in the log I see it is starting EVERY second hour:
2023-04-13 04:00:00.901 T:762 info : VideoInfoScanner: Starting scan ..
2023-04-13 06:00:00.962 T:5294 info : VideoInfoScanner: Starting scan ..
2023-04-13 08:00:00.776 T:9891 info : VideoInfoScanner: Starting scan ..

Also why it is trying to scan folders that are deleted? Which table in database is the source of folders to be scanned?

2023-04-13 08:00:30.241 T:9891 warning : Process directory 'nfs://192.168.1.101/volume1/NAS/SERIES/ABC/' does not exist - skipping scan.
2023-04-13 08:00:30.242 T:9891 warning : Process directory 'nfs://192.168.1.101/volume1/NAS/SERIES/DEF/s01/' does not exist - skipping scan.
2023-04-13 08:00:36.327 T:9891 warning : Process directory 'nfs://192.168.1.101/volume1/NAS/SERIES/GHI (2001)/' does not exist - skipping scan.

[Bisonator]

Hi

I today had the OP's issue on my fresh install of Kodi v20.2 and Library Auto Updater v1.2.4: When running Library Auto Updater, it would finish an update, perform a clean, then start from scratch and perform another update, immediately.

What fixed it for me (for the video library at least) was to:

1. disable the "Clean Libraries" option under "Cleaning" in Library Auto Updater
2. keep the following in my advancedsettings.xml:

<videolibrary>
	<cleanonupdate>true</cleanonupdate>
</videolibrary>

I now don't get the looping when manually (or automatically) running Library Auto Updater...
AND I get my video library cleaned after each library update, which is how I want it to work.

I've yet to confirm if the following works for music libraries (because an update takes ages and I couldn't yet find documentation confirming if this exists or not):

<musiclibrary>
	<cleanonupdate>true</cleanonupdate>
</musiclibrary>

If the above does work for the music library too, then I'd consider it a sufficient workaround for the issue.

Additional:

1. Before today's reinstall I was running kodi vMatrix (can't remember exactly which build) with Library Auto Updater v1.2.4 and didn't get this problem. I also had "cleanonupdate" set to true in advancedsettings.xml on that previous install, so not sure why it's behaving differently now. Presumably changes to kodi's code for vNexus has caused it. (Btw, I'm running an OSMC Vero 4k+ using their build, so it's possible their code has introduced it, but unlikely.)

2. The only other significant change I did is to have swapped to the "The TvDB v4" scraper from the older Team Kodi one.

Cheers

B

EDITS: to get my code snippets displaying correctly - apologies for all the notifications if you got them! (Always learning)

[primehalo]

Just had this endless loop happen on a fresh install of Kodi 20.2.0 on an Nvidia Shield TV Pro with no sources set up yet:

https://youtu.be/DsX0dd1Xv84

Disabling the Clean Libraries option got it working correctly.

[timkgh]

Maybe this project is in need of a new maintainer. There hasn't been any real code activity in a couple of years.

[robweber]

I had attempted to fix this a while back and just never completed it. Looking at it again I think I was making this way to difficult in my mind. Below is an attempt at a fix, ultimately I just moved a few things around. The main issue, that I could discover, was the double creation of the Monitor class that triggered the Clean Database code. I think moving this so it only gets created once - with the service - might do the trick. Anyone brave enough give it a try: https://github.com/robweber/xbmclibraryautoupdate/tree/refactor_manual

[primehalo]

I'd feel okay testing it but I don't have that Nvidia Shield on which it was happening. I could try it on my own Nvidia Shield but I don't know how to install it outside of using the repository.

[robweber]

I don't think this issue is specific to Nvidia Shields. I'm not super concerned it will break anything but confirmation it will fix this issue would be good.

[timkgh]

@robweber Can you make a release please? It can't be any worse than the current release which doesn't work at all.

[DaveTiff]

Hi, updated to latest Libelec on a Pi 4, just updated to Library Auto Update 1.2.4 and still if I turn on the Clean Option (which is really what I want) it still gets stuck in a loop. Looks like it doesn't even get to cleaning part, just update library and immediately update library again. This carry's on in the background until you uncheck the Clean option and press ok.

My loop looks different than the video (primehalo) above

[robweber]

Pushed a release with the changes to the Kodi repo: xbmc/repo-scripts#2529

[DaveTiff]

When will this be available? or can I download a copy and install form zip?

[robweber]

You can definitely download a ZIP from GitHub. Click the Code button and click download zip. Waiting on Kodi Devs to approve the merge so no idea on availability through the normal Addon updates. Usually takes a week or so from submission depending on how busy they are.

[DaveTiff]

Installed the zip, and no change, still gets stuck in a loop when clean option is selected, very frustrating. Let me know if there is anything I can try.

[melorin83]

Is installing the working old 1.2.1 version nowadays a bad idea? Would I miss any important function of the newer versions?

[robweber]

Just want to confirm that anyone seeing the looping behavior using the new version (1.2.5 - currently only available via a ZIP file on GitHub) is trying this with a manual scan? To test this properly I'd install the update and reboot Kodi to clear out any errant processes.

In my testing the manual scan option - with cleaning enabled - was the only time I could duplicate the looping behavior. The fix in 1.2.5 was to address that. If the program is looping still during scheduled runs I've never been able to duplicate that. I just did several tests again now and all had the library update concluding and then the clean operation triggering. Everything ran and exited properly.

Just want to make sure all the comments here are testing the same things on the same most recent version.

[DaveTiff]

Have turned off and restarted my Kodi Pi, tried manual and auto and still gets stuck in a loop.

[DaveTiff]

Can I install an old version that does not have this problem?

[robweber]

I've heard that 1.2.1 doesn't do this but looking at the code differences between the two it doesn't make sense why one would work and the other not. Most of the changes are Kodi API or language fixes. The biggest changes were from 1.2.4 to 1.2.5 and those I just did.

I'm not denying that it happens but I can't reproduce it or point to a single thing that would cause it. The actual program loop going on here is fairly simple.

1. Check if it's time to run an update
2. If it is trigger Kodi to run the scan
3. When scan completes check if we should clean the library
4. If we should then tell Kodi to do that too

I'm open to ideas but just run out of places to look.

[DaveTiff]

Just installed 1.2.1 and all working fine, happy now

[melorin83]

Where can I find 1.2.1?

[DaveTiff]

Where can I find 1.2.1?

Click on matrix

[timkgh]

Sounds like you can just release 1.2.1 as 1.2.6 and call it a day :)

[robweber]

I only wish it was that easy. Comparing 1.2.1 to 1.2.4 (as 1.2.5 was an attempt to fix this again) there are 70 files changes. 2 of them are actual Python files. The rest are language translations, the README, addon.xml or other non-script files that don't affect the service at runtime. Of the 2 changed Python files the changes amount to a tweak to the custom paths editor and how the settings for the timer are indexed. Nothing that would indicate some kind of looping behavior.

I'm going to leave this open in case new information surfaces that points in a better direction. For what it's worth I run this on my home system (Nexus) on a 2 hour scan, with cleaning enabled, and have never seen this happen.

[timkgh]

That's really strange indeed. FWIW, I run it on both a Google TV w/ Chromecast and FireTV Max and it happens on both. Both are recent clean installs of latest Kodi. Something is going here, either platform dependent or user settings.
Maybe we need a build with good debug messages/tracing?

[electr1cBugaloo]

Another data point - I just had this happen to me on a fresh OSMC install on a Vero V, addon version 1.2.5 and Kodi 20.5. I also deleted a lot of movie files / folders and moved things around prior to the occured looping. Cleanup was enabled in the addon settings. Disabling cleanup in the addon settings seems to have stopped the loop and as suggested above, I instead enabled cleanonupdate in advancedsettings to achieve the desired behaviour of clean up.

[YucksueY]

I seemed to have got it working after experiencing the same issues as others.

It is my first time posting in this type of forum so bear with me as I think it best that give as many details as possible because I don't have enough technical knowledge to know what might be important.

Okay... so after some fiddling, I noticed that if I chose to only clean up Videos, running a manual update completed successfully.
I went into settings and notice that I had made a mistake in the Music cron expression by leaving out a space (*/15* * * *).
I did not see it kick up an error at that point, but what I realised is that the looping seemed to have started only after I had made that mistake.

So, I went into settings and deleted the entire Music cron expression first, making sure I did not leave any unseen spaces, and then carefully set it to run every minute.
I then set it to cleanup Both libraries. Clicked Okay.
I allowed it to auto update. It started to loop.
I noticed the addon notification stuck saying that it will run in 5 mins, every time the interface flashed. I found it strange that it did not show it as 1 min.
After about 30sec I disabled the addon to make it stop looping.
Restarted Kodi.

I went back into settings, disabled Use* Advanced Timer. Clicked Okay.
Went in again and changed the time just to make sure the setting registered. Clicked Okay.
Ran a manual update. Completed Successfully.
Restarted Kodi.

Went back in and correctly entered the 15min cron expression. Clicked Okay.
Ran a manual update. Completed Successfully.

Kodi ran for approximately 2.5hrs with no looping.
Both Libraries were updated and cleaned successfully via auto update.
Restarted Kodi.

Mustered up the courage to change the time settings in various ways for each library.
Restarted Kodi randomly in between. Both manual and auto update completed successfully each time.

I really hope this helps.

p.s. PC; Windows 11 Pro; Kodi and all addons are current

[robweber]

Thanks for the detailed steps. I've never successfully been able to reproduce this behavior on my own. Even just now with a Win 11 computer I couldn't get the looping behavior to occur. I did however find an issue with the cron expression parsing based on your instructions.

There are some situations where the cron library throws a different type of error other than a ValueError, which is the one I was checking for. I've added some additional checks and better error handling in these situations to avoid cron syntax issues.

I'll continue to watch for this but it seems either system dependent or there are a series of circumstantial timing things going on I just can't seem to recreate.

[Vislike]

Throwing in some cents, i currently have the same problem stuck in loop. RPi 3, libreelc kodi 21.1 autoupdater 1.2.5. Media sources using webdav. Using clean library after update.

From debug log i can see:

2025-01-03 13:39:32.183 T:1325    debug <general>: service.libraryautoupdate-1.2.5 : update timers
2025-01-03 13:39:32.187 T:1325    debug <general>: service.libraryautoupdate-1.2.5 : Creating timer for Video Library
2025-01-03 13:39:32.187 T:1325    debug <general>: service.libraryautoupdate-1.2.5 : 24
2025-01-03 13:39:32.190 T:1325    debug <general>: service.libraryautoupdate-1.2.5 : Created 1 schedules

Looks fine, i have a cron schedule of once a week.

I trigger manual update:

2025-01-03 13:39:33.813 T:1325    debug <general>: service.libraryautoupdate-1.2.5 : Update Library Manual Run...
2025-01-03 13:39:33.902 T:1325    debug <general>: service.libraryautoupdate-1.2.5 : checking: /storage/videos/
2025-01-03 13:39:33.902 T:1325    debug <general>: service.libraryautoupdate-1.2.5 : checking: /storage/tvshows/
2025-01-03 13:39:33.902 T:1325    debug <general>: service.libraryautoupdate-1.2.5 : checking: davs://_redacted_/Media/Movies/
2025-01-03 13:39:35.591 T:1325    debug <general>: service.libraryautoupdate-1.2.5 : checking: davs://_redacted_/Media/TV%20Shows/
2025-01-03 13:39:35.720 T:1325    debug <general>: service.libraryautoupdate-1.2.5 : checking: addons://sources/video/
2025-01-03 13:39:35.730 T:1325    debug <general>: service.libraryautoupdate-1.2.5 : Update Video Library
2025-01-03 13:39:35.730 T:1325    debug <general>: service.libraryautoupdate-1.2.5 : {"jsonrpc": "2.0", "method": "VideoLibrary.Scan", "params": {"showdialogs": true}, "id": 44}
2025-01-03 13:39:35.905 T:1308     info <general>: VideoInfoScanner: Starting scan ..

Update finishes and it starts again:

2025-01-03 13:41:00.237 T:1308     info <general>: VideoInfoScanner: Finished scan. Scanning for video info took 84356 ms
2025-01-03 13:41:00.243 T:900     debug <general>: service.libraryautoupdate-1.2.5 : Cleaning Database
2025-01-03 13:41:00.244 T:900     debug <general>: service.libraryautoupdate-1.2.5 : {"jsonrpc": "2.0", "method": "VideoLibrary.Scan", "params": {"showdialogs": true}, "id": 44}
2025-01-03 13:41:00.543 T:1340     info <general>: VideoInfoScanner: Starting scan ..

We can see here that it triggers a VideoLibrary.Scan again.
What i found out so far is that this is the default behavior for the CronSchedule() class, created here:

xbmclibraryautoupdate/resources/lib/service.py

Line 326 in 5eba6f2

aJob = CronSchedule()

And my best guess is that none of the ifs on 329 or 332 is true. My setting is "both" so the first part of the ifs should be true in both cases, leaving database == 'video' or database == 'audio'. So my guess database is neither video or audio in my case, and the default behavior of CronSchedule() a VideoLibrary.Scan is run and the loop is on.