Images / Photos

[klemens-u]

Hi,

very interesting project! What are your thoughts/plans for images?

I'm looking for a semantic search and structuring solution for my Nextcloud images.

Thanks!

[cbcoutinho]

Hi @klemens-u thanks for your interest in the project!

Currently the Nextcloud MCP Server supports images via basic webdav tools, which means that agents can download image content and read them. Semantic search is primarily focused on text-based files, where some files (e.g. PDFs) can be transformed into text before text-embedding to enable semantic and/or keyword search.

Images are not a semantic search target; however, this is primarily because I haven't considered this feature nor spent time to implement it. What do you have in mind regarding "semantic search and structuring" for images stored in Nextcloud?

[cbcoutinho]

@klemens-u I'm currently working on adding semantic search to images, let me know if you have any particular requests regarding this feature and I can try to incorporate them prior to merging the PR

[patatosorus]

Hi,
I'm joining to this issue/question from klemens.
In my case, I am lookking for a way to expose specially memories coupled with recognize that I have worked to get a beautiful and well indexed collection of my pictures and videos.

Thanks to recognize and memories, you can name and cluster people on the pictures and videos.
And I think it would be nice to be able to search through the metadata we've built on.

Have you planned to add memories (dont think recognise is really necessary) ?

[cbcoutinho]

Hi @patatosorus thanks for your input

I have been working on a branch that parses photos accessible from the default files app, see pr. Although I use memories to view my photos, i haven't checked out it's API for a potential integration - if you have any tips regarding how you'd like this work I'm willing to listen.

Btw, image embedding is an expensive proposition. Therefore it's much more difficult to test locally and requires either an API (I'm testing with AWS Bedrock) or a GPU with an image/multimodal embedding model.

If you or anyone lurking this thread had any tips here, I'm all ears.

[patatosorus]

Thanks for the quick reply !

I know from my researches that memories has a intern API for the nextcloud frontend.
In my case, I have around 120 Go of pictures. I ran a embedding once, it cost me a certain time and CPU usage but now, it configured to embed the images during the night and it doesn't seem to bother my server.

I imagine the use as : You ask your agent a photo of a friend according to a specific event and thanks to the memories api, it finds them pretty easily.

For the embedding pb you mentionned, dont you already use an embedding model for your rag ?

Note : I dont if you already tried it, but I think recognize is designed to be CPU friendly.