Is there a max folder/file count?

[CraigeryTheKid]

I had 2500 pictures split into 10 sub-folders, and it felt like shuffle was either not getting to all of them, or hitting duplicates in less time than a cycle took. I moved all files to the top /pictures folder today, and my brain is tricking me into thinking "it worked" seeing new pictures.

Was the 2500 total count too high? Or making it shuffle through 10 folders too many?

Have been using picframe for a few months, to the point that I've built a total of 3 now, giving one to my parents for Christmas.
Please and thank you!

[paddywwoof]

Hi @Grapefruit82 I don't think the number of images or folders should be a problem. However there is definitely a "perception" problem with the way that the default config file is set up. The reshuffle_num is set to 1 which means that every run through there is quite a high probability of seeing an image repeated that was only recently shown, and the converse, you will have to wait nearly two complete cycles before seeing a few pictures that appeared early.

This "illusion" has cropped up before #128 and the simple, but non-intuitive, fix is to change the resuffle_num to 10. The show will then appear completely random, and even with a few hundred images you will not be able to perceive the fact that the sequence is repeating.

[CraigeryTheKid]

Thanks for the reply, link to issue #128, and the simple "solution".
My brain being wrong was absolutely on the list, and I'm glad to hear that's the case.

What an interesting study in perception and statistics this turned out to be!

[sapnho]

What I was always wondering about: What does reshuffle = 10 actually do? Does it repeat the order of the same once shuffled list 10 times?

[paddywwoof]

Yes

picframe/picframe/model.py

Line 312 in 5574fde

if self.shuffle and self.__num_run_through >= self.get_model_config()['reshuffle_num']:

whenever the slideshow gets to the last image it increases a count of the number of times it's gone through the list. When that count reaches the reshuffle_num it shuffle it.

This is one of those examples where 'random' isn't the best option. If you have 1,000 images in a slideshow then there is a 50% chance that one of the last 22 images before the shuffle will appear in the first 22 images after the shuffle. Much better to keep the sequence for several run-throughs, in which case there is a 0% chance that any of the last 978 images will appear in the first 22 :)

[sapnho]

Thanks, @paddywwoof. Challenge accepted! 👍 I have just changed it to 10, rebooted, and will watch if my wife will notice a difference during the holidays. She is normally the first one to say that the frame has favorite pictures. I mean, why should the Pi not have favorites? After all, it is just a remarkable machine... :-)

[Tron69]

I do not have the problem that images are displayed twice in quick succession but I do not see images from additional directories.
My images (>30000) are stored in directories. Each directory represents an own year with further directories under it.
Now I start the PicFrame so I get maximum pictures from the first 3-4 folders, so from the years 1999-2001. These are just ~2000 pictures. It seems to me that not all pictures are read in. Pictures from the further folders were not displayed even if I let the software run for a few hours.
With the PictureFrame2020.py version I did not have this problem. After a picture from 1999 I got a picture from 2015 and so on.

In my configuration I have:
shuffle: True
time_delay: 60.0
reshuffle_num: 1

[jgodfrey]

Do you have the ability to look at the SQLite database file? If so, it'd be interesting to see what's there. How many records are in the file table (it should be 1 record per image)? Also, each record in that table has both a last_displayed and a displayed_count field. Those contain the last date/time the image was displayed and the number of times the image has been displayed in total. Does that show anything interesting?

[Tron69]

I have additionally switched on logging and set it to INFO. Here it shows that the reading of the images takes some time. I started yesterday at 14:30. Around 15:40 there are some warnings in the log that it can not convert .heic files. The directory where these images are is the 5th last.
Maybe this is also a hint why only a part of the images is displayed.

During my evaluation of the database I found the following:
Number of entries in the table file: 31203
Number of entries with displayed_count > 0: 1269
Number of entries with displayed_count = 1: 1260
Number of entries with displayed_count = 2: 3
Number of entries with displayed_count = 3: 5
Number of entries with displayed_count = 4: 1

When will the counter be reset? At every restart?
To test the different parameters I restarted the program a few times. Possibly the higher counts come from there, because the parameter reshuffle_num is currently set to 10.

I find it interesting that all displayed images come from the first 38 directories. So the maximum folder_id of the displayed images was 38. If I look at these from the table folder, then in my case these are just the years 1999-2001.

Now that the images have been completely read into the database, I restart the program and see if all images are now mixed.

[jgodfrey]

When will the counter be reset?

It doesn't get reset automatically. Really, it was added as a sanity check when some of us were questioning whether all images were actually getting viewed. Right now, if you have a need to reset it, you'll have to do it manually, via SQL.

To test the different parameters I restarted the program a few times. Possibly the higher counts come from there

Yeah, that'd be my assumption also.

[paddywwoof]

I wonder if there's an error in a try catch that kills one of the threads so no more images get loaded but the playing thread carries on. In some way... What happens if the heic files are isolated? Do you have the heic setup installed?

[Tron69]

My focus here was not on the error that heic files were not loaded. I also did not install the heic extension. I just wanted to show that other directories were read in, but they were not displayed.

As seen in the DB, the approx. 30000 images were inserted into the DB. After a restart, images from all directories are now displayed. I also checked the displayed_count again after several hours and could not find any image that had a counter higher than 1.

Translated with www.DeepL.com/Translator (free version)

[paddywwoof]

Yes, that's what I understood. But it seemed odd that the images would be read from the directories but then not shown. Picframe has various "threaded" loops and a lot of the potentially problem function calls are inside try blocks so something might stop part of the program working properly but the rest carries on, but with something missing. I still don't really know what the problem was, but let us know if anything like it crops up again.

[jgodfrey]

@Tron69 - Just guessing, but I think some of this perception issue may be caused by the way the current "playlist" is built and processed internally. I haven't looked at this code in detail recently, but as I recall, it works something like this:

• Frame is started for the first time, and begins to crawl folders for your (30,000) images
• We (nearly immediately) attempt to build a playlist for viewing using whatever images have been added at that point.
• If nothing has been added to the db yet, we wait a bit and check again (really just waiting for the db to start to build).
• As soon as we have any images to display, we start the slideshow, even though the available image count may be fairly low in the beginning.

Let's say that's 10 images (for example)

• We build a "playlist" out of the available images (10 in the example) and begin to display them.
• While we're displaying the initial 10 images, the caching mechanism continues to run and load more images.
• In your case (I think based on earlier info here), that 10 image playlist will take 10 minutes to be displayed.
• Only after the current playlist completes do we build a new playlist for display.

Now, we should have more images available, but very possibly not all of them (in your 30,000 case).

• So, maybe that 2nd playlist has 1,000 images available. If that's what we have, that's what we use for the next slideshow.
• In that case, we'd start a 1,000 image slideshow that'd take 1,000 minutes (16.6 hours) to run. Again, while this is running, we continue to load images in the background.
• Once that slideshow finishes, we again go back to the db to build a new playlist based on the current image information.
• By now, that may or may not be your entire image list.

The point is, once we create a playlist to drive the slideshow, we don't go back to db or rebuild a new playlist until the current one finishes. So, in your case, that "partial" playlist can take a long time to play through. So, even though we may have finished adding 30K images well before the current slideshow completes, we don't really know if until afterwards - when we rebuild the list.

Additionally, it'd cause the "displayed_count" metric we store in the db to look strange. In the above outline, the images shown in the initial "10 image" slideshow will appear in the 10 image show, the 1,000 image show, and the 30,000 image show. So, those would show they've been displayed 3 times, while images only appearing in the final 30,000 show would only have been displayed once.

When you restart the app, it builds a new playlist using the db's current contents - which might explain why things settled down on a restart.

Bottom line, the slideshow may not be exactly what you expect while the cache is being built. But, once we have a complete cache (AND HAVE REQUESTED A NEW PLAYLIST) it should be "as expected"...

Sorry for the long-winded post, but maybe it's helpful... :^)

[github-actions[bot]]

This discussion has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion for related concerns.