Needs to handle API throttling and error detection

[TenguTech]

Summary
When importing large sets of photos from Flickr the plugin needs to handle API throttling and gracefully handle errors cause by it. Pausing after X API calls and/or detecting that it has been throttled then restarting later.

Description
I have just imported 3000+ photos from Flickr into a self hosted Piwigo instance.

I found that after about 990 photos are imported, the green photo imported message would change to a red error message with something about file size.

Sorry I don't have the exact message, but I think it was cause by getting Zero length file when trying to retrieve the photo due to API throttling.

If I waited a period of time and started again, I would get lots of green 'we all ready have the image' messages, then green importing new image messages, and finally red messages again when it got API throttled again.

Suggestions
I suggest the following:

1. Pause the Import plugging after a set number of API calls
   I know when I checked the API stats after the first problem it said a bit over 2500 calls in not that long.
   The Flickr API documentation says no more than 3600 calls in an hour.
   So I think the API throttles for a fast rate in a short period of time.

Maybe the Plugin needs a 'self pause' function so it doesn't exceed this.

2. The Plugin seems to need better error detection.
   If the plugin is getting the same error multiple times in a row maybe it should stop?

If you were getting photos and start to get Zero size file it probably means you are being throttled.
And it would be useful if gracefully handled.

Conclusion
I'm glad I don't have the 300,000 photos that needed to be imported by the person who reported issue #30
But trying to import even a few thousand photos seems to be a problem.

I think the plugin needs to self throttle so it doesn't exceed the 3600 calls an hour and exceed what appears to be a short burst limit.
It would also be useful if it detected errors that were caused by throttling and paused or self throttled in response.

For reference, here is the graph I got from having to run the import 3 times today.
The first call was done yesterday.

[TenguTech]

On a related note: The plugin says there are 3791 items to import, It seemed to indicate that all were imported in the last run, but my gallery only has 3707. 2 of which are test images. So there are 86 missing images. I suppose I need to go read logs somewhere?

[samwilson]

The missing ones might be videos.

For logs, I think we should shift from the popup 'growl' notifications to a scrolling log that's visible for longer (whether the log should be kept between runs, I'm not sure, but probably not).

And that's a good idea about stopping the import if there's more than x errors in a row.

[TenguTech]

Plus if there is a 3600 calls per hour limit, the plugin really needs to self throttle.

Not quite sure how many call are required to get a photo.
At lease 2 per photo I'd say. But maybe 3 or 4.
If 3600 exhausts at 990 that is 3.63....

So maybe:

• 1 to pull the list of photos
• 1 per album for the album info
• 1 per photo for the photo info
  • 1 to get the photo itself

I suppose I should go read the code to work this out.

But the main point I'm trying to make is that:
Excessive API calls are going to get people's API Keys banned.

So the plugin really needs to needs to keep track of the API calls it is making and pause for a time when it is close to the limit.
In my opinion it is safer to pause before it hits the limit that pause when it hits the limit.

The plugin could even do the maths required before doing the import.
It knows how many photos and albums need to be imported.
It could calculate how many API calls it thinks it will need to do and tell the user it will do it in X blocks with a pause between them.
Then tell the user this before they click the 'Import All' button.

Yes, my idea is getting more complicated as I write.
I'm just trying to make it more user friendly. :)