Duplicating PerformConversionsJob in Redis queue S3/DO

[ljanecek]

For some mysterious reason, there are duplicate jobs for the same object, identical configuration of conversions, in the queue.

Their number is not uniform for all objects.

It looks to me like it is checking if the image is generated and saved. Since it is stored in the cloud, they are not available for a while and in the meantime another job is created for the same image.

Until they are available in the cloud, then the next job will not appear. By the time they are all finished, the image is physically resaved in the cloud several times.

Im using spatie/laravel-medialibrary: 10.5.2

The job object is:

   "conversions":{
      "items":[
         {
            "fileNamer":{
               
            },
            "extractVideoFrameAtSecond":0,
            "manipulations":{
               "manipulationSequence":{
                  "groups":[
                     {
                        "optimize":"{""Spatie\\ImageOptimizer\\Optimizers\\Jpegoptim"":["-m80",""--force"",""--strip-all"",""--all-progressive""],""Spatie\\ImageOptimizer\\Optimizers\\Pngquant"":[""--force""],""Spatie\\ImageOptimizer\\Optimizers\\Optipng"":["-i0","-o2",""-quiet""],""Spatie\\ImageOptimizer\\Optimizers\\Svgo"":[""--disable=cleanupIDs""],""Spatie\\ImageOptimizer\\Optimizers\\Gifsicle"":[""-b"","-O3"],""Spatie\\ImageOptimizer\\Optimizers\\Cwebp"":["-m 6","-pass 10",""-mt"","-q 90"]}",
                        "format":"jpg"
                     }
                  ]
               }
            },
            "performOnCollections":[
               
            ],
            "performOnQueue":true,
            "keepOriginalImageFormat":false,
            "generateResponsiveImages":false,
            "loadingAttributeValue":null,
            "pdfPageNumber":1,
            "name":"preview"
         },
         {
            "fileNamer":{
               
            },
            "extractVideoFrameAtSecond":0,
            "manipulations":{
               "manipulationSequence":{
                  "groups":[
                     {
                        "optimize":"{""Spatie\\ImageOptimizer\\Optimizers\\Jpegoptim"":["-m80",""--force"",""--strip-all"",""--all-progressive""],""Spatie\\ImageOptimizer\\Optimizers\\Pngquant"":[""--force""],""Spatie\\ImageOptimizer\\Optimizers\\Optipng"":["-i0","-o2",""-quiet""],""Spatie\\ImageOptimizer\\Optimizers\\Svgo"":[""--disable=cleanupIDs""],""Spatie\\ImageOptimizer\\Optimizers\\Gifsicle"":[""-b"","-O3"],""Spatie\\ImageOptimizer\\Optimizers\\Cwebp"":["-m 6","-pass 10",""-mt"","-q 90"]}",
                        "format":"jpg",
                        "width":"300",
                        "height":"300",
                        "sharpen":"8"
                     }
                  ]
               }
            },
            "performOnCollections":[
               
            ],
            "performOnQueue":true,
            "keepOriginalImageFormat":true,
            "generateResponsiveImages":false,
            "loadingAttributeValue":null,
            "pdfPageNumber":1,
            "name":"sm"
         },
         {
            "fileNamer":{
               
            },
            "extractVideoFrameAtSecond":0,
            "manipulations":{
               "manipulationSequence":{
                  "groups":[
                     {
                        "optimize":"{""Spatie\\ImageOptimizer\\Optimizers\\Jpegoptim"":["-m80",""--force"",""--strip-all"",""--all-progressive""],""Spatie\\ImageOptimizer\\Optimizers\\Pngquant"":[""--force""],""Spatie\\ImageOptimizer\\Optimizers\\Optipng"":["-i0","-o2",""-quiet""],""Spatie\\ImageOptimizer\\Optimizers\\Svgo"":[""--disable=cleanupIDs""],""Spatie\\ImageOptimizer\\Optimizers\\Gifsicle"":[""-b"","-O3"],""Spatie\\ImageOptimizer\\Optimizers\\Cwebp"":["-m 6","-pass 10",""-mt"","-q 90"]}",
                        "format":"jpg",
                        "width":"500",
                        "height":"500",
                        "sharpen":"3"
                     }
                  ]
               }
            },
            "performOnCollections":[
               
            ],
            "performOnQueue":true,
            "keepOriginalImageFormat":true,
            "generateResponsiveImages":false,
            "loadingAttributeValue":null,
            "pdfPageNumber":1,
            "name":"md"
         },
         {
            "fileNamer":{
               
            },
            "extractVideoFrameAtSecond":0,
            "manipulations":{
               "manipulationSequence":{
                  "groups":[
                     {
                        "optimize":"{""Spatie\\ImageOptimizer\\Optimizers\\Jpegoptim"":["-m80",""--force"",""--strip-all"",""--all-progressive""],""Spatie\\ImageOptimizer\\Optimizers\\Pngquant"":[""--force""],""Spatie\\ImageOptimizer\\Optimizers\\Optipng"":["-i0","-o2",""-quiet""],""Spatie\\ImageOptimizer\\Optimizers\\Svgo"":[""--disable=cleanupIDs""],""Spatie\\ImageOptimizer\\Optimizers\\Gifsicle"":[""-b"","-O3"],""Spatie\\ImageOptimizer\\Optimizers\\Cwebp"":["-m 6","-pass 10",""-mt"","-q 90"]}",
                        "format":"jpg",
                        "width":"900",
                        "height":"900",
                        "sharpen":"2"
                     }
                  ]
               }
            },
            "performOnCollections":[
               
            ],
            "performOnQueue":true,
            "keepOriginalImageFormat":true,
            "generateResponsiveImages":false,
            "loadingAttributeValue":null,
            "pdfPageNumber":1,
            "name":"lg"
         },
         {
            "fileNamer":{
               
            },
            "extractVideoFrameAtSecond":0,
            "manipulations":{
               "manipulationSequence":{
                  "groups":[
                     {
                        "optimize":"{""Spatie\\ImageOptimizer\\Optimizers\\Jpegoptim"":["-m80",""--force"",""--strip-all"",""--all-progressive""],""Spatie\\ImageOptimizer\\Optimizers\\Pngquant"":[""--force""],""Spatie\\ImageOptimizer\\Optimizers\\Optipng"":["-i0","-o2",""-quiet""],""Spatie\\ImageOptimizer\\Optimizers\\Svgo"":[""--disable=cleanupIDs""],""Spatie\\ImageOptimizer\\Optimizers\\Gifsicle"":[""-b"","-O3"],""Spatie\\ImageOptimizer\\Optimizers\\Cwebp"":["-m 6","-pass 10",""-mt"","-q 90"]}",
                        "format":"jpg",
                        "width":"1400",
                        "height":"1400",
                        "sharpen":"1"
                     }
                  ]
               }
            },
            "performOnCollections":[
               
            ],
            "performOnQueue":true,
            "keepOriginalImageFormat":true,
            "generateResponsiveImages":false,
            "loadingAttributeValue":null,
            "pdfPageNumber":1,
            "name":"xl"
         }
      ],
      "escapeWhenCastingToString":false
   },
   "media":{
      "class":"Ignite\\Crud\\Models\\Media",
      "id":7,
      "relations":[
         
      ],
      "connection":"mysql",
      "collectionClass":null
   },
   "onlyMissing":false,
   "queue":"images"
}

Do you have an idea?

Thanks.

[freekmurze]

There are no similar reports for this, so I think it is caused by something specific to your projects.

Should you find that the package is in error, feel free to send a PR to fix it.

[ljanecek]

Finally, adding ShouldBeUnique to PerformConversionsJob helped me solve this problem. Do you think it would be a good idea to do a merge request for this @freekmurze ? At the moment I have it as a custom job modified in the configuration.