Cannot Find Images for this Search Filter

[vk379]

Hey,
I recently noticed, google_images_download is failing to download images at random times.
An example of this would be, if I made a loop to download a certain image, and tried to download it 10 times, it would download it most of the time, but fail at random points. For example, iteration 1 of the loop successfully downloads the images, but iteration 9 fails , for no reason, even though the arguments are kept constant.
Could use any and all help on this issue.
The Error I receive is:
Unfortunately all 50 could not be downloaded because some images were not downloadable. 0 is all we got for this search filter!

Thanks,
-vk379

[S-Cardenas]

I have been seeing this issue also and would love to know what's behind it. Could it be that Google is prevent you from downloading images if it detects the same IP address being used too quickly?

[vk379]

I'm not entirely sure. I fixed the issue personally but my code is really inefficient. I am basically running a while loop until till it works.. which means I am calling google images around 3-4 times till I get the URLs I want. Pretty brute force but does the job, would love to see a solution for this issue though.

[S-Cardenas]

I was thinking of doing the same thing but don't want to brute force it. I'm more worried about the root of the issue being Google related.

[S-Cardenas]

After some digging around, I found the issue to be the result of Google updating the response format. Traditionally, each of Google's image result came in the form of:

<div jscontroller="Q7Rsec" data-ri="0" class="rg_bx rg_di rg_el ivg-i" data-ved="0ahUKEwj0zuvFn_TlAhXJc98KHQC0CAcQMwhkKAAwAA">
    <a jsname="hSRGPd" href="#" jsaction="fire.ivg_o;mouseover:str.hmov;mouseout:str.hmou" class="rg_l" rel="noopener">
        <div class="THL2l"></div><img id="CixhSoPkCojjCM:" src="data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" jsaction="load:str.tbn" class="rg_ic rg_i" alt="Image result for economy chart" data-deferred="1">
        <div class="rg_ilmbg"> 2161&nbsp;&#215;&nbsp;1910 </div>
    </a>
    <a class="iKjWAf irc-nic isr-rtc a-no-hover-decoration" href="#" jsaction="mouseover:m8Yy5c;mousedown:QEDpme;focus:QEDpme;" rel="noopener" target="_blank">
        <div class="mVDMnf nJGrxf">The $80 Trillion World Economy in One Chart</div>
        <div class="nJGrxf FnqxG"><span>visualcapitalist.com</span></div>
    </a>
    <div class="rg_meta notranslate">{"id":"CixhSoPkCojjCM:","isu":"visualcapitalist.com","itg":0,"ity":"jpg","oh":1910,"ou":"http://2oqz471sa19h3vbwa53m33yj-wpengine.netdna-ssl.com/wp-content/uploads/2018/10/world-economy-gdp.jpg","ow":2161,"pt":"The $80 Trillion World Economy in One Chart","rh":"visualcapitalist.com","rid":"vzfo7BtwQ7sOEM","rmt":0,"rt":0,"ru":"https://www.visualcapitalist.com/80-trillion-world-economy-one-chart/","sc":1,"st":"Visual Capitalist","th":211,"tu":"https://encrypted-tbn0.gstatic.com/images?q\u003dtbn:ANd9GcRxpTvHqGYeKsCQATZP0ChgnXw2b4PAzSyBWHkpYNfFE1oqrDi7kg\u0026s","tw":239}</div>
    <div class="ll0QOb"></div>
</div>

This library, google-images-download, then searches for each original image link by searching for rg_meta notranslate in the body of the response. This can be seen in the method _get_next_item of google_images_download.py.

However, Google will occasionally send the response in a different format, such as this:

<div jsaction="IE7JUb:e5gl8b;MW7oTe:fL5Ibf;dtRDof:s370ud;R3mad:ZCNXMe;v03O1c:cJhY7b;" data-ved="2ahUKEwjbwvWwovTlAhWJNN8KHdolChQQMygAegUIARD_AQ" data-ictx="1" data-id="CixhSoPkCojjCM" jsname="N9Xkfe" data-ri="0" class="isv-r PNCib MSM1fd BUooTd" jscontroller="SI4J6c" jsmodel="uZbpBf sB4qxc" jsdata="j0Opre;CixhSoPkCojjCM;7" style="width:179px;" data-tbnid="CixhSoPkCojjCM" data-ct="0" data-cb="0" data-cl="0" data-cr="0" data-tw="239" data-ow="2161" data-oh="1910">
    <a class="wXeWr islib nfEiy mM5pbd" jsname="sTFXNd" jsaction="click:J9iaEb;" jsaction="mousedown:npT2md; touchstart:npT2md;" data-nav="1" tabindex="0" style="height:158px;">
        <div class="bRMDJf islir" jsname="DeysSe" style="background:rgb(248,69,133);width:179px; height:158px;" jsaction="mousedown:npT2md; touchstart:npT2md;"><img class="rg_i Q4LuWd tx8vtf" src="data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" data-iid="12" data-iurl="https://encrypted-tbn0.gstatic.com/images?q=tbn%3AANd9GcRbMq0HH9-11adBbwZ2LP9hFwRpBoe6UbFGgUvr2zNe0O5dhxyp" jsname="Q4LuWd" alt="Image result for economy chart" /></div>
        <div class="c7cjWc"></div>
        <div class="h312td  RtIwE" jsname="bOERI">
            <div class="O1vY7"><span>2161 × 1910</span></div>
        </div>
        <div class="PiLIec" jsaction="click: gFs2Re"></div>
    </a>
    <a class="VFACy kGQAp" data-ved="2ahUKEwjbwvWwovTlAhWJNN8KHdolChQQr4kDegUIARCAAg" jsname="uy6ald" rel="noopener" target="_blank" href="https://www.visualcapitalist.com/80-trillion-world-economy-one-chart/" jsaction="focus:kvVbVb; mousedown:kvVbVb; touchstart:kvVbVb;">
        <div class="sMi44c lNHeqe">
            <div class="WGvvNb">The $80 Trillion World Economy in One Chart</div>
            <div class="fxgdke">visualcapitalist.com</div>
        </div>
    </a>
</div>

This type of response can not be parsed correctly by the library because the library is specifically looking for rg_meta notranslate to find the original image link. In this updated response, the rg_meta notranslate tag and original image link are absent from the element. The original image link is being provided in a javascript callback AF_initDataCallback.

It's not clear why Google is sending two differently formatted responses. It's also not clear if they will continue to do so in the feature. This particular "bug" could be fixed via pull request but we the community should be mindful of any future changes Google implements to the response format.
@vk379 @hardikvasa

[vk379]

Wow this is an amazing discovery. This makes sense, as to why I am getting so many errors. I think recently google themselves have unrolled a lot of updates, specifically dealing with Google Search and the implementation of a new NLP software called BERT. Maybe they changed a little bit of the code on their end during this which could be causing these changes.

[S-Cardenas]

It seems like they're slowly or partially rolling it out since only some of their responses are different.

[pixelicous]

I just started using this library, I never manage to get to download anything. Tried also downloading the chromedriver and specifying it in the attributes.. Nada.. also tried looping till i get downloads, at the third try it always gets hung on "Evaluating", restarting the script seem to get the loop rolling again, till the third try..

Unfortunately all 100 could not be downloaded because some images were not downloadable. 0 is all we got for this search filter!

[S-Cardenas]

@pixelicous Could you check if the HTML being returned from google matches either of formats mentioned in: #280 (comment)

[vk379]

This issue just got worse, my aforementioned method of looping until I get the URLs isn't working now, as the URLs are almost never coming up. This is a huge issue. I'm not sure if it's even being worked or not though :(

[galfaroth]

I started to use this lib yesterday and was also unable to download almost anything. Sometimes some keywords worked, but some simple ones like "city" did not succeed.

[pixelicous]

@S-Cardenas Could you point me on how to get the html returned using the library?

From the other replies it seems I am not the only one 😿

[AlNik999]

the same here. Can't load even a single apple image

[S-Cardenas]

@pixelicous I believe the s in https://github.com/hardikvasa/google-images-download/blob/master/google_images_download/google_images_download.py#L714 is the raw html that is returned from the server.