[metascraper-amazon] Image selector matches incorrect image

[agchou]

I'm running into issues with the image value not being the main image for metascraper-amazon. There are actually multiple .a-dyanmic-image classes on the screen as seen in the attached photo. Can we create some rules with priority over this like wrapUrl($ => $('#landingImage').attr('src')) or wrapUrl($ => $('.a-dynamic-image').first().attr('src'))?

[Kikobeats]

yeah, of course, just add the right rule here:
https://github.com/microlinkhq/metascraper/blob/master/packages/metascraper-amazon/index.js#L51

Can you specific the URl for creating a unit test?

[Kikobeats]

Hey @agchou, I think you create your own package for support this new custom rule.

Can you share with us? I want to improve this in the metascraper-amazon package 😄

[Kikobeats]

Happy to accept improvements over metascraper-amazon; I'm going to close the issue since it's old; If the package looks outdated for you, please ping me!

[andrewdavidkay]

Yea @Kikobeats I get just a tiny 1 pixel image every time. How do we go about fixing this? Can the rule be overridden?

[Kikobeats]

@andyk2177 need to add the specific rule for contemplating that case.

Please, share the URL that is causing this behavior.

We can add a code ward to don't consider images with less than N pixels.

[andrewdavidkay]

Well, seems to be any Amazon link for me that is doing it but here is an example https://www.amazon.com/JNH-Lifestyles-Canadian-Hemlock-Infrared/dp/B00F2Y5B6W?tag=profiledotim-20

[andrewdavidkay]

I think we might just need a more specific class name to grab maybe?

[Kikobeats]

@andyk2177 yes, you're right, the problem is Amazon has a lot of different product views; need to setup the rules in a way we can maximize get the proper image.

Can you make a PR? Just you need is to add the specific image selector here.

[andrewdavidkay]

Ok sure, why are there two selectors though? Which one is prioritized? So for example with my url above I get this image back but the page does have a data-old-hires attribute so not sure why that one wasn't prioritized?

[Kikobeats]

the best way to determinate that is adding a test per every link and be sure the output is the thing you expect

[bobber205]

Getting "robot check" every link I've tried for an amazon product -- anyone else seeing this?

Example URL: https://www.amazon.com/dp/B07SY4C5QF/ref=cm_sw_r_tw_apa_i_2qJLDbGGS3H0Q

[andrewdavidkay]

@bobber205 it's probably because your User-Agent header looks like it is automated and coming from a script ( it is ) but you should be able to set it to anything you want. I'm setting it to a browser like this:

try {
    const { body: html } = await got(url, {
      headers: {
        "User-Agent": req.headers["user-agent"]
      }
    });
    data = await metascraper({ url, html });
    statusCode = 200;
  } catch (err) {
    statusCode = 401;
    data = {
      message: `Scraping the open graph data from "${url}" failed.`,
      suggestion:
        "Make sure your URL is correct and the webpage has open graph data, meta tags or twitter card data."
    };