Unable to Handle Cookie Banners Effectively

[MongiSaad]

We are encountering difficulties managing cookie banners and would like guidance on how to resolve this issue in the code. Specifically, we attempted to modify the cookie-banner-remover.ts file, but this led to multiple errors.

Our idea was to adjust the code in this file to dynamically search for all buttons on a website and programmatically click any button containing the word "accept" or its synonyms.

Do you think this approach is feasible? If not, could you suggest alternative solutions or point us in the right direction?

We are using this Git repository for a university project, and your help would be greatly appreciated.

[eric-zeng]

Hi, that sounds like a potentially feasible approach to me. Are the errors you encountered crashing the crawler, or is the new code not reliably finding the accept buttons? If it's a crawler-specific problem, I can try to help if you send more details.

My implementation is unfortunately just a quick-and-dirty approach that deletes cookie banner elements if there's a known signature for it, like how some ad-blockers do it. If you need something more robust, I might also suggest looking into the Consent-O-Matic browser extension. This is from another research project, and it can be configured to automatically accept or reject cookies. You could either modify the Puppeteer launch options so that adscraper launches with this extension active, or you could read their code and see how they approach interaction with cookie banners.