ChangeDetection.io missing selenium\webdriver functionality

[hellofaduck]

Describe the issue
When i trying to switch Fetch Method to WebDriver Chrome/Javascript it returns me an error like this:

Content fetcher 'html_webdriver' did not respond.
Be sure that the selenium/webdriver runner is running and accessible via network from this container/host.
Did you follow the instructions in the wiki?

WebDriver Host: http://browser-chrome:4444/wd/hub
[Go here for more information](https://github.com/dgtlmoon/changedetection.io/wiki/Fetching-pages-with-WebDriver)
Content fetcher did not respond properly, unable to use it. HTTPConnectionPool(host='browser-chrome', port=4444): Max retries exceeded with url: /wd/hub/session (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known'))

Screenshots

[tteck]

I feel, that's up to the user to install if needed.
https://www.selenium.dev/documentation/webdriver/getting_started/

[hellofaduck]

Hmm, okey i try to install selenium to same lxc ...

[Koliera]

Thanks for the update. Missed it in December.
While it's great, I do think that the installer is missing the chrome/firefox install and what's needed for the visual selection option.
DBTech has recently covered the chrome part, but still cannot do the visual selection part...
Any chance of a V2 of this tool?

[smtdev]

The package now comes with Playwright Chrome fetcher support (not installed by default).

[hellofaduck]

What does this mean exactly to us? We still need to install playwright it self right? I cant solve this task, and cant figure out how to install playwright to same lxc as chandetetection, seem playwright works only in docker. I tried it in docker and it works very bad in my case, change detection often cant check the page and return an error, but sometimes it's mostly ok, cant see any pattern of problem here.. :(

[OddSquirrel]

That's the problem. Yeah, of course, we could just install it ourselves. That's what the developer recommends as well, since he sells working instances, and making your own setup harder drives revenue.

It's really too bad that it's missing in this script, since their point is to allow even dumbos like me to get something up and running. But with changedetection.io, it's almost worthless without playwright chrome and should best be included. Just saying.

[tteck]

Has anyone checked out
https://www.selenium.dev/blog/2022/introducing-selenium-manager/
wget -q https://github.com/SeleniumHQ/selenium/raw/trunk/common/manager/linux/selenium-manager

or
https://pypi.org/project/webdriver-manager/

[hellofaduck]

Yes, but with selenium we cant use all "visual" features of changedetection, we cant click on site, cant visualy select region. It's all possible with playwright only when i checked it last time with docker :(
I really appreciate your hard work @tteck and want to say thank you for bringing this really useful project to us, but as @OddSquirrel said previously, raw changedetection.io service without playwright is almost fenced only with very simple use cases...i can say that i am not novice in linux and etc, i am working as sysadmin many years but maybe i am stupid or have some mindblock on this problem, but i really dont understand how to use playwright without docker if someone here more clever than me please share your widsdom with me :)

[Koliera]

Yup, I think the main thing missing from this is having a graphical capability of selecting which specific part of the page you'd wish to track. You need that playwright for that but I've absolutely haven't managed to get it working.

[zerofruchtshake]

Maybe this can help someone? I'm too stupid to get it to work..
dgtlmoon/changedetection.io#547

[Koliera]

nope, still not sure. Any chance for an update to the script to include all the functionality? Would be an awesome finalisation of the script @tteck

[ka2er]

workable solution :

## ad chromium driver and deps
apt install chromium-driver
## as LXC is running as root we need to tweak some security settings
echo 'export CHROMIUM_FLAGS="$CHROMIUM_FLAGS --no-sandbox --disable-dev-shm-usage --headless --remote-debugging-port=9222"' >> /etc/chromium.d/default-flags
## if you know how to inject ENV var to changedetection.io process please ignore host file mod (WEBDRIVER_URL environment variable - http://browser-chrome:4444/wd/hub by default)
echo '127.0.0.1 browser-chrome' >> /etc/hosts
## launch chromedriver
chromedriver --port=4444  --allowed-ips=<YOUR_LXC_IP> --allowed-origins=* --url-base=wd/hub
## install crontab @reboot to start previous line daemon
crontab -e 
....
@reboot chromedriver --port=4444  --allowed-ips=<YOUR_LXC_IP> --allowed-origins=* --url-base=wd/hub
...

This is a quick and dirty workaround.
There should be more secure and elegant way to add it but I'm neither webdriver specialist nor chrome headless so please be indulgent.

[lordratner]

Just piling on, there aren't many reasons you would not want the playwright web detection installed, at worst it could be an option during LXC creation? But I think it should just be included, since it is such a major part of changedetection.

Thanks!

[lordratner]

Ok looks like someone already did the leg work here:

https://www.reddit.com/r/selfhosted/comments/w2corm/using_changedetectionio_installed_via_pip_not/

I'll see if I can adapt this to the install script.

[lordratner]

For anyone interested, I've got a failed Pull Request that fixes the issue: #1393

It'll get Change Detection working with all the features enabled. Just make sure you go to settings --> Fetching and enable Playwright.

The Visual Filter Selector is the most interesting tool in Change Detection, so you'll probably find it more useful with this addition.

[tteck]

Is this working? Does it need to be merged into main, or just drop dev?

[lordratner]

I'll test it and get back to you. I'm pretty unfamiliar with GitHub so it takes me a while to maneuver around, but I'll figure it out.

[lordratner]

Getting this error:

[ERROR] in line 17: exit code 100: while executing command $STD apt-get install -y curl sudo mc git build-essential dumb-init gconf-service ffmpeg \ 
++ [[ 17 -eq 22 ]]
+ exit

Here's the whole block:

 - Installing Dependencies (Patience)...+ apt-get install -y curl sudo mc git build-essential dumb-init gconf-service ffmpeg ' '
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package  
++ error_handler 17 '$STD apt-get install -y curl sudo mc git build-essential dumb-init gconf-service ffmpeg \ '
++ local exit_code=100
++ local line_number=17
++ local 'command=$STD apt-get install -y curl sudo mc git build-essential dumb-init gconf-service ffmpeg \ '
++ local 'error_message=\033[01;31m[ERROR]\033[m in line \033[01;31m17\033[m: exit code \033[01;31m100\033[m: while executing command \033[33m$STD apt-get install -y curl sudo mc git build-essential dumb-init gconf-service ffmpeg \ \033[m'
++ echo -e '\n\033[01;31m[ERROR]\033[m in line \033[01;31m17\033[m: exit code \033[01;31m100\033[m: while executing command \033[33m$STD apt-get install -y curl sudo mc git build-essential dumb-init gconf-service ffmpeg \ \033[m'

[ERROR] in line 17: exit code 100: while executing command $STD apt-get install -y curl sudo mc git build-essential dumb-init gconf-service ffmpeg \ 
++ [[ 17 -eq 22 ]]
+ exit

Now, when I open the container console and run apt-get install -y curl sudo mc git build-essential dumb-init gconf-service ffmpeg It works fine.

[lordratner]

I've also run

apt-get install -y \
  curl \
  sudo \
  mc \
  git \
  build-essential \
  dumb-init \
  gconf-service \
  ffmpeg \ 
  libappindicator1 \  
  libappindicator3-1 \  
  libasound2 \ 
  libatk-bridge2.0-0 \
  libatk1.0-0 \
  libcairo2 \
  libcups2 \
  libdbus-1-3 \
  libexpat1 \
  libgbm-dev \
  libgbm1 \
  libgconf-2-4 \
  libgdk-pixbuf2.0-0 \
  libglib2.0-0 \
  libgtk-3-0 \
  libnspr4 \
  libnss3 \
  libpango-1.0-0 \
  libpangocairo-1.0-0 \
  qpdf \
  xdg-utils \
  xvfb

from the dev script and they all install fine in the LXC console

[lordratner]

I think the comments are killing the script

[lordratner]

I don't see the dev branch anymore. Did you try removing the comments so I can't test the install again?

[Koliera]

Any updates on this? Is it possible to maybe get a docker-compose so that it could be run on a separate docker lcx?

[lordratner]

It disappeared from dev, so I think he just didn't want to deal with it anymore.

Most of the scripts are designed to run without Docker. Docker adds an extra layer in an LXC that can cause problems depending on your setup. It works, but it's not as good as a "bare metal" install in an LXC.

[tteck]

69bdd64

[Koliera]

@lordratner - any updates?

[gitandriu]

What's the proposed best solution currently? @ka2er workaround? Any progress with the updates to the @tteck script?

Any guidance would be appreciated. Thanks

[tteck]

msg_info "Installing Browserless & Playwright"
mkdir /opt/browserless
$STD python3 -m pip install playwright
$STD git clone https://github.com/browserless/chrome /opt/browserless
$STD npm install --prefix /opt/browserless
$STD npm run build --prefix /opt/browserless
$STD npm prune production --prefix /opt/browserless
msg_ok "Installed Browserless & Playwright"

was added, haven't had any comments.

[gitandriu]

Awesome, works great, thanks much!

I had just updated the container but had to remove and recreate and was installed then.

[FormallyKnownAs]

I just did a fresh install and keep getting the error below when trying to use the Chromium driver. Any have an idea how to fix?

No module named 'playwright._impl._api_types'