Help on "Message: chrome not reachable"

[ahirnish]

Hi @umihico,

I am using your selenium-docker solution to run headless Chrome browser in AWS Lambda for my project. It is a great help already :)

I need a bit of support and your expert advice on this setup if you dont mind. I'll give a bit of context what I am doing so that its easy for you to give advice. -

I created this headless browser lambda and I am using it to render some images from a URL and then save these images as screenshots in S3. This whole process runs for a big number of URLs (like a stream of URLs to this lambda as input) via AWS Step Functions.
Although its working but there are always about 50-60 URLs (total URLs 3000) which runs into these following messages -

• Message: chrome not reachable
• Message: unknown error: unhandled inspector error: {"code":-32000,"message":"Unable to capture screenshot"}
• Message: unknown error: cannot activate web view

I am trying to understand the problems as to why it happens only for few URLs and not for all. Your help will help me understand more.

Hope to hear back from you. Thank you.

This is my Dockerfile -

#based on https://github.com/umihico/docker-selenium-lambda
FROM public.ecr.aws/lambda/python:3.8 as build

# download chrome binaries, unzip them desired location
RUN yum install -y unzip && \
    curl -Lo "/tmp/chromedriver.zip" "https://chromedriver.storage.googleapis.com/98.0.4758.48/chromedriver_linux64.zip" && \
    curl -Lo "/tmp/chrome-linux.zip" "https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Linux_x64%2F950363%2Fchrome-linux.zip?alt=media" && \
    unzip /tmp/chromedriver.zip -d /opt/ && \
    unzip /tmp/chrome-linux.zip -d /opt/

# as of feb 2023 python3.8 was stable
FROM public.ecr.aws/lambda/python:3.8

# install necessary packages for running chrome
RUN yum install atk cups-libs gtk3 libXcomposite alsa-lib \
    libXcursor libXdamage libXext libXi libXrandr libXScrnSaver \
    libXtst pango at-spi2-atk libXt xorg-x11-server-Xvfb \
    xorg-x11-xauth dbus-glib dbus-glib-devel -y

# install selenium
RUN pip install selenium

COPY --from=build /opt/chrome-linux /opt/chrome
COPY --from=build /opt/chromedriver /opt/

#COPY lambda_function.py requirements.txt ./
# copy requirements
COPY . requirements.txt ./

COPY mraid.js ./tmp/

# install requirements
RUN python3.8 -m pip install -r requirements.txt -t .

# Command can be overwritten by providing a different command in the template directly.
# entry point
CMD ["lambda_function.lambda_handler"]

This is more like help and not an issue. Please suggest a better way to ask this help if this is no the right place. THank you.

[x-N0]

Why not retry these that are failing?,
Are always the same bunch that are failing?

@ahirnish

[ahirnish]

input is different each time - its not fixed. No of affected inputs are in the range 50-100 everytime. Message #2 can be more application specific but #1 and #3 are more tool specific. @x-N0

[x-N0]

What I'd do is to flip the versions to see if that gets the issue resolved.

[Turkmen1Mehmet]

mehmetturkmen@192 docker-selenium-lambda-memo % docker run -p 8080:8080 --platform=linux/amd64 my-selenium-lambda
26 Jun 2023 07:26:04,904 [INFO] (rapid) exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)
26 Jun 2023 07:26:13,082 [INFO] (rapid) extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory
26 Jun 2023 07:26:13,083 [INFO] (rapid) Configuring and starting Operator Domain
26 Jun 2023 07:26:13,084 [INFO] (rapid) Starting runtime domain
26 Jun 2023 07:26:13,088 [WARNING] (rapid) Cannot list external agents error=open /opt/extensions: no such file or directory
START RequestId: 0c4ae207-b6d5-425e-8a02-ca28fb990da5 Version: $LATEST
26 Jun 2023 07:26:13,094 [INFO] (rapid) Starting runtime without AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN , Expected?: false
Chromedriver sürümü: None
Chromedriver yolu: /opt/chromedriver

[ERROR] WebDriverException: Message: unknown error: Chrome failed to start: crashed.

(chrome not reachable)

(The process started from chrome location /opt/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Stacktrace:
#0 0x0040006d74e3
#1 0x004000406c76
#2 0x00400042fd78
#3 0x00400042c029
#4 0x00400046accc
#5 0x00400046a47f
#6 0x004000461de3
#7 0x0040004372dd
#8 0x00400043834e
#9 0x0040006973e4
#10 0x00400069b3d7
#11 0x0040006a5b20
#12 0x00400069c023
#13 0x00400066a1aa
#14 0x0040006c06b8
#15 0x0040006c0847
#16 0x0040006d0243
#17 0x004002aaf44b start_thread
  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/chrome/webdriv  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/chromium/webdr  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/webdriv  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/webdriv  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/webdriv  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/errorha    raise exception_class(message, screen, stacktrace)
END RequestId: 1d423694-c9a9-41c7-a660-4badff838b6a
REPORT RequestId: 1d423694-c9a9-41c7-a660-4badff838b6a Init Duration: 2.28 ms Duration: 16456.85 ms Billed Duration: 16457 ms Memory Size: 3008 MB Max Memory Used: 3008 MB

I'm getting this error right now, do you have any suggestions for a solution?
@ahirnish @umihico

[umihico]

@ahirnish @Turkmen1Mehmet

The shown code above and my code is too different.

If my original code works fine locally, you can modify step by step to convert my file to your file. Then, you can tell us which added line start causing bugs, instead of just pasting entire file.

[Turkmen1Mehmet]

@ahirnish @umihico
First of all, I made a small change in the code. First I run the code piece by piece. WebDriver.init()

mehmetturkmen@192 docker-selenium-lambda-memo % docker run -p 8080:8080 --platform=linux/amd64 my-selenium-lambda
26 Jun 2023 10:47:27,361 [INFO] (rapid) exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)
26 Jun 2023 10:47:38,253 [INFO] (rapid) extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory
26 Jun 2023 10:47:38,255 [INFO] (rapid) Configuring and starting Operator Domain
26 Jun 2023 10:47:38,257 [INFO] (rapid) Starting runtime domain
START RequestId: c3d5437a-a8af-484c-9473-16eda043a525 Version: $LATEST
26 Jun 2023 10:47:38,261 [WARNING] (rapid) Cannot list external agents error=open /opt/extensions: no such file or directory
26 Jun 2023 10:47:38,263 [INFO] (rapid) Starting runtime without AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN , Expected?: false
[ERROR] TypeError: WebDriver.init() got multiple values for argument 'option    chrome = webdriver.Chrome("/opt/chromedriver",options=options)
END RequestId: 495b85d4-3693-4dd9-826c-e4207ceb8b64
REPORT RequestId: 495b85d4-3693-4dd9-826c-e4207ceb8b64 Init Duration: 1.77 ms Duration: 1566.42 ms Billed Duration: 1567 ms Memory Size: 3008 MB Max Memory Used: 3008 MB
^C26 Jun 2023 10:56:08,387 [INFO] (rapid) Received signal signal=interrupt
26 Jun 2023 10:56:08,387 [INFO] (rapid) Shutting down...
26 Jun 2023 10:56:08,390 [WARNING] (rapid) Reset initiated: SandboxTerminated
26 Jun 2023 10:56:08,391 [INFO] (rapid) Sending SIGKILL to runtime-1(17).
26 Jun 2023 10:56:08,413 [INFO] (rapid) Stopping runtime domain
26 Jun 2023 10:56:08,415 [INFO] (rapid) Waiting for runtime domain processes termination
26 Jun 2023 10:56:08,416 [INFO] (rapid) Stopping operator domain
26 Jun 2023 10:56:08,416 [INFO] (rapid) Starting runtime domain

I also updated the code
This is the error I got this time

mehmetturkmen@192 docker-selenium-lambda-memo % docker run -p 8080:8080 --platform=linux/amd64 my-selenium-lambda
26 Jun 2023 11:07:18,953 [INFO] (rapid) exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)
26 Jun 2023 11:07:24,248 [INFO] (rapid) extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory
26 Jun 2023 11:07:24,251 [INFO] (rapid) Configuring and starting Operator Domain
26 Jun 2023 11:07:24,251 [INFO] (rapid) Starting runtime domain
26 Jun 2023 11:07:24,258 [WARNING] (rapid) Cannot list external agents error=open /opt/extensions: no such file or directory
START RequestId: 6637e50c-74dc-4eb4-9484-986025c641a9 Version: $LATEST
26 Jun 2023 11:07:24,266 [INFO] (rapid) Starting runtime without AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN , Expected?: false
[ERROR] WebDriverException: Message: unknown error: Chrome failed to start: crashed.
(chrome not reachable)
(The process started from chrome location /opt/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Stacktrace:
#0 0x0040006d74e3
#1 0x004000406c76
#2 0x00400042fd78
#3 0x00400042c029
#4 0x00400046accc
#5 0x00400046a47f
#6 0x004000461de3
#7 0x0040004372dd
#8 0x00400043834e
#9 0x0040006973e4
#10 0x00400069b3d7
#11 0x0040006a5b20
#12 0x00400069c023
#13 0x00400066a1aa
#14 0x0040006c06b8
#15 0x0040006c0847
#16 0x0040006d0243
#17 0x004002aaf44b start_thread
  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/chrome/webdriv  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/chromium/webdr  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/webdriv  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/webdriv  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/webdriv  File "/var/lang/lib/python3.10/site-packages/selenium/webdriver/remote/errorha    raise exception_class(message, screen, stacktrace)
END RequestId: 7ab9eb86-0a3f-42a9-8bfc-c4cd9fc5acc7
REPORT RequestId: 7ab9eb86-0a3f-42a9-8bfc-c4cd9fc5acc7 Init Duration: 4.71 ms Duration: 17264.78 ms Billed Duration: 17265 ms Memory Size: 3008 MB Max Memory Used: 3008 MB

Current main.py code

from selenium import webdriver
from tempfile import mkdtemp
from selenium.webdriver.common.by import By
import os
import sys
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.color import Color
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.chrome.options import Options
import re
import requests
import time
from bs4 import BeautifulSoup
import sqlite3
import urllib.request
import zipfile

def handler(event=None, context=None):
    chromedriver_path = "/opt/chromedriver"
    agent = {"User-Agent": 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}

    op = webdriver.ChromeOptions()
    op.binary_location = '/opt/chrome/chrome'
    op.add_argument('--headless')
    op.add_argument('--no-sandbox')
    op.add_argument("--disable-gpu")
    op.add_argument("--window-size=1280x1696")
    op.add_argument("--single-process")
    op.add_argument("--disable-dev-shm-usage")
    op.add_argument("--disable-dev-tools")
    op.add_argument("--no-zygote")
    op.add_argument(f"--user-data-dir={mkdtemp()}")
    op.add_argument(f"--data-path={mkdtemp()}")
    op.add_argument(f"--disk-cache-dir={mkdtemp()}")
    op.add_argument("--remote-debugging-port=9222")
    op.add_argument(f"--executable-path={chromedriver_path}")

    browser = webdriver.Chrome(options=op)
    browser.get("https://www.zara.com/tr/")
    return browser.find_element(by=By.XPATH, value="//html").text

[umihico]

@Turkmen1Mehmet
In comparison my original code, the diff is more than 10 lines. Please try between them, one line by line. You'll notice inserting specific line cause the error, then please share the detail in another issue.