Concourse CI Windows Worker - concourse

I'm trying to setup a Concourse CI environment with a Windows 7 worker.
I have one machine under Ubuntu Server (16.04) hosting my TSA server and one worker (for the support of git resources), and a second one under Windows 7 hosting a worker.
Everything seems to work fine as:
I can login into the web ui
the fly -t my_concourseci workers command returns :
name containers platform tags team state version
ubuntu 1 linux none none running 1.1
windows7 0 windows none none running 1.1
the fly -t my_concourseci execute -c test.yml command returns:
executing build 146
initializing
running echo Hello World!
Hello World!
with the following content in test.yml file:
platform: windows
run:
path: echo
args: [ "Hello World!" ]
Nevertheless when I add an input to my task:
platform: windows
> inputs:
> - name: concourse
run:
path: echo
args: [ "Hello World!" ]
I get the following error:
executing build 148
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5698k 0 5698k 0 0 1948k 0 --:--:-- 0:00:02 --:--:-- 1949k
initializing
failed to stream in to volume
errored
If I look at the windows worker log I got this error that comes up:
{"timestamp":"1500642862.643555164",
"source":"baggageclaim",
"message":"baggageclaim.api.volume-server.stream-in.bad-stream-payload",
"log_level":1,
"data":{"error":"tar extract failed (exit status 2). output: \"\\ngzip: stdin: not in gzip format\\n/usr/bin/tar: Child returned status 1\\n/usr/bin/tar: Error is not recoverable: exiting now\\n\"",
"session":"2.1.8730",
"volume":"15bf1fc6-0727-4962-6c84-18446e54ab96"}
}
Any ideas about what can cause a not in gzip format error ? Knowing that if I run the exact same task on a linux platform every works fine.
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
inputs:
- name: concourse
run:
path: echo
args: [ "Hello World!" ]
----- STDOUT
executing build 149
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5699k 0 5699k 0 0 1917k 0 --:--:-- 0:00:02 --:--:-- 1918k
initializing
Pulling busybox#sha256:2605a2c4875ce5eb27a9f7403263190cd1af31e48a2044d400320548356251c4...
sha256:2605a2c4875ce5eb27a9f7403263190cd1af31e48a2044d400320548356251c4: Pulling from library/busybox
9e87eff13613: Pulling fs layer
9e87eff13613: Verifying Checksum
9e87eff13613: Download complete
9e87eff13613: Pull complete
Digest: sha256:2605a2c4875ce5eb27a9f7403263190cd1af31e48a2044d400320548356251c4
Status: Downloaded newer image for busybox#sha256:2605a2c4875ce5eb27a9f7403263190cd1af31e48a2044d400320548356251c4
Successfully pulled busybox#sha256:2605a2c4875ce5eb27a9f7403263190cd1af31e48a2044d400320548356251c4.
running echo Hello World!
succeeded
Thanks.

Launch powershell as Administrator, and run the concourse_worker.exe from there. That worked for me.
I am running a Windows Server 2016 Base on AWS based on this AMI ami-e1876a98

Thanks for your answers, it helps me solve my problem.
I were starting my concourse worker within a MSYS 1.0 environment. The thing is that MSYS contains tar and gunzip binaries within the $PATH. When I started concourse worker inside a Power Shell or cmd.exe without any MSYS unix-like binaries in the $PATH it worked like a charm !
Note: Be sure to have no MSYS binaries in the $PATH Windows environment variable for this to work, especially check that Git-Bash environment tools are not added to your windows $PATH environment variable.
Thanks again.

Related

VUE 3 - weird failure in VS Code and flawless running in console

Windows 10 x64 21H2, running my project in VS Code and get the following error:
Executing task: npm run serve
> weatherApp#0.1.0 serve
> vue-cli-service serve
Browserslist: caniuse-lite is outdated. Please run:
npx browserslist#latest --update-db
Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating
INFO Starting development server...
10% building 2/5 modules 3 active ...babel-loader\lib\index.js!C:\tmp\VUE Weather apps\VUE 3 - weather\node_modules\eslint-loader\index.js??ref--13-0!C:\tmp\VUE Weather apps\VUE 3 - weather\src\main.jsError: error:0308010C:digital envelope routines::unsupported
at new Hash (node:internal/crypto/hash:71:19)
at Object.createHash (node:crypto:133:10)
at module.exports (C:\tmp\VUE Weather apps\VUE 3 -
...
{
opensslErrorStack: [ 'error:03000086:digital envelope routines::initialization error' ],
library: 'digital envelope routines',
reason: 'unsupported',
code: 'ERR_OSSL_EVP_UNSUPPORTED'
}
Node.js v18.12.1
* The terminal process "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -Command npm run serve" terminated with exit code: 1.
* Terminal will be reused by tasks, press any key to close it.
At the same time the command npm run serve runs w/o any issues. What should I do to have the issue fixed?

Issues with multiple GitHub self-runners on the same server

Are there any reasons why this is not a good idea? I ask because I constantly experience very, very inconsistent results. For example, while setting up my GH Actions over the last few days, I must have run at least 200 workflows. However, for the first time ever, I am now seeing this error:
Run ruby/setup-ruby#v1
with:
ruby-version: 3.0.2
bundler-cache: true
bundler: default
working-directory: .
cache-version: 0
env:
BUNDLE_GEMS__CONTRIBSYS__COM: ***
ImageOS: ubuntu20
Modifying PATH
Entries added to PATH to use selected Ruby:
/opt/hostedtoolcache/Ruby/3.0.2/x64/bin
Downloading Ruby
https://github.com/ruby/ruby-builder/releases/download/toolcache/ruby-3.0.2-ubuntu-20.04.tar.gz
Took 0.71 seconds
Extracting Ruby
/usr/bin/tar -xz -C /opt/hostedtoolcache/Ruby/3.0.2 -f /home/ubuntu/actions-runner-2/_work/_temp/7d0937cf-69b1-4c73-b1bd-7386fca820a2
/usr/bin/tar: x64/lib: Cannot utime: No such file or directory
/usr/bin/tar: Exiting with failure status due to previous errors
Took 0.52 seconds
Error: The process '/usr/bin/tar' failed with exit code 2
I have absolutely no clue whatsoever why this would be presenting itself. If I re-run the same workflow, the error goes away. I'm not sure if this is because one runner is conflicting with another while trying to access the /opt/hostedtoolcache/ directory or something else.
Here's the exact same job re-run without any issues:

"image is too large" keeps on happening to openbmc image for Raspberrypi platform

Could someone please give me advice to make an openbmc image for Raspberrypi platform ?
Before I tried, I looked through related documents and believed an openbmc image can be worked on Raspberrypi.
Like OpenBMC with Raspberry Pi (2 or 3) and build bmcweb?
and https://kevinleeblog.github.io/project1/2019/11/25/openbmc-for-raspberry-pi-zero/.
So, I followed these instructions and tried the following steps.
#1: Git clone openbmc.git to my local PC.
tm#tm-VB1:~/Rpi4-64$ git clone https://github.com/openbmc/openbmc.git
Snip the logs but it looks no problem.
Receiving objects: 100% (182121/182121), 84.10 MiB | 5.55 MiB/s, done.
Resolving deltas: 100% (96860/96860), done.
#2: set TEMPLATECONF for raspberrypi
tm#tm-VB1:~/Rpi4-64$ export TEMPLATECONF=meta-evb/meta-evb-raspberrypi/conf
tm#tm-VB1:~/Rpi4-64$ echo $TEMPLATECONF
meta-evb/meta-evb-raspberrypi/conf
#3: set up the environment by "openbmc-env"
tm#tm-VB1:~/Rpi4-64/openbmc$ . openbmc-env
### Initializing OE build env ###
Snip the logs but it looks no problem. As you know, the script automatically creates a subdirectory,build, under openbmc.
Common targets are:
obmc-phosphor-image
tm#tm-VB1:~/Rpi4-64/openbmc/build$
#4: Change the directory and edit local.conf for my Raspberrypi platform.
tm#tm-VB1:~/Rpi4-64/openbmc/build$ cat ./conf/local.conf
Snip the log for unchanged part.
MACHINE ??= "raspberrypi4-64" <<< Change here for my platform.
DL_DIR ?= "/home/tm/Yocto/downloads" <<< Add here for build-time reduction at retry.
SSTATE_DIR ?= "/home/tm/Yocto/sstate-cache" <<< Add here for build-time reduction at retry.
#5: Change FLASH_SIZE variable based on the following sugestion. https://github.com/openbmc/openbmc/issues/3590
tm#tm-VB1:~/Rpi4-64/openbmc/meta-phosphor/classes$ cat image_types_phosphor.bbclass
Snip the log.
# Flash characteristics in KB unless otherwise noted
FLASH_SIZE ?= "131072" <<< I changed only this variable from 32768 to 131072.
#6: bitbake starts.
tm#tm-VB1:~/Rpi4-64/openbmc/bitbake obmc-phosphor-image
Then, ERROR happened.
ERROR: Logfile of failure stored in: /home/tm/Rpi/openbmc/build/tmp/work/raspberrypi-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2055074
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.09147 s, 367 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=495980 name=/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/u-boot.bin
DEBUG: Spanning start=0K end=512K
DEBUG: Compare needed=495980 available=524288 margin=28308
484+1 records in
484+1 records out
495980 bytes (496 kB, 484 KiB) copied, 0.00120141 s, 413 MB/s
DEBUG: Considering file size=8266960 name=/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/fitImage-obmc-phosphor-initramfs-raspberrypi-raspberrypi
DEBUG: Spanning start=512K end=4864K
>>>DEBUG: Compare needed=8266960 available=4456448 margin=-3810512
ERROR: Image '/home/tm/Rpi/openbmc/build/tmp/deploy/images/raspberrypi/fitImage-obmc-phosphor-initramfs-raspberrypi-raspberrypi' is too large!
DEBUG: Python function do_generate_static finished
It said margin=-3810512.
Now, my 2nd try.
I removed the whole openbmc directory and did the same steps above.
But this time, I change FLASH_SIZE from 32768 to 262144.
It is the same result like below.
ERROR: obmc-phosphor-image-1.0-r0 do_generate_static: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
ERROR: Logfile of failure stored in: /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2061792
ERROR: Task (/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3915 tasks of which 2633 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static
Summary: There were 2 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
tm#tm-VB1:~/Rpi4/openbmc/build$ cat /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2061792
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.177223 s, 189 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=548224 name=/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin
DEBUG: Spanning start=0K end=512K
>>>DEBUG: Compare needed=548224 available=524288 margin=-23936
ERROR: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
DEBUG: Python function do_generate_static finished
tm#tm-VB1:~/Rpi4/openbmc/build$
It said margin=-23936.
OK. Image is too large. So,my 3rd try.
I removed the whole openbmc directory and did the same steps above.
But this time, I change FLASH_SIZE from 32768 to 9437184.
It is the same result like below.
ERROR: obmc-phosphor-image-1.0-r0 do_generate_static: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
ERROR: Logfile of failure stored in: /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2058361
ERROR: Task (/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3935 tasks of which 0 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/openbmc/meta-phosphor/recipes-phosphor/images/obmc-phosphor-image.bb:do_generate_static
Summary: There were 4 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
tm#tm-VB1:~/Rpi4/openbmc$
tm#tm-VB1:~/Rpi4/openbmc$ cat /home/tm/Rpi4/openbmc/build/tmp/work/raspberrypi4-openbmc-linux-gnueabi/obmc-phosphor-image/1.0-r0/temp/log.do_generate_static.2058361
DEBUG: Executing python function do_generate_static
DEBUG: Executing shell function do_mk_static_nor_image
32768+0 records in
32768+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.173685 s, 193 MB/s
DEBUG: Shell function do_mk_static_nor_image finished
DEBUG: Considering file size=548224 name=/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin
DEBUG: Spanning start=0K end=512K
>>>DEBUG: Compare needed=548224 available=524288 margin=-23936
ERROR: Image '/home/tm/Rpi4/openbmc/build/tmp/deploy/images/raspberrypi4/u-boot.bin' is too large!
DEBUG: Python function do_generate_static finished
tm#tm-VB1:~/Rpi4/openbmc$
It said the same margin as 256MB case.
My 4th try.
I removed the whole openbmc directory and did the same steps above.
I changed MACHINE ??= "raspberrypi4-64" to "raspberrypi2"
But this time, I change FLASH_SIZE from 32768 to 33554432.
It is the same result before.
My 5th try.
I removed the whole openbmc directory and did the same steps above.
I used MACHINE ??= "raspberrypi2"
But this time, I change FLASH_SIZE from 32768 to 67108864.
It is the same result before.
After I tried several variations, it always said "image is too large" although I changed FLASH_SIZE to much much larger one.
So, I am wondering if I have missed some important configuration or it needs another parameter to fix this except FLASH_SIZE.
By the way, I tried romulus and made it.
My environment is ubuntu-20.04.2.0-desktop-amd64.
I really appreciate someone could kindly give me advice to make this work.
Interesting, I don't have a quick fix for you but I did notice the partition that is over sized is the uboot partition. The uboot is a smaller separate binary installed on the machine. It looks as if your uboot build is over 512k and the partition is set for 512k. Your flash size is massize
FLASH_SIZE = 9437184" that is more then a gig, (because FLASH_SIZE is in K)
If I were you I would first try to build an older version of openbmc for raspberry pi. (It used to work so you just need to find the commit before uboot grew to big). Use git to move back a month until you find it works.
If that does not work I would try to modify the partition table.
here is where you failing
this looks fine building the uboot image looks fine
increasing the kernel offset make if build, but the other targets in openbmc will not be happy with this solution. So maybe meta-raspberry-pi will have to override the partition table (if uboot can not be shrunk)
What ever you do, open an issue on the github and share you changes. Also use the discord, and gerrit.
I just replicated this issue. We should fix it

Selenium webdriver: Unknown error: cannot determine loading status [duplicate]

I'm using InstaPy which use Python and Selenium. I start the script per Cron and from time to time it crashes. So it'r really irregular, sometimes it runs well through. I'v posted on GitHub Repo as well already but didn't get an answer there, so i'm asking here now if someone has an idea why.
It's a digital ocean ubuntu server and i'm using it on headless mode. The driver version are visible on the log. here are error messages:
ERROR [2018-12-10 09:53:54] [user] Error occurred while deleting cookies from web browser!
b'Message: invalid session id\n (Driver info: chromedriver=2.44.609551 (5d576e9a44fe4c5b6a07e568f1ebc753f1214634),platform=Linux 4.15.0-42-generic x86_64)\n'
Traceback (most recent call last):
File "/root/InstaPy/instapy/util.py", line 1410, in smart_run
yield
File "./my_config.py", line 43, in <module>
session.follow_user_followers(['xxxx','xxxx','xxxx','xxxx'], amount=100, randomize=True, interact=True)
File "/root/InstaPy/instapy/instapy.py", line 2907, in follow_user_followers
self.logfolder)
File "/root/InstaPy/instapy/unfollow_util.py", line 883, in get_given_user_followers
channel, jumps, logger, logfolder)
File "/root/InstaPy/instapy/unfollow_util.py", line 722, in get_users_through_dialog
person_list = dialog_username_extractor(buttons)
File "/root/InstaPy/instapy/unfollow_util.py", line 747, in dialog_username_extractor
person_list.append(person.find_element_by_xpath("../../../*")
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webelement.py", line 351, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webelement.py", line 659, in find_element
{"using": by, "value": value})['value']
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webelement.py", line 633, in _execute
return self._parent.execute(command, params)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash
from unknown error: cannot determine loading status
from tab crashed
(Session info: headless chrome=70.0.3538.110)
(Driver info: chromedriver=2.44.609551 (5d576e9a44fe4c5b6a07e568f1ebc753f1214634),platform=Linux 4.15.0-42-generic x86_64)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/InstaPy/instapy/instapy.py", line 3845, in end
self.browser.delete_all_cookies()
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 878, in delete_all_cookies
self.execute(Command.DELETE_ALL_COOKIES)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: chrome not reachable
(Session info: headless chrome=71.0.3578.80)
(Driver info: chromedriver=2.44.609551 (5d576e9a44fe4c5b6a07e568f1ebc753f1214634),platform=Linux 4.15.0-42-generic x86_64)
Any idea what the reason could be and how to solve it?
Thanks for the inputs. And the guys from http://treestones.ch/ helped me out.
Though you see the error as:
Error occurred while deleting cookies from web browser!
b'Message: invalid session id\n (Driver info: chromedriver=2.44.609551 (5d576e9a44fe4c5b6a07e568f1ebc753f1214634),platform=Linux 4.15.0-42-generic x86_64)\n'
The main exception is:
selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash
from unknown error: cannot determine loading status
from tab crashed
Your code trials would have given us some clues what going wrong.
Solution
There are diverse solution to this issue. However as per UnknownError: session deleted because of page crash from tab crashed this issue can be solved by either of the following solutions:
Add the following chrome_options:
chrome_options.add_argument('--no-sandbox')
Chrome seem to crash in Docker containers on certain pages due to too small /dev/shm. So you may have to fix the small /dev/shm size.
An example:
sudo mount -t tmpfs -o rw,nosuid,nodev,noexec,relatime,size=512M tmpfs /dev/shm
It also works if you use -v /dev/shm:/dev/shm option to share host /dev/shm
Another way to make it work would be to add the chrome_options as --disable-dev-shm-usage. This will force Chrome to use the /tmp directory instead. This may slow down the execution though since disk will be used instead of memory.
chrome_options.add_argument('--disable-dev-shm-usage')
from tab crashed
from tab crashed was WIP(Work In Progress) with the Chromium Team for quite some time now which relates to Linux attempting to always use /dev/shm for non-executable memory. Here are the references :
Linux: Chrome/Chromium SIGBUS/Aw, Snap! on small /dev/shm
Chrome crashes/fails to load when /dev/shm is too small, and location can't be overridden
As per Comment61#Issue 736452 the fix seems to be have landed with Chrome v65.0.3299.6
Reference
You can find a couple of relevant discussions in:
org.openqa.selenium.SessionNotCreatedException: session not created exception from tab crashed error when executing from Jenkins CI server
In case someone is facing this problem with docker containers:
use the flag --shm-size=2g when creating the container and the error is gone.
This flag make the container to use the host's shared memory.
Example
$ docker run -d --net gridNet2020 --shm-size="2g" -e SE_OPTS="-browser applicationName=zChromeNodePdf30,browserName=chrome,maxInstances=1,version=78.0_debug_pdf" -e HUB_HOST=selenium-hub-3.141.59 -P -p 5700:5555 --name zChromeNodePdf30 -v /var/lib/docker/sharedFolder:/home/seluser/Downloads selenium/node-chrome:3.141.59-xenon
Source: https://github.com/SeleniumHQ/docker-selenium
I was getting the following error on my Ubuntu server:
selenium.common.exceptions.WebDriverException: Message: unknown error:
session deleted because of page crash from tab crashed (Session
info: headless chrome=86.0.4240.111) (Driver info:
chromedriver=2.41.578700
(2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux
5.4.0-1029-aws x86_64)
It turned out the the cause of the error was insufficient disk space on the server and the solution was to extend my disk space. You can check this question for more information.
We need to specify the shm memory separatly, --shm-size=2g
In case of docker,
use the following config - this working fine for me
services:
chrome:
image: selenium/node-chrome:4.0.0-rc-1-prerelease-20210823
shm_size: 2gb
Message: unknown error: session deleted because of page crash from unknown error: cannot determine loading status from tab crashed
(Session info: headless chrome=95.0.4638.69)
This error occurred because there was not enough waiting time for web pages to load
The answers above solved my issue, but since i needed to run it from a docker-compose.yml i used this configuration which calls my regular unchanged DockerFile
docker-compose.yml
version: '1.0'
services:
my_app:
build:
context: .
#when building
shm_size: 1gb
#when running
shm_size: 1gb
DockerFile (selenium on Ubuntu -WSL-)
FROM python:3.10
# install google chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
RUN apt-get -y update
RUN apt-get install -y google-chrome-stable
# install chromedriver
RUN apt-get install -yqq unzip
RUN wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`/chromedriver_linux64.zip
RUN unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/
# set display port to avoid crash
ENV DISPLAY=:99
# install selenium
RUN pip install selenium==3.8.0
#install and prepar app
COPY ./requirements.txt ./
# COPY . /app
RUN pip3 install -r requirements.txt
RUN apt-get install -y libnss3
ENV APP_DIR=/app/my_app
RUN mkdir -p ${APP_DIR}
WORKDIR ${APP_DIR}
# COPY . ${APP_DIR} #not needed since we are mapping the volume in docker-compose
CMD [ "my_app.py" ]
ENTRYPOINT [ "python" ]
This happened to me while trying to open a new web page with the same driver in Chromium. It worked fine in my local machine where I use Chrome.
Did not worked:
driver = webdriver.Chrome(options=options)
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {
"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
driver.get('url1')
# Do operations with url1
driver.get('url2')
# Do operations with url2 -> did not work and crashed
Below is the solution, I am using which is working for me. i.e re-initializing the driver
def setup_driver():
global driver
driver = webdriver.Chrome(options=options)
driver.maximize_window()
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {
"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
setup_driver()
driver.get('url1')
# Do operations with url1
driver.close()
setup_driver()
driver.get('url2')
# Do operations with url2
driver.close()
I'm not sure whether this is the only possible cause and solution, but after thorough investigation of this error which I encountered every now and then, I found the following evidences:
In the log of the Selenium Grid nodes (which you can show by executing the following command on the docker host: sudo docker logs <container-id>) I found many errors reading: [SEVERE]: bind() failed: Cannot assign requested address (99). From what I read, this error usually means that there are no available ports.
When showing the processes running inside a node (sudo docker exec -it bash and then ps aux), I found more than 300 instances of chrome-driver processes (you can count them using ps aux|grep driver|wc -l)
When running locally, I know that the chrome-driver process is normally invoked when you create an instance of ChromeDriver and is terminated when you call driver.Quit() (I work in C#, not Python). Therefore I concluded that some tests don't call drive.Quit().
The conclusion
In my case, I found that even though we had a call to driver.Quit() in the [TearDown] method (we use NUnit), we had some more code before that line, that could throw and exception. When one of these preceding lines threw an exception, the line that calls driver.Quit() is not reached, and therefore over time we were "leaking" chrome-driver processes on the Selenium Grid nodes. These orphan processes caused a resource leak of available ports (and probably also memory), which also caused the browser's page to crash.
The solution
Given the above conclusion, the solution was pretty straight forward. We had to wrap the code that precedes driver.Quit() in a try/finally, and put the call to driver.Quit() in the finally clause, like this:
[TearDown]
public void MyTearDown()
{
try
{
// Perform any tear down code you like, like saving screenshots, page source, etc.
}
finally
{
_driver?.Quit();
}
}
I was having the same problem, I checked the log at which point in my script the bug happened and I added some wait, ie, time.sleep(2) just before the bug, and my problem was fixed.

How to test Unity projects with Travis CI?

I was following this specific guide: https://jonathan.porta.codes/2015/04/17/automatically-build-your-unity3d-project-in-the-cloud-using-travisci-for-free/ to test my unity project with travis CI but have been running into the same problem and don't understand how to change the 'installer' program to fix this problem:
$ ./Scripts/install.sh
Downloading from http://download.unity3d.com/download_unity/a6d8d714de6f/MacEditorInstaller/ Unity-5.4.0f3.pkg:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 16433 100 16433 0 0 16498 0 --:--:-- --:--:-- --:--:-- 16498
Installing Unity.pkg
installer: Error the package path specified was invalid: 'Unity.pkg'.
The command "./Scripts/install.sh" failed and exited with 1 during .
Your build has been stopped.
Here are the scripts I wrote using the tutorial:
.travis.yml
language: objective-c
osx_image: xcode61
rvm:
- 2.1.2
install:
- ./Scripts/install.sh
script:
- ./Scripts/build.sh
install.sh
#! /bin/sh
echo 'Downloading from http://download.unity3d.com/download_unity/a6d8d714de6f/MacEditorInstaller/Unity-5.4.0f3.pkg:
curl -o Unity.pkg http://download.unity3d.com/download_unity/a6d8d714de6f/MacEditorInstaller/Unity-5.4.0f3.pkg:
echo 'Installing Unity.pkg'
sudo installer -dumplog -package Unity.pkg -target /
Any help for this problem or guidance towards a solution would be greatly appreciated, thanks
Remove the colon at the end of the url inside install.sh
curl -o Unity.pkg http://download.unity3d.com/download_unity/a6d8d714de6f/MacEditorInstaller/Unity-5.4.0f3.pkg:
Unity-5.4.0f3.pkg: => Unity-5.4.0f3.pkg