I have a test pipeline in Azure Devops. In the pipeline, I want to uninstall pyspark module. I do it with pip uninstall pyspark. Below is the pipeline
trigger: none
jobs:
- job: 'QA_Pipeline'
timeoutInMinutes: 300
pool: vmss-deep-dev-pool-002
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '3.8'
- script: |
pip install --upgrade --force-reinstall pip setuptools wheel
pip uninstall pyarrow
pip uninstall pyspark
pip install -I azure-cli==2.18.0
pip install -I databricks-connect==9.1.30
displayName: 'Install modules'
...
...
When I execute this pipeline, the pyspark installation is found but the uninstallation never finishes.
Below is the stacktrace
Collecting pip
Using cached pip-23.0-py3-none-any.whl (2.1 MB)
Collecting setuptools
Using cached setuptools-67.0.0-py3-none-any.whl (1.1 MB)
Collecting wheel
Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
Installing collected packages: wheel, setuptools, pip
Attempting uninstall: wheel
Found existing installation: wheel 0.38.4
Uninstalling wheel-0.38.4:
Successfully uninstalled wheel-0.38.4
Attempting uninstall: setuptools
Found existing installation: setuptools 67.0.0
Uninstalling setuptools-67.0.0:
Successfully uninstalled setuptools-67.0.0
Attempting uninstall: pip
Found existing installation: pip 23.0
Uninstalling pip-23.0:
Successfully uninstalled pip-23.0
Successfully installed pip-23.0 setuptools-67.0.0 wheel-0.38.4
WARNING: Skipping pyarrow as it is not installed.
Found existing installation: pyspark 3.3.1
Uninstalling pyspark-3.3.1:
Would remove:
/opt/hostedtoolcache/Python/3.8.16/x64/bin/beeline
/opt/hostedtoolcache/Python/3.8.16/x64/bin/beeline.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/docker-image-tool.sh
/opt/hostedtoolcache/Python/3.8.16/x64/bin/find-spark-home
/opt/hostedtoolcache/Python/3.8.16/x64/bin/find-spark-home.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/find_spark_home.py
/opt/hostedtoolcache/Python/3.8.16/x64/bin/load-spark-env.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/load-spark-env.sh
/opt/hostedtoolcache/Python/3.8.16/x64/bin/pyspark
/opt/hostedtoolcache/Python/3.8.16/x64/bin/pyspark.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/pyspark2.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/run-example
/opt/hostedtoolcache/Python/3.8.16/x64/bin/run-example.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-class
/opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-class.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-class2.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-shell
/opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-shell.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-shell2.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-sql
/opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-sql.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-sql2.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-submit
/opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-submit.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-submit2.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/sparkR
/opt/hostedtoolcache/Python/3.8.16/x64/bin/sparkR.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/bin/sparkR2.cmd
/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pyspark-3.3.1.dist-info/*
/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pyspark/*
It has been around 4 hours and I do not see any progress. I tried stopping the job and restarting but it does not help. Any advice will be appreciated. Thanks
pip uninstall -y pyspark seems to do the trick. It works now
Related
I am currently coding my .gitlab-ci.yml to setup the CI/CD of my project.But when the pipeline runs I get an ERROR during the installation of my docker-compose.#
My .gitlab-ci.yml file
image: docker:stable
stages:
- deploy
services:
- docker:dind
variables:
DOCKER_CLIENT_TIMEOUT: 240
COMPOSE_HTTP_TIMEOUT: 240
before_script:
- apk add python3 py3-pip
- python3 -m pip install --upgrade pip setuptools wheel
- pip3 install dxlmispservice
- pip3 install setuptools-rust
- pip3 install docker-compose
The Log of the Pipeline
Building wheel for PyYAML (pyproject.toml): started
Building wheel for PyYAML (pyproject.toml): finished with status 'done'
Created wheel for PyYAML: filename=PyYAML-5.4.1-cp38-cp38-linux_x86_64.whl size=45656 sha256=9de5adf1f8c62bc5c5b9da385015c331f5c6100035a7de96c1ffab4ad2ec2eb2
Stored in directory: /root/.cache/pip/wheels/dd/c5/1d/5d7436173d3efd4a14dcb510eb0b29525ecb6b0e41489e716e
Building wheel for cffi (setup.py): started
Building wheel for cffi (setup.py): finished with status 'error'
error: subprocess-exited-with-error
I installed Scipy using the command sudo apt-get install python-numpy python-scipy and looks like the latest version of both are installed. But when I try to install a python package named wfdb using pip install wfdb, it get an error while Building wheel for scipy (PEP 517).
ERROR: Failed building wheel for scipy
Running setup.py clean for scipy
ERROR: Command errored out with exit status 1:
command: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-
fhfmfpqi/scipy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-
fhfmfpqi/scipy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)
(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code,
__file__,
'"'"'exec'"'"'))' clean --all
cwd: /tmp/pip-install-fhfmfpqi/scipy
Complete output (9 lines):
`setup.py clean` is not supported, use one of the following instead:
- `git clean -xdf` (cleans all files)
- `git clean -Xdf` (cleans all versioned files, doesn't touch
files that aren't checked into the git repo)
Add `--force` to your command to use it anyway if you must (unsupported).
----------------------------------------
ERROR: Failed cleaning build dir for scipy
Failed to build scipy
ERROR: Could not build wheels for scipy which use PEP 517 and cannot be installed directly`
I upgraded pip using pip install --upgrade pip.
python version used: 3.7
Can I get some help on this?
Clean out the install (I don't know which system you're using) i.e. get rid of the $PATH variables and just do a fresh install.
If you're using Anaconda:
conda install -c anaconda scipy
should do it
also the goops old
pip install scipy
gets it done as well. May I ask why you approached your installation in this way (again, I don't know which system you're on)
Never do sudo pip. Instead do pip install --user. Or (better) work in a virtualenv.
EDIT: typo: there is no space between -- and user.
I'm trying to run Cypress tests on the Hosted Linux Pool for Azure Pipelines. Unfortunately, the Hosted Agent doesn't have all the dependencies for Cypress installed.
Running the documented apt-get doesn't work:
2018-05-18T21:03:14.7423331Z ##[section]Starting: Install cypress dependencies
2018-05-18T21:03:14.7474742Z ==============================================================================
2018-05-18T21:03:14.7488281Z Task : Bash
2018-05-18T21:03:14.7501148Z Description : This is an early preview. Run a Bash script on macOS, Linux, or Windows
2018-05-18T21:03:14.7513088Z Version : 3.127.0
2018-05-18T21:03:14.7524823Z Author : Microsoft Corporation
2018-05-18T21:03:14.7537179Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkID=613738)
2018-05-18T21:03:14.7549730Z ==============================================================================
2018-05-18T21:03:15.0174503Z Generating script.
2018-05-18T21:03:15.0535056Z Script contents:
2018-05-18T21:03:15.0547355Z apt-get install xvfb libgtk2.0-0 libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2
2018-05-18T21:03:15.0656822Z [command]/bin/bash --noprofile --norc /opt/vsts/work/_temp/cac4d3f9-42e7-49f3-94f6-7d0444827d83.sh
2018-05-18T21:03:15.6040707Z Reading package lists...
2018-05-18T21:03:15.6085335Z Building dependency tree...
2018-05-18T21:03:15.6153815Z Reading state information...
2018-05-18T21:03:15.6186788Z Package libgconf-2-4 is not available, but is referred to by another package.
2018-05-18T21:03:15.6198707Z This may mean that the package is missing, has been obsoleted, or
2018-05-18T21:03:15.6211380Z is only available from another source
2018-05-18T21:03:15.6216969Z
2018-05-18T21:03:15.6229592Z E: Unable to locate package xvfb
2018-05-18T21:03:15.6242128Z E: Unable to locate package libnotify-dev
2018-05-18T21:03:15.6254440Z E: Package 'libgconf-2-4' has no installation candidate
2018-05-18T21:03:15.6268141Z E: Unable to locate package libxss1
2018-05-18T21:03:15.6370826Z ##[error]Bash exited with code '100'.
2018-05-18T21:03:15.7283047Z ##[section]Finishing: Install cypress dependencies
Running apt-get update takes forever (25 minutes and still not done).
Any clues on how to get Cypress working quickly on the Hosted Agent is welcome.
Note: I tried running on Windows, that works, fortunately, so I'm not completely blocked. But to use the Windows Agent I now have 2 agent phases which adds overhead due to artefact downloads and npm install overhead.
Solved the issue using a pre-pended Shell (Bash) task in VSTS agent phase prior to running Cypress:
apt-get -qq -y update
apt-get -qq -y install xvfb libgtk2.0-0 libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2
I get the following error when attempting to install scipy to PyCharm 3.2:
Collecting scipy
Using cached scipy-0.19.0.zip
Installing collected packages: scipy
Running setup.py install for scipy: started
Running setup.py install for scipy: finished with status 'error'
Complete output from command C:\Users\leigh\AppData\Local\Programs\Python\Python36-32\python.exe -u -c "import setuptools, tokenize;file='C:\Users\leigh\AppData\Local\Temp\pycharm-packaging\scipy\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record C:\Users\leigh\AppData\Local\Temp\pip-8kx0fj8t-record\install-record.txt --single-version-externally-managed --compile:
Note: if you need reliable uninstall behavior, then install
with pip instead of using `setup.py install`:
- `pip install .` (from a git repo or downloaded source
release)
- `pip install scipy` (last SciPy release on PyPI)
Does anyone know how to deal with this?
I am not sure, but you might find your solution in this stackoverflow thread: Can't install Scipy through pip.
If your problem persists, assuming you're on Windows (10), you could try:
Clone library from Github.
git clone https://github.com/scipy/scipy.git
Using CMD, navigate to the scipy folder
cd C:\..(your path)..\scipy
Then install by
python setup.py install
Hope it works out for you!
I am running virtualenvwrapper and friends on Ubuntu 12.04 (virtualenvwrapper 1.7.1.2, virtualenv 1.7.1.2, pip 1.0, Distribute 0.6.24, Python 2.7). To be able to pip install matplotlib I need a newer version of Distribute/setuptools, so I do the following:
$ mkvirtualenv new_venv
$ pip install matplotlib # fails
$ easy_install -U distribute
# installs Distribute 0.7.3 in the venv,
# which then installs setuptools 1.3.1
$ pip install matplotlib # works
However, when I run pip freeze I get:
argparse==1.2.1
distribute==0.7.3 # here's Distribute
matplotlib==1.3.1
nose==1.3.0
numpy==1.8.0
pyparsing==2.0.1
python-dateutil==2.2
six==1.4.1
tornado==3.1.1
wsgiref==0.1.2
Where is setuptools? I've tried just pip install setuptools and it also fails to show up there.
pip won't list packages that pip itself depends on unless you tell it to. You may include such packages by doing:
pip freeze --all
freeze lists packages in a requirements-format - for use in requirements files etc - while list is just that, a list.
See Pip freeze vs. pip list for more details.
On Ubuntu setuptools is installed by the systems package manager. So it is not listed when you pip freeze. While creating a virtualenv, you will get these messages
mkvirtualenv test
New python executable in test/bin/python
Installing setuptools, pip...done.
If you do a pip freeze immediately after this, you will get something like this
argparse==1.2.1
wsgiref==0.1.2
Here setuptools and pip wont be listed.
If you want everything you can do a pip list which gives something like this.
argparse (1.2.1)
pip (1.5.6)
setuptools (3.6)
wsgiref (0.1.2)