These are the steps I am following:
mkdir spark_lib; cd spark_lib
pip install jsonpath_rw_ext==1.1.3 -t .
zip -r9 ../spark_lib.zip *
Initialize spark context variable sc
sc.addPyFile('spark_lib.zip')
def f(x):
import jsonpath_rw_ext
return jsonpath_rw_ext.match('$.a.b', x)
sc.parallelize([{"a":{"b":10}}]).map(f).collect()
This gives me a pbr versioning error
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 268, in dump_stream
vs = list(itertools.islice(iterator, batch))
File "<ipython-input-11-2619fc011e24>", line 6, in f
File "./spark_lib.zip/jsonpath_rw_ext/__init__.py", line 18, in <module>
File "./spark_lib.zip/pbr/version.py", line 467, in version_string
return self.semantic_version().brief_string()
File "./spark_lib.zip/pbr/version.py", line 462, in semantic_version
self._semantic = self._get_version_from_pkg_resources()
File "./spark_lib.zip/pbr/version.py", line 449, in _get_version_from_pkg_resources
result_string = packaging.get_version(self.package)
File "./spark_lib.zip/pbr/packaging.py", line 824, in get_version
name=package_name))
Exception: Versioning for this project requires either an sdist tarball,
or access to an upstream git repository. It's also possible that
there is a mismatch between the package name in setup.cfg
and the argument given to pbr.version.VersionInfo.
Project name jsonpath_rw_ext was given, but was not able to be found.
After reading through another similar bug https://bugs.launchpad.net/python-swiftclient/+bug/1379579, found out that pbr needs updated setuptools.
I included setuptools in my pip installation command and everything worked fine.
pip install jsonpath_rw_ext setuptools -t .
Related
add the development packages, build and get the development server running:
"PYTHON_VER is python3.7."
"Python 3 module install via pip"
"SYS_PYTHON is /root/.pyenv/shims/python3.7."
"ENV_PYTHON is /data/opt/hue/hue/build/env/bin/python3.7."
added in ~/.bashrc
export PYTHON_VER=python3.7
then
git clone https://github.com/cloudera/hue.git
cd hue
make apps
/data/opt/hue/hue/build/env/lib/python3.7/site-packages/_distutils_hack/__init__.py:19: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
"Distutils was imported before Setuptools, but importing Setuptools "
/data/opt/hue/hue/build/env/lib/python3.7/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")
Exception:
Traceback (most recent call last):
File "/data/opt/hue/hue/build/env/lib/python3.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/data/opt/hue/hue/build/env/lib/python3.7/site-packages/pip/commands/install.py", line 335, in run
wb.build(autobuilding=True)
File "/data/opt/hue/hue/build/env/lib/python3.7/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/data/opt/hue/hue/build/env/lib/python3.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/data/opt/hue/hue/build/env/lib/python3.7/site-packages/pip/req/req_set.py", line 634, in _prepare_file
abstract_dist.prep_for_dist()
File "/data/opt/hue/hue/build/env/lib/python3.7/site-packages/pip/req/req_set.py", line 129, in prep_for_dist
self.req_to_install.run_egg_info()
File "/data/opt/hue/hue/build/env/lib/python3.7/site-packages/pip/req/req_install.py", line 412, in run_egg_info
self.setup_py, self.name,
File "/data/opt/hue/hue/build/env/lib/python3.7/site-packages/pip/req/req_install.py", line 387, in setup_py
import setuptools # noqa
File "/data/opt/hue/hue/build/env/lib/python3.7/site-packages/setuptools/__init__.py", line 8, in <module>
import _distutils_hack.override # noqa: F401
File "/data/opt/hue/hue/build/env/lib/python3.7/site-packages/_distutils_hack/override.py", line 1, in <module>
__import__('_distutils_hack').do_override()
File "/data/opt/hue/hue/build/env/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 77, in do_override
ensure_local_distutils()
File "/data/opt/hue/hue/build/env/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 64, in ensure_local_distutils
assert '_distutils' in core.__file__, core.__file__
AssertionError: /root/.pyenv/versions/3.7.0/lib/python3.7/distutils/core.py
Answer1
pip3 install --upgrade pip setuptools
Answer2
pip uninstall pyzmq
# then
pip install pyzmq
it doesn't work,any idea?
I need to install robotframework-autoitlibrary to use on my test cases.
My problem is when I try to install AutoIt Library through command line with the following command:
pip install -U robotframework-autoitlibrary --no-cache-dir --pre
I have this error:
C:\windows\system32>pip install -U robotframework-autoitlibrary --no-cache-
dir --pre
Collecting robotframework-autoitlibrary
Downloading
https://files.pythonhosted.org/packages/4e/a4/9e51fe35b1da7a006b773c9c234f78e89bcc4f267152c4e9fa8260631fa8/robotframework-autoitlibrary-1.2.2.zip (701kB)
100% |################################| 706kB 1.6MB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\users\user-a~1\appdata\local\temp\pip-install-
oro1ov\robotframework-autoitlibrary\setup.py", line 93, in <module>
destPath = os.path.normpath(os.path.join(os.getenv("HOMEDRIVE"),
r"\RobotFramework\Extensions\AutoItLibrary"))
File "c:\python27\lib\ntpath.py", line 65, in join
result_drive, result_path = splitdrive(path)
File "c:\python27\lib\ntpath.py", line 115, in splitdrive
if len(p) > 1:
TypeError: object of type 'NoneType' has no len()
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in
c:\users\user-a~1\appdata\local\temp\pip-install-oro1ov\robotframework-
autoitlibrary\
My currently installed python packages and their versions are:
Pillow==5.3.0
Pygments==2.3.1
pypiwin32==223
Pypubsub==4.0.0
pywin32==224
robotframework==3.1
robotframework-ride==1.5.2.1
robotframework-selenium2library==3.0.0
robotframework-seleniumlibrary==3.3.0
selenium==3.141.0
six==1.12.0
typing==3.6.6
urllib3==1.24.1
wxPython==4.0.3
I already try this command:
pip install --upgrade setuptools
When input echo %HOMEDRIVE%, the output is:
C:\Users\cmpeixoto>echo %HOMEDRIVE%
C:
Thanks for your help,
The error looks like the environment variable HOMEDRIVE is not set, even though it has a value according to your test (the library installer uses it to copy some files).
Can you try this - manually set it, and straight after that run the pip - in the same Command Prompt (cmd) session:
set HOMEDRIVE=C:
pip install -U robotframework-autoitlibrary --no-cache-dir --pre
I have google-cloud-storage pip installed into a lib directory and vendored in. It's running just fine locally during development of my python appengine app. However, when trying to run unit tests via nose and testbed I'm getting "The 'google-cloud-core' distribution was not found and is required by the application". Here is the stack:
Traceback (most recent call last):
File "/Users/jason/dev/gain-data/venv/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/Users/jason/dev/gain-data/venv/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/Users/jason/dev/gain-data/venv/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/Users/jason/dev/gain-data/data/storage/__init__.py", line 4, in <module>
from google.cloud.storage import Blob, Client
File "/Users/jason/dev/gain-data/lib/google/cloud/storage/__init__.py", line 42, in <module>
from google.cloud.storage.batch import Batch
File "/Users/jason/dev/gain-data/lib/google/cloud/storage/batch.py", line 30, in <module>
from google.cloud.storage.connection import Connection
File "/Users/jason/dev/gain-data/lib/google/cloud/storage/connection.py", line 17, in <module>
from google.cloud import connection as base_connection
File "/Users/jason/dev/gain-data/lib/google/cloud/connection.py", line 31, in <module>
get_distribution('google-cloud-core').version)
File "/Users/jason/dev/gain-data/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 557, in get_distribution
dist = get_provider(dist)
File "/Users/jason/dev/gain-data/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 431, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
File "/Users/jason/dev/gain-data/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 968, in require
needed = self.resolve(parse_requirements(requirements))
File "/Users/jason/dev/gain-data/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 854, in resolve
raise DistributionNotFound(req, requirers)
DistributionNotFound: The 'google-cloud-core' distribution was not found and is required by the application
Any thoughts?
I had the same issue with google-cloud-translate, I was forced to also install the package "globally", i.e. pip install google-cloud-translate.
After struggling a lot with this same issue I found out that the error was because the vendor pip lib wasn't in the PYTHONPATH before calling the nosetests.
Try adding the vendor lib to the PYTHONPATH and then run the tests.
export PYTHONPATH="$(HOME)/Projects/myproject/pip_lib:$$PYTHONPATH"; \
nosetests .
Im trying to install aws for the mac command line, I guess I dont understand what I need to do I installed the aws bundle with wget on the terminal, unziped it did everything, but when I need to configure my credentials nothing comes up when I put aws configure..
Here are the Instructions:
http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html
Here is what is ouputed
an$ aws configuration
Traceback (most recent call last):
File "/usr/local/bin/aws", line 15, in <module>
import awscli.clidriver
File "/usr/local/aws/lib/python2.7/site-packages/awscli/clidriver.py", line 31, in <module>
from awscli.help import ProviderHelpCommand
File "/usr/local/aws/lib/python2.7/site-packages/awscli/help.py", line 20, in <module>
from docutils.core import publish_string
File "/usr/local/aws/lib/python2.7/site-packages/docutils/core.py", line 20, in <module>
from docutils import frontend, io, utils, readers, writers
File "/usr/local/aws/lib/python2.7/site-packages/docutils/frontend.py", line 41, in <module>
import docutils.utils
File "/usr/local/aws/lib/python2.7/site-packages/docutils/utils/__init__.py", line 20, in <module>
import docutils.io
File "/usr/local/aws/lib/python2.7/site-packages/docutils/io.py", line 18, in <module>
from docutils.utils.error_reporting import locale_encoding, ErrorString, ErrorOutput
File "/usr/local/aws/lib/python2.7/site-packages/docutils/utils/error_reporting.py", line 47, in <module>
locale_encoding = locale.getlocale()[1] or locale.getdefaultlocale()[1]
File "/usr/local/aws/lib/python2.7/locale.py", line 511, in getdefaultlocale
return _parse_localename(localename)
File "/usr/local/aws/lib/python2.7/locale.py", line 443, in _parse_localename
raise ValueError, 'unknown locale: %s' % localename
ValueError: unknown locale: UTF-8
Any Ideas_?
try adding below lines to ~/.bash_profile:
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
Installing AWSCLI in Windows Machine
I had similar issue with Windows 10 (64 bit). Python 3.5 and Python 2.7 are installed in my PC. I was getting ImportError: No module named awscli.clidriver.
Then I added %USERPROFILE%\AppData\Roaming\Python\Python35\Scripts in environment path variable and removed Python 2.7 from the environment path variable. Now I can successfully use awscli.
I have created a step by step AWSCLI installation guide in this Github repository: https://github.com/arsho/installation/tree/master/awscli_installation.
I had to install EKS supported version and I ended up getting it to work with ignoring six:
$ pip3 install awscli --ignore-installed six
In my case nothing worked, untill i gave more permissions, i run aws command with a non root user
chown amzadm.root /usr/bin/aws
chown amzadm.root -R /usr/lib/python2.6/site-packages/
chown amzadm.root -R /usr/lib/python2.6/site-packages/awscli/
I fixed this by adding a line to the 'aws' script just before the import (line 19). So now the file reads:
sys.path.append('/Users/<username>/.local/lib/aws/lib/python2.7/site-packages/')
import awscli.clidriver
This work for me. In /.barshrc file
export AWS_DEFAULT_OUTPUT="json"
I'm brand new to Python programming and trying to get myself a functional base from which I can run things like the IPython Notebook which looks pretty exciting.
Thus far I have both Python 2.7 and 3.3 from python.org installed in OS X 10.6 (Snow Leopard) as well as ActiveTcl 8.5.13. Almost everything that I've tried thus far works as expected. I'm focused on learning 3.3, but want to have the option of using 2.7 too. I read up in several documents that I need to start gaining access to PyPI packages using a Python package manager and that distribute is the one I should use for 3k. So I installed that according to the documentation I found and it seemed to work fine.
I also installed pip as directed, and a number of others.
At this point, I have:
$ pip freeze
distribute==0.6.34
ipython==0.13.1
nose==1.2.1 (installed after IPython)
pexpect==2.4 (installed after IPython)
pyflakes3k==0.4.3
readline==6.2.4.1 (installed after IPython)
At this point, I'm doing this from ipython.org guidance
And when I did $ easy_install pexpect, I got a bunch of errors:
$ easy_install pexpect
Searching for pexpect
Reading http://pypi.python.org/simple/pexpect/
Reading http://pexpect.sourceforge.net/
Reading http://sourceforge.net/project/showfiles.php?group_id=59762
Best match: pexpect 2.4
Downloading http://pypi.python.org/packages/source/p/pexpect/pexpect-2.4.tar.gz#md5=fe82d69be19ec96d3a6650af947d5665
Processing pexpect-2.4.tar.gz
Writing /var/folders/td/td0Sh8EfGFuMCnKex1v+q++++TI/-Tmp-/easy_install-s4dtyy/pexpect-2.4/setup.cfg
Running pexpect-2.4/setup.py -q bdist_egg --dist-dir /var/folders/td/td0Sh8EfGFuMCnKex1v+q++++TI/-Tmp-/easy_install-s4dtyy/pexpect-2.4/egg-dist-tmp-5h5cg4
File "build/bdist.macosx-10.6-intel/egg/fdpexpect.py", line 36
raise ExceptionPexpect, 'The fd argument is not a valid file descriptor.'
^
SyntaxError: invalid syntax
File "build/bdist.macosx-10.6-intel/egg/FSM.py", line 77
return `self.value`
^
SyntaxError: invalid syntax
File "build/bdist.macosx-10.6-intel/egg/pexpect.py", line 82
except ImportError, e:
^
SyntaxError: invalid syntax
zip_safe flag not set; analyzing archive contents...
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pexpect-2.4-py3.3.egg/fdpexpect.py", line 36
raise ExceptionPexpect, 'The fd argument is not a valid file descriptor.'
^
SyntaxError: invalid syntax
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pexpect-2.4-py3.3.egg/FSM.py", line 77
return `self.value`
^
SyntaxError: invalid syntax
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pexpect-2.4-py3.3.egg/pexpect.py", line 82
except ImportError, e:
^
SyntaxError: invalid syntax
Adding pexpect 2.4 to easy-install.pth file
Installed /Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pexpect-2.4-py3.3.egg
Processing dependencies for pexpect
Finished processing dependencies for pexpect
That looks bad to me (although I don't yet have the expertise to really interpret it), and so I'm not sure if I have a complete install of pexpect.
After installing nose (before pexpect as per URL above), I tried running iptest and iptest3 from the command line, and both failed to find the command, but after I did easy_install ipython again (after nose), I noticed that this install added iptest3 (as well as ipcluster3 and a few other scripts) to my path, and now my bash shell can find iptest3, but when I run it, I get some more bad-looking output:
$ iptest3
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.3/bin/iptest3", line 9, in <module>
load_entry_point('ipython==0.13.1', 'console_scripts', 'iptest3')()
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/distribute-0.6.34-py3.3.egg/pkg_resources.py", line 343, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/distribute-0.6.34-py3.3.egg/pkg_resources.py", line 2308, in load_entry_point
return ep.load()
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/distribute-0.6.34-py3.3.egg/pkg_resources.py", line 2014, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/__init__.py", line 43, in <module>
from .config.loader import Config
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/config/__init__.py", line 16, in <module>
from .application import *
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/config/application.py", line 31, in <module>
from IPython.config.configurable import SingletonConfigurable
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/config/configurable.py", line 26, in <module>
from .loader import Config
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/config/loader.py", line 27, in <module>
from IPython.utils.path import filefind, get_ipython_dir
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/utils/path.py", line 25, in <module>
from IPython.utils.process import system
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/utils/process.py", line 27, in <module>
from ._process_posix import _find_cmd, system, getoutput, arg_split
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/utils/_process_posix.py", line 22, in <module>
from IPython.external import pexpect
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/external/pexpect/__init__.py", line 2, in <module>
import pexpect
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pexpect-2.4-py3.3.egg/pexpect.py", line 82
except ImportError, e:
^
SyntaxError: invalid syntax
After doing all that, I noted that my /Library/Frameworks/Python.framework/Versions/Current had been pointing to 2.7, and I guessed that that might be related to my problems and changed the symbolic link to point to 3.3, but iptest3 still fails with the error above.
Any other thoughts on what to do to fix this? It's clear that iptest is pretty important to doing anything else (like IPython Notebook) I want to do.
There is a py3-compatible fork of pexpect called pexpect-u (the u is for unicode, the main difference between the two). You need this to run the pexpect-based parts of IPython on Python 3.
Should be a simple
pip install pexpect-u
side note: pexpect-u is by IPython developer Thomas Kluyver, who did most of the heavy lifting bringing py3 compatibility to IPython.