I just now installed apache-spark 3.1.2 on ubuntu 16.04. After installation and setting up PATH, SPARK_HOME, PYSPARK_PYTHON environment variables, when I try to launch pyspark I am getting below error:
$ $SPARK_HOME/bin/pyspark
Python 3.5.2 (default, Jan 26 2021, 13:30:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
File "/opt/spark/python/pyspark/shell.py", line 29, in <module>
from pyspark.context import SparkContext
File "/opt/spark/python/pyspark/__init__.py", line 53, in <module>
from pyspark.rdd import RDD, RDDBarrier
File "/opt/spark/python/pyspark/rdd.py", line 48, in <module>
from pyspark.traceback_utils import SCCallSiteSync
File "/opt/spark/python/pyspark/traceback_utils.py", line 23, in <module>
CallSite = namedtuple("CallSite", "function file linenum")
File "/opt/spark/python/pyspark/serializers.py", line 390, in namedtuple
for k, v in _old_namedtuple_kwdefaults.items():
AttributeError: 'NoneType' object has no attribute 'items'
Error in sys.excepthook:
I get same error when I try to import pyspark from python3
$ python3
Python 3.5.2 (default, Jan 26 2021, 13:30:48)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyspark
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.5/dist-packages/pyspark/__init__.py", line 53, in <module>
from pyspark.rdd import RDD, RDDBarrier
File "/usr/local/lib/python3.5/dist-packages/pyspark/rdd.py", line 48, in <module>
from pyspark.traceback_utils import SCCallSiteSync
File "/usr/local/lib/python3.5/dist-packages/pyspark/traceback_utils.py", line 23, in <module>
CallSite = namedtuple("CallSite", "function file linenum")
File "/usr/local/lib/python3.5/dist-packages/pyspark/serializers.py", line 390, in namedtuple
for k, v in _old_namedtuple_kwdefaults.items():
AttributeError: 'NoneType' object has no attribute 'items'
Error in sys.excepthook:
I am clearly missing some dependency/configuration but unable to figure out which one?
I did notice that pyspark command references
/opt/spark/python/pyspark/
while python3 references
/usr/local/lib/python3.5/dist-packages/pyspark
Spark home is /opt/spark. spark-shell is working perfectly fine by the way. So it appears that the issue is with python.
Update: I installed spark 3.0.3 and the issue isn't happening. Looks like this is something specific to 3.1.2
According to the documentation, Python 3.5.2 is not supported by Spark 3.1.2. Simply update Python to version 3.6 or above.
Related
Tried to open file with unicode name on moviepy v1.0.0 and v1.0.1. Ubuntu does not work. Mac works.
Ubuntu:
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from moviepy.video.io.VideoFileClip import VideoFileClip
>>> videopath='PSY - GANGNAM STYLE(강남스타일) M V.mp4'
>>> clip = VideoFileClip(videopath)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../lib/python3.6/site-packages/moviepy/video/io/VideoFileClip.py", line 91, in __init__
fps_source=fps_source)
File ".../lib/python3.6/site-packages/moviepy/video/io/ffmpeg_reader.py", line 33, in __init__
fps_source)
File ".../lib/python3.6/site-packages/moviepy/video/io/ffmpeg_reader.py", line 276, in ffmpeg_parse_infos
"path.")%filename)
OSError: MoviePy error: the file PSY - GANGNAM STYLE(강남스타일) M V.mp4 could not be found!
Please check that you entered the correct path.
ffmpeg -version
ffmpeg version 4.1.3-0york1~14.04 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04.4)
Mac:
Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 14:01:38)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from moviepy.video.io.VideoFileClip import VideoFileClip
>>> videopath='PSY - GANGNAM STYLE(강남스타일) M V.mp4'
>>> clip = VideoFileClip(videopath)
>>>
ffmpeg -version
ffmpeg version 4.1.3 Copyright (c) 2000-2019 the FFmpeg developers
built with Apple LLVM version 10.0.1 (clang-1001.0.46.3)
How to make moviepy to open unicode file name on Ubuntu?
Update: The issue is name. It looks the same when copy/paste, but it seems has different encoding.
>>> videopath1
'PSY - GANGNAM STYLE(강남스타일) M V.mp4'
>>> videopath
'PSY - GANGNAM STYLE(강남스타일) M V.mp4'
>>>
>>>
>>> clip = VideoFileClip(videopath1)
>>> clip = VideoFileClip(videopath)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "../lib/python3.6/site-packages/moviepy/video/io/VideoFileClip.py", line 91, in __init__
fps_source=fps_source)
File "../lib/python3.6/site-packages/moviepy/video/io/ffmpeg_reader.py", line 33, in __init__
fps_source)
File "../lib/python3.6/site-packages/moviepy/video/io/ffmpeg_reader.py", line 276, in ffmpeg_parse_infos
"path.")%filename)
OSError: MoviePy error: the file PSY - GANGNAM STYLE(강남스타일) M V.mp4 could not be found!
Please check that you entered the correct path.
For above 2 names, if copying to clipboard, somehow they are different. Therefore, it has nothing to do with moviepy.
Encoding is the same (namely UTF-8), the difference is that videopath1 is composed and videopath is decomposed.
"PSY - GANGNAM STYLE(\x{ac15}\x{b0a8}\x{c2a4}\x{d0c0}\x{c77c}) M V.mp4"
"PSY - GANGNAM STYLE(\x{1100}\x{1161}\x{11bc}\x{1102}\x{1161}\x{11b7}\x{1109}\x{1173}\x{1110}\x{1161}\x{110b}\x{1175}\x{11af}) M V.mp4"
My issue is about the import of scipy
Reproducing code example:
import scipy
Error message:
Traceback (most recent call last):
File "D:/Projects/gait/tests/__init__.py", line 2, in <module>
import scipy
File "C:\Users\Leynier\AppData\Local\Programs\Python\Python37-32\lib\site-packages\scipy\__init__.py", line 110, in <module>
from scipy._lib._version import NumpyVersion as _NumpyVersion
File "C:\Users\Leynier\AppData\Local\Programs\Python\Python37-32\lib\site-packages\scipy\_lib\__init__.py", line 14, in <module>
from scipy._lib._testutils import PytestTester
ValueError: source code string cannot contain null bytes
Process finished with exit code 1
Scipy/Numpy/Python version information:
Scipy version: 1.3.0
Numpy version: 1.16.4
Python version: 3.7.3
OS: Windows 10
I'm setting up awslogs agent on ec2 instance, When i run the python script of awslogs. I'm getting below message.
Downloading the latest CloudWatch Logs agent bits ... ERROR: Failed to create virtualenv. Try manually installing with pip and adding it to the sudo user's PATH before running this script.
And awslogs-agent-setup.log show below error.
Environment: CentOS 6.10 and Python 2.6
Traceback (most recent call last):
File "/usr/bin/pip", line 7, in <module>
from pip._internal import main
File "/usr/lib/python2.6/site-packages/pip-19.0.3-py2.6.egg/pip/_internal/__init__.py", line 19, in <module>
from pip._vendor.urllib3.exceptions import DependencyWarning
File "/usr/lib/python2.6/site-packages/pip-19.0.3-py2.6.egg/pip/_vendor/urllib3/__init__.py", line 8, in <module>
from .connectionpool import (
File "/usr/lib/python2.6/site-packages/pip-19.0.3-py2.6.egg/pip/_vendor/urllib3/connectionpool.py", line 92
_blocking_errnos = {errno.EAGAIN, errno.EWOULDBLOCK}
^
SyntaxError: invalid syntax
/usr/bin/virtualenv
Traceback (most recent call last):
File "/usr/bin/virtualenv", line 7, in <module>
from virtualenv import main
File "/usr/lib/python2.6/site-packages/virtualenv.py", line 51, in <module>
print("ERROR: {}".format(sys.exc_info()[1]))
ValueError: zero length field name in format
Basically, this error is due to your python version 2.6. Could you please update your python version from 2.6 to 2.7 or 3.1.
This should help.
Traceback (most recent call last):
File "/Users/codymitchell/Documents/code/py/zipline/venv/bin/jupyter-notebook", line 6, in <module>
from notebook.notebookapp import main
File "/Users/codymitchell/Documents/code/py/zipline/venv/lib/python3.5/site-packages/notebook/__init__.py", line 25, in <module>
from .nbextensions import install_nbextension
File "/Users/codymitchell/Documents/code/py/zipline/venv/lib/python3.5/site-packages/notebook/nbextensions.py", line 610, in <module>
from .extensions import (
File "/Users/codymitchell/Documents/code/py/zipline/venv/lib/python3.5/site-packages/notebook/extensions.py", line 8, in <module>
from tornado.log import LogFormatter
File "/Users/codymitchell/Documents/code/py/zipline/venv/lib/python3.5/site-packages/tornado/log.py", line 34, in <module>
from tornado.escape import _unicode
File "/Users/codymitchell/Documents/code/py/zipline/venv/lib/python3.5/site-packages/tornado/escape.py", line 27, in <module>
from tornado.util import unicode_type
File "/Users/codymitchell/Documents/code/py/zipline/venv/lib/python3.5/site-packages/tornado/util.py", line 21, in <module>
from typing import (
ImportError: cannot import name 'Type'
I had the same issue, here's how I solved it
Python 3.5.1 doesn't have typing.Type
https://github.com/django-extensions/django-extensions/issues/1188
C:\Users\rigupta>D:\Anaconda3\python
Python 3.5.1 |Anaconda 4.0.0 (64-bit)| (default, Feb 16 2016, 09:49:46) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import typing
>>> typing.Type
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'typing' has no attribute 'Type'
>>> typing.type
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'typing' has no attribute 'type'
>>>
so I upgraded my python version to 3.5.4 by running below command which resolved the issue
conda update --all
https://www.youtube.com/watch?v=Q-dZCAk1tfc
I had the exact same problem. I am running windows 7 64-bit, running on Portable Python 3.7.3.
When Jupyter was failing, I was running Portable Python at this path:
E:\applications\python37\Portable Python-3.7.3\App\Python
Uninstalling and reinstalling Jupyter, typing modules did not solve the problem.
I was suspicions of problematic path-names, so I copied the python installation to a path without whitespaces (and maybe a shorter path?):
E:\applications\python37\Python
Finally, I managed to get Jupyter Notebook to work!
Upon installing python xy 2.7.5 on my laptop, I tried to launch the Spyder IDE. This yielded the following error:
C:\Python27\Scripts>spyder.exe
Traceback (most recent call last):
File "C:\Python27\Scripts\spyder-script.py", line 9, in
load_entry_point('spyder==2.2.0', 'console_scripts', 'spyder')()
File "C:\Python27\lib\site-packages\spyderlib\start_app.py", line 72, in main
from spyderlib import spyder
File "C:\Python27\lib\site-packages\spyderlib\spyder.py", line 112, in
from spyderlib.plugins.ipythonconsole import IPythonConsole
File "C:\Python27\lib\site-packages\spyderlib\plugins\ipythonconsole.py", line 31, in
from IPython.frontend.qt.kernelmanager import QtKernelManager
File "C:\Python27\lib\site-packages\IPython\frontend\qt\kernelmanager.py", line 9, in
from IPython.zmq.kernelmanager import KernelManager, SubSocketChannel, \
File "C:\Python27\lib\site-packages\IPython\zmq\__init__.py", line 67, in
check_for_zmq('2.1.4')
File "C:\Python27\lib\site-packages\IPython\zmq\__init__.py", line 51, in check_for_zmq
import zmq
File "C:\Python27\lib\site-packages\zmq\__init__.py", line 35, in
_libzmq = ctypes.CDLL(bundled[0], mode=ctypes.RTLD_GLOBAL)
File "C:\Python27\lib\ctypes\__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
WindowsError: [Error 127] Die angegebene Prozedur wurde nicht gefunden
Using a python console, I was able to run python code as well as matplotlib and numpy code. Ipython works, however Ipython (Qt) does not work. what could be the reason for this?
The host OS is Windows XP.
This is actually a problem with the most recent builds of zeromq, which are messaging libraries that are required for IPython, and not with Spyder itself. The latest zeromq builds are not compatible with Win XP. This is a know issue with recent versions of Python(x,y) on Win XP which can be solved by following the guidance in this bug report.
It works by installing
https://pythonxy.googlecode.com/files/pyzmq-13.0.2-4_py27.exe
and
http://h5py.googlecode.com/files/h5py-2.1.3.win32-py2.7.msi
,
As it is posted in http://code.google.com/p/pythonxy/issues/detail?id=670