I am starting with pySpark. I have installed it in anadonda in Win10. I have copied an example and when I execute the code, I am getting this error:
Traceback (most recent call last):
File ".\testingSpark.py", line 7, in <module>
spark = SparkSession.builder.master("local").getOrCreate()
File "D:\Windows\Anaconda3\lib\site-packages\pyspark\sql\session.py", line 173, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File "D:\Windows\Anaconda3\lib\site-packages\pyspark\context.py", line 331, in getOrCreate
SparkContext(conf=conf or SparkConf())
File "D:\Windows\Anaconda3\lib\site-packages\pyspark\context.py", line 118, in __init__
conf, jsc, profiler_cls)
File "D:\Windows\Anaconda3\lib\site-packages\pyspark\context.py", line 188, in _do_init
self._javaAccumulator = self._jvm.PythonAccumulatorV2(host, port)
TypeError: 'JavaPackage' object is not callable
I have read about it, but I can find nothing to fix this error. Please, help me!
Related
I recently wrote a package which is on 1.2.9 but when trying to import it using pip install rolimons Python tries to import 1.2.9 which fails and moves on until version 1.1.3 which is the only version that doesn't error out.
Here is the error:
ERROR: Command errored out with exit status 1:
command: /home/runner/rolimonspy-test/venv/bin/python3 /home/runner/rolimonspy-test/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp89bhulpc
cwd: /tmp/pip-install-baj2a_r_/rolimons_41a3df9c7f414766a57996a21dc8cd26
Complete output (16 lines):
Traceback (most recent call last):
File "/home/runner/rolimonspy-test/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 280, in <module>
main()
File "/home/runner/rolimonspy-test/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 263, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/runner/rolimonspy-test/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 133, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-029qi198/overlay/lib/python3.8/site-packages/poetry/core/masonry/api.py", line 41, in prepare_metadata_for_build_wheel
builder = WheelBuilder(poetry)
File "/tmp/pip-build-env-029qi198/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 56, in __init__
super().__init__(poetry, executable=executable)
File "/tmp/pip-build-env-029qi198/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/builder.py", line 83, in __init__
self._module = Module(
File "/tmp/pip-build-env-029qi198/overlay/lib/python3.8/site-packages/poetry/core/masonry/utils/module.py", line 69, in __init__
raise ModuleOrPackageNotFound(
poetry.core.masonry.utils.module.ModuleOrPackageNotFound: No file/folder found for package python-template
More information about the structure and package here:
I'm trying to initialise spark using pyspark in Spyder (IDE) but always get error. (OS - Ubuntu 16.04 LTS)
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
File "<ipython-input-7-86b3963ebca1>", line 3, in <module>
.appName("test_import")\
File "/home/fly/anaconda3/lib/python3.7/site-packages/pyspark/sql/session.py", line 173, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File "/home/fly/anaconda3/lib/python3.7/site-packages/pyspark/context.py", line 349, in getOrCreate
SparkContext(conf=conf or SparkConf())
File "/home/fly/anaconda3/lib/python3.7/site-packages/pyspark/context.py", line 118, in __init__
conf, jsc, profiler_cls)
File "/home/fly/anaconda3/lib/python3.7/site-packages/pyspark/context.py", line 187, in _do_init
self._accumulatorServer = accumulators._start_update_server(auth_token)
File "/home/fly/anaconda3/lib/python3.7/site-packages/pyspark/accumulators.py", line 291, in _start_update_server
server = AccumulatorServer(("localhost", 0), _UpdateRequestHandler, auth_token)
File "/home/fly/anaconda3/lib/python3.7/site-packages/pyspark/accumulators.py", line 274, in __init__
SocketServer.TCPServer.__init__(self, server_address, RequestHandlerClass)
File "/home/fly/anaconda3/lib/python3.7/socketserver.py", line 452, in __init__
self.server_bind()
File "/home/fly/anaconda3/lib/python3.7/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
gaierror: [Errno -2] Name or service not known
This is most likely because you aren't able to connect with the spark-master. Go to your /etc/hosts file and add this line to it.
127.0.0.1 localhost
I am using django-pipeline, On running "sudo python manage.py collectstatic"
Getting this error:
Traceback (most recent call last):
File "manage.py", line 10, in <module> execute_from_command_line(sys.argv)
File "/Users/office/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line utility.execute()
File "/Users/office/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/office/lib/python2.7/site-packages/django/core/management/base.py", line 390, in run_from_argv self.execute(*args, **cmd_options)
File "/Users/office/lib/python2.7/site-packages/django/core/management/base.py", line 441, in execute output = self.handle(*args, **options)
File "/Users/office/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle collected = self.collect()
File "/Users/office/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 120, in collect raise processed ValueError: The file 'bower_components/eonasdan-bootstrap-datetimepicker/build/fonts/glyphicons-halflings-regular.eot' could not be found with <pipeline.storage.PipelineCachedStorage object at 0x10d274e10>.
Any idea how to fix this ?
Using django-pipeline-forgiving(https://pypi.python.org/pypi/django-pipeline-forgiving) solved the issue.
pip install django-pipeline-forgiving
Set in your settings.py: STATICFILES_STORAGE = 'django_pipeline_forgiving.storages.PipelineForgivingStorage'
REPL run ipython an error occurred.
WARNING: Readline services not available or not loaded.
Traceback (most recent call last):
File "C:\Program Files\Sublime Text 3\Data\Packages/SublimeREPL/config/Python/ipy_repl.py", line 52, in <module>
embedded_shell.initialize()
File "<string>", line 2, in initialize
File "D:\Python27\lib\site-packages\IPython\config\application.py", line 75, in catch_config_error
return method(app, *args, **kwargs)
File "D:\Python27\lib\site-packages\IPython\terminal\ipapp.py", line 332, in initialize
self.init_shell()
File "D:\Python27\lib\site-packages\IPython\terminal\console\app.py", line 114, in init_shell
client=self.kernel_client,
File "D:\Python27\lib\site-packages\IPython\config\configurable.py", line 337, in instance
inst = cls(*args, **kwargs)
File "D:\Python27\lib\site-packages\IPython\core\interactiveshell.py", line 524, in __init__
self.init_completer()
File "D:\Python27\lib\site-packages\IPython\terminal\console\interactiveshell.py", line 129, in init_completer
self.Completer = ZMQCompleter(self, self.client, config=self.config)
File "D:\Python27\lib\site-packages\IPython\terminal\console\completer.py", line 37, in __init__
self.readline.set_completer_delims('\r\n')
AttributeError: 'module' object has no attribute 'set_completer_delims'
I experienced this problem whilst running ipython console in the WinPython distribution. In my case the problem was fixed by installing pyreadline:
pip install pyreadline
Hope this works for you too.
I was running ipython successfully on fedora 18 until now: I'm getting the following exception when trying to launch it:
Traceback (most recent call last):
File "/usr/bin/ipython", line 9, in <module>
load_entry_point('ipython==1.1.0', 'console_scripts', 'ipython')()
File "/usr/lib/python2.7/site-packages/IPython/__init__.py", line 118, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "/usr/lib/python2.7/site-packages/IPython/config/application.py", line 544, in launch_instance
app.initialize(argv)
File "<string>", line 2, in initialize
File "/usr/lib/python2.7/site-packages/IPython/config/application.py", line 89, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/IPython/terminal/ipapp.py", line 323, in initialize
self.init_shell()
File "/usr/lib/python2.7/site-packages/IPython/terminal/ipapp.py", line 339, in init_shell
ipython_dir=self.ipython_dir, user_ns=self.user_ns)
File "/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 349, in instance
inst = cls(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/IPython/terminal/interactiveshell.py", line 320, in __init__
**kwargs
File "/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 456, in __init__
self.init_history()
File "/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 1487, in init_history
self.history_manager = HistoryManager(shell=self, parent=self)
File "/usr/lib/python2.7/site-packages/IPython/core/history.py", line 481, in __init__
self.new_session()
File "<string>", line 2, in new_session
File "/usr/lib/python2.7/site-packages/IPython/core/history.py", line 65, in needs_sqlite
return f(self, *a, **kw)
File "/usr/lib/python2.7/site-packages/IPython/core/history.py", line 500, in new_session
self.session_number = cur.lastrowid
OperationalError: disk I/O error
If you suspect this is an IPython bug, please report it at:
https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev#scipy.org
You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.
Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
c.Application.verbose_crash=True
I tried to upgrade ipython to the latest version using pip, which did not help. Any solution or workaround is very much welcomed.
IPython store history in a profile generally in ~/.ipython/profile_default/history.sqlite. There seem to be a disk error reading/writting to it.
Check the permissions of the file/folders, if necessary delete the file.
I solved it by set c.NotebookNotary.db_file = u':memory:' in jupyter configure file ~/.jupyter/jupyter_notebook_config.py