I'm trying to run a pyspark job in a local enviroment.
After setting up pipenv and installing the module (numpy) successfully, the module still not visible to the code.
Using pip to install the library instead of pipenv works. What am I missing here?
The terminal output is shown below.
PS C:\Users\user\Desktop\spark\test> pipenv shell
Shell for C:\Users\user\.virtualenvs\test-sCQB0P3C already activated.
No action taken to avoid nested environments.
PS C:\Users\user\Desktop\spark\test> pipenv graph
numpy==1.20.3
pipenv==2020.11.15
- certifi [required: Any, installed: 2020.12.5]
- pip [required: >=18.0, installed: 21.1.1]
- setuptools [required: >=36.2.1, installed: 56.0.0]
- virtualenv [required: Any, installed: 20.4.6]
- appdirs [required: >=1.4.3,<2, installed: 1.4.4]
- distlib [required: >=0.3.1,<1, installed: 0.3.1]
- filelock [required: >=3.0.0,<4, installed: 3.0.12]
- six [required: >=1.9.0,<2, installed: 1.16.0]
- virtualenv-clone [required: >=0.2.5, installed: 0.5.4]
pyspark==2.4.0
- py4j [required: ==0.10.7, installed: 0.10.7]
PS C:\Users\user\Desktop\spark\test> spark-submit --master local[*] --files
configs\etl_config.json jobs\etl_job.py
Traceback (most recent call last):
File "C:/Users/user/Desktop/spark/test/jobs/etl_job.py", line 40, in <module>
from dependencies.class import XLoader
File "C:\Users\user\Desktop\spark\test\dependencies\X.py", line 2, in <module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
make sure you are in the same directory as your Pipfile
pipenv shell
and then
pipenv install
Related
(base) C:\Users\dadas>pip3 install scipy
Requirement already satisfied: scipy in c:\users\dadas\appdata\local\programs\python\python38-32\lib\site-packages (1.5.2)
Requirement already satisfied: numpy>=1.14.5 in c:\users\dadas\appdata\local\programs\python\python38-32\lib\site-packages (from scipy) (1.19.1)
(base) C:\Users\dadas>pip install scipy
Requirement already satisfied: scipy in c:\programdata\anaconda3\lib\site-packages (1.4.1)
Requirement already satisfied: numpy>=1.13.3 in c:\programdata\anaconda3\lib\site-packages (from scipy) (1.18.1)
(base) C:\Users\dadas>python
Python 3.7.6 (default, Jan 8 2020, 20:23:39) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
import scipy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\ProgramData\Anaconda3\lib\site-packages\scipy\__init__.py", line 138, in <module>
from scipy.version import version as __version__
ModuleNotFoundError: No module named 'scipy.version'
If I correctly understand, you have installed python 3.8 and when you run pip3 or pip it goes to this install.
Nevertheless you also have Python 3.7.6 installed which seems to be your base python (when you run python this is this version that is running).
Try to run this code from the command:
python -m pip install scipy
I'm both updating and installing packages/libaries, but are only able to use them through the Python3.6 console.
10:40 ~ $ pip3.6 install -U --user scikit-learn
Requirement already up-to-date: scikit-learn in ./.local/lib/python3.6/site-packages
Requirement already up-to-date: numpy>=1.11.0 in ./.local/lib/python3.6/site-packages (from scikit-learn)
Requirement already up-to-date: joblib>=0.11 in ./.local/lib/python3.6/site-packages (from scikit-learn)
Requirement already up-to-date: scipy>=0.17.0 in ./.local/lib/python3.6/site-packages (from scikit-learn)
When using pip install it fails with :
10:40 ~ $ pip install -U --user scikit-learn
Collecting scikit-learn
Using cached https://files.pythonhosted.org/packages/18/28/5a48b00599b476875415b97bdfdb3849bafb31183c1d785501dbc8a77aa2/scikit-learn-0.22.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-JYw8w1/scikit-learn/setup.py", line 47, in <module>
import sklearn
File "sklearn/__init__.py", line 81, in <module>
from . import __check_build # noqa: F401
File "sklearn/__check_build/__init__.py", line 46, in <module>
raise_build_error(e)
File "sklearn/__check_build/__init__.py", line 41, in raise_build_error
%s""" % (e, local_dir, ''.join(dir_content).strip(), msg))
ImportError: No module named _check_build
___________________________________________________________________________
Contents of sklearn/__check_build:
__init__.py _check_build.pyx __init__.pyc
setup.py
___________________________________________________________________________
It seems that scikit-learn has not been built correctly.
If you have installed scikit-learn from source, please do not forget
to build the package before using it: run `python setup.py install` or
`make` in the source directory.
It appears that you are importing a local scikit-learn source tree. For
this, you need to have an inplace install. Maybe you are in the source
directory and you need to try from another location.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-JYw8w1/scikit-learn/
If you've installed a package using pip3.6, make sure that your kernel version for your notebook is 3.6 -- if it's (say) 3.7, then it won't work, because each version of Python has its own separate set of installed modules.
I imported the notebook, and forgot to change Kernel Python3.6. It was still on Python3. Changed notebook to 3.6, and the it used the updated libary's-
When you run pip on PythonAnywhere, outside virtual environment, it installs packages for python2. The latest version of scikit-learn to work with python2 is 0.20.4. So, install it with pip install scikit-learn==0.20.4
You tried to install 0.22.1 with python2.
While Installing Man in the Middle Framework (MITMf) i get an ImportError:No Module named capstone..
Although Capstone is already installed in my Kali Machine
I downloaded MITMf from https://github.com/byt3bl33d3r/MITMf
Heres what i get
root#kali:~/Desktop/MITMf# python mitmf.py --help
Traceback (most recent call last):
File "mitmf.py", line 36, in <module>
from plugins import *
File "/root/Desktop/MITMf/plugins/filepwn.py", line 72, in <module>
from libs.bdfactory import pebin
File "/root/Desktop/MITMf/libs/bdfactory/pebin.py", line 49, in <module>
from intel.intelCore import intelCore
File "/root/Desktop/MITMf/libs/bdfactory/intel/intelCore.py", line 38, in <module>
from capstone import *
ImportError: No module named capstone
root#kali:~/Desktop/MITMf# pip install capstone
Requirement already satisfied: capstone in /usr/lib/python3/dist-packages (3.0.5)
root#kali:~/Desktop/MITMf#
You need to install the python2.7 version of capstone. It looks like your python installation is a little weird. On Debian systems such as Kali, pip should be the installer for python2 packages and pip3 should be the installer for python3 packages.
From the pip man page:
On Debian, pip is the command to use when installing packages for Python 2, while pip3 is the command to use when installing packages for Python 3.
You probably should fix your pip installation by linking pip2 to pip. The way we do this is removing the pip binary and creating a symlink from pip2:
[k#box]$ sudo rm /usr/bin/pip
[k#box]$ sudo ln -s /usr/bin/pip2 /usr/bin/pip
Then you should be able to install the python2.7 version with pip
[k#box]$ pip install capstone
Explanation:
You have installed the python3.6 version of capstone. You can see in the output of your pip install command:
root#kali:~/Desktop/MITMf# pip install capstone
Requirement already satisfied: capstone in /usr/lib/python3/dist-packages (3.0.5)
If you look at the script mitmf.py, you will notice that the top line specifies python2.7
[k#box]$ head -n3 mitmf.py
#!/usr/bin/env python2.7
# Copyright (c) 2014-2016 Moxie Marlinspike, Marcello Salvati
I'm using Ubuntu 17.04 on Dell Laptop.
I'm trying to install the cloudstorage module.
When installing the module with Python2 using sudo, it works, however on importing it fails:
python storage2.py
Traceback (most recent call last):
File "storage2.py", line 6, in
import cloudstorage as gcs
File "/home/maxim/.local/lib/python2.7/site-packages/cloudstorage/init.py", line 56
def get_driver(driver: DriverName) -> Drivers:
Also, when I try installing this module with Python3 it fails:
https://apaste.info/QKYD
Why the import fails and how to fix it?
How to install this module in Python3?
I cannot use package parse-rest.
Module parse_rest.connection is not found.
I installed it using pip install parse-rest.
Could you tell me what is missing, please!!!
looking at H:\py_env\ where is my virtualenv installed it does not exists.
ImportError at /
No module named parse_rest.connection
Request Method: GET
Request URL: local
Django Version: 1.7.1
Exception Type: ImportError
Exception Value:
No module named parse_rest.connection
Exception Location: H:\py_env\src\propiedades\models.py in <module>, line 6
Python Executable: C:\Python27\python.exe
Python Version: 2.7.8
Python Path:
['H:\\py_env\\src', ...]
********************************************
(py_env) H:\py_env>pip install parse_rest
Downloading/unpacking parse-rest
Downloading parse_rest-0.10.2013.tar.gz
Running setup.py (path:H:\py_env\build\parse-rest\setup.py) egg_info for package parse-rest
Installing collected packages: parse-rest
Running setup.py install for parse-rest
Successfully installed parse-rest
Cleaning up...
(py_env) H:\py_env>pip freeze
Django==1.7.1
parse-rest==0.10.2013
(py_env) H:\py_env>pip show --files parse_rest
(py_env) H:\py_env>pip show --files parse-rest
---
Name: parse-rest
Version: 0.10.2013
Location: h:\py_env\lib\site-packages
Requires:
Files:
..\parse_rest\installation.py
..\parse_rest\query.py
..\parse_rest\settings_local.py
..\parse_rest\tests.py
..\parse_rest\__init__.py
..\parse_rest\installation.pyc
..\parse_rest\query.pyc
..\parse_rest\settings_local.pyc
..\parse_rest\tests.pyc
..\parse_rest\__init__.pyc
.\
dependency_links.txt
PKG-INFO
SOURCES.txt
top_level.txt
(py_env) H:\py_env>
I was running "manage.py runserver" instead of "python manage.py runsever" and it run the default Python located in c:\ file, not in the virtualenv.