I'm barely out of infancy in python and trying to figure out how to package a few of my tools for distributing, and having a hard time understanding the documentation.
After reading the docs, I have a package structure that looks like this:
/UTM (git repo)
/utmizer
/raster
raster.py
__init__.py
convert.py
LICENSE
README.md
setup.py
utmize.py
The __init__.py looks like this because I'm not sure what I'm doing:
import utmizer.utmize as utmize
from utmizer.raster import raster
from .convert import Converter
My import statements throughout look like this:
convert.py
import fiona
import geopandas as gpd
import json
import os
import re
from raster import raster
utmize.py
from utmizer.convert import Converter
import getopt
import os
from pathlib import Path
import sys
And finally, the setup.py looks like this:
from pathlib import Path
import setuptools
from setuptools import find_packages
VERSION = '0.0.1'
HERE = Path(__file__).parent.resolve()
long_description = (HERE / 'README.md').read_text(encoding='utf-8')
setuptools.setup(
name='utmizer',
version=VERSION,
author='username',
author_email='my#email.com',
description='Automatic projection of data to appropriate UTM zone',
long_description=long_description,
long_description_content_type='text/markdown',
url='https://github.com/username/UTM',
install_requires=find_packages(),
classifiers=[
"Programming Language :: Python :: 3"
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
python_requires='>=3.8, <4',
)
When I run python setup.py bdist_wheel, it completes; but when I install it in a new conda instance and attempt to import utmizer it throws a ModuleNotFoundError.
First things:
Move setup.py LICENSE and README.md to the root directory of your project (git repository), outside of the actual top-level importable packages.
Add a package initializer __init__.py in utmizer/raster, and any other importable package and sub-package.
So you should get a directory tree like this:
/UTM (git repo)
setup.py
LICENSE
README.md
/utmizer
/raster
__init__.py
raster.py
__init__.py
convert.py
utmize.py
Seems like you mixed up packages and install_requires in your setup.py. Not sure exactly what you need in install_requires, but I guess at least geopandas and fiona.
Also no need to restrict to Python <4, because no one knows if it will be indeed incompatible. As a rule, you should only add restrictions for things you know for sure are not compatible at the time of writing (no one can guess the future).
from pathlib import Path
import setuptools
VERSION = '0.0.1'
HERE = Path(__file__).parent.resolve()
long_description = (HERE / 'README.md').read_text(encoding='utf-8')
setuptools.setup(
name='utmizer',
version=VERSION,
author='username',
author_email='my#email.com',
description='Automatic projection of data to appropriate UTM zone',
long_description=long_description,
long_description_content_type='text/markdown',
url='https://github.com/username/UTM',
packages=setuptools.find_packages(),
install_requires=[
'fiona',
'geopandas',
],
classifiers=[
"Programming Language :: Python :: 3"
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
python_requires='>=3.8',
)
You probably can leave the __init__.py files empty.
In general you should always use absolute imports starting from the top-level importable packages (or modules). In your case there is is only one top-level importable package utmizer (and no top-level importable module).
So in utmizer/convert.py:
import fiona
import geopandas as gpd
import json
import os
import re
from utmizer.raster import raster
And utmizer/utmize.py seems already correct.
Related
I am trying to compile a Python package that I recently migrated from Python2 to Python3.
When running the source code in Python3 it works as expected, so do the source and compiled versions for Python2, but when I cythonize and compile the Python3 package the resulting binaries throw this error when importing a certain module:
dynamic module does not define module export function (PyInit_NKPD)
EDIT:
When I then close the Python interpreter, open it again and import the same model it works.
Interestingly when I put the code on my local drive it imports fine, but when I put it on my server and import it from there I get the above error. And to make matters worse, sometimes it imports fine from the server as well if I just wait a while and try again:
lockjaw:controller frank$ python3.7
Python 3.7.7 (v3.7.7:d7c567b08f, Mar 10 2020, 02:56:16)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import common
added to sys.path: /opt/ohufx/transfer/NUBRIDGE_COMPILE/osx/NKPD/src
trying to import model.NukepediaDB...
NukepediaDB imported
trying to import model.NKPD...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "src/controller/common.py", line 15, in init controller.common
import model.NKPD # this does not import
ImportError: dynamic module does not define module export function (PyInit_NKPD)
>>> import common
added to sys.path: /opt/ohufx/transfer/NUBRIDGE_COMPILE/osx/NKPD/src
trying to import model.NukepediaDB...
trying to import model.NKPD...
NKPD imported
model.NKPD always seems to import fine, but model.NKPD seems to be the stumbling block, even though this may be a red herring. It almost behaves like sometimes the server connection drops out after the first import, though in that case the other incarnations (Python 2/source code) would throw errors as well (not to mention a whole lot of other things would break in my office).
I have reduced the package to three modules with nothing but import and print statements which can be found here (including the resuling C++ and binary files).
The structure is very simple:
Where common is the module I import to test by cd-ing into the src/controller folder, opening Python3.7 and importing common. Common then imports model.NukepediaDB and model.NKPD, the latter causing the trouble.
I cythonized the python files with this line:
/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 py2cpp.py
Then compiled with this:
/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py build_ext --inplace
I would love to understand why the import sometimes fails when run from the server, as I have to distribute this package to others and need to ensure it runs reliably.
Thanks,
Frank
I've installed keras-rl package on my computer, using their instructions:
git clone https://github.com/matthiasplappert/keras-rl.git
cd keras-rl
python setup.py install
So my conda environment sees this package, however when I am trying to import it in Spyder as a part of my code, i.e. import keras-rl, I get the following error:
SyntaxError: invalid syntax
with a pointer to the dash of keras-rl.
Question: How could I import the keras-rl (or any other package with the dash in the name) in Spyder?
We can install keras-rl by simply executing
pip install keras-rl
There are various functionalities from keras-rl that we can make use for running RL based algorithms in a specified environment
few examples below
from rl.agents.dqn import DQNAgent
from rl.policy import BoltzmannQPolicy
from rl.memory import SequentialMemory
This is how we can use the package.
If you look at the examples present in the github repository, you will see that the various functionalities are imported from rl. Like this:
(root) ~/condaexpts/keras-rl/examples $ grep -h import * | grep rl
from rl.agents import ContinuousDQNAgent
from rl.memory import SequentialMemory
from rl.random import OrnsteinUhlenbeckProcess
from rl.core import Processor
from rl.agents.cem import CEMAgent
from rl.memory import EpisodeParameterMemory
from rl.agents import DDPGAgent
from rl.memory import SequentialMemory
from rl.random import OrnsteinUhlenbeckProcess
from rl.agents.dqn import DQNAgent
from rl.policy import LinearAnnealedPolicy, BoltzmannQPolicy, EpsGreedyQPolicy
from rl.memory import SequentialMemory
from rl.core import Processor
from rl.callbacks import FileLogger, ModelIntervalCheckpoint
from rl.agents.dqn import DQNAgent
from rl.policy import BoltzmannQPolicy
from rl.memory import SequentialMemory
I had the same problem. After lots of examination I found the correct way.
You can import RL by writing this:
"import rl"
and then write your code like:
rl.core.Agent()
For some years alread I have had the following import in my code:
from etree.ElementTree import fromstring, parse, VERSION
Today I made a mistake while moving (in Eclipse/pyDev) a few unrelated sourcefiles to another folder. The folder was not a package and it cost me some cleans, rebuilds and del *.pyc-s to get them found again. That part has been solved, but now, the import above breaks with "unresoved import...". When I remove the etree-prefix, the imports are resolved, but at runtime I get
from ElementTree import fromstring, parse, VERSION
File "C:\Program Files\Python\EPD-7.3-2x64\Lib\xml\etree\ElementTree.py", line 127, in <module>
from . import ElementPath
ValueError: Attempted relative import in non-package
What is going wrong..?
You should not have been able to.
The import normally would be from xml.etree.ElementTree import ...; the top-level package name is xml.etree, not etree.
It looks as if you added the xml.etree package to the Python sys.path module search path. Don't do that. Remove C:\Program Files\Python\EPD-7.3-2x64\Lib\xml\etree from your sys.path (or the PYTHONPATH environment variable) and import from the correct top-level package name instead.
Background
I have a Python project with this directory structure:
py/:
db/ __init__.py run.py
py/db:
handle.py __init__.py util.py
The files are simple enough that I'm not sure I need to post them; nevertheless:
py/run.py
from db.handle import Handle
py/db/handle.py:
import util
class Handle:
def __init__(self, x):
self.x = util.addtwo(x)
py/db/util.py:
def addtwo(x):
return x + 2
If I run handle.py from within the db subdirectory, it imports util without error. However, when I run run.py, handle.py fails with an import error. I can guess that handle.py is being run in the py directory (instead of py/db), and putting a call to os.getcwd() in handle.py confirms this. I can fix this problem using sys.path like so (in run.py):
import sys
sys.path.append("db")
from db.handle import Handle
Question
When importing, from a subdirectory, a module that contains imports to other local modules in that directory, why doesn't Python check the current directory of the module making the import statement? In my example, why doesn't Python check the db first when handle.py contains import statements? Is there a PEP that describes this or is it a behavior with an obvious rationale that I missed?
I thought it might be related to PEP 328:
all import statements be absolute by default (searching sys.path only) with special syntax (leading dots) for accessing package-relative imports.
but I'm not sure.
Your import is "absolute" and module names is looked for in the PYTHONPATH, and that typically includes the current directory.
If you want to import a module from the same folder that your module is in, you use a relative import:
from . import util
or
from .util import addtwo
This (or similar) question has been asked many times before, but none of the solutions offered work in my case.
My project structure is like this :
| project_2
main.py
__init__.py
systems.py
| config
__init__.py
options.py
| database
__init__.py
database.py
entity.py
| tests
__init__.py
test_systems.py
test_options.py
test_database.py
test_entity.py
Obviously I need to import all the modules in the test modules under the tests package. I tried relative imports with the dot syntax:
from ..systems import System
from ..config import options
from ..database.entity import Entity
Returns a ValueError: Attempt relative import in non-package. I have tried that with a package structure where everything (including systems) is in its own package. It fails with the same message.
What really bothers me is that this is supposed to work: PEP 328, but it does not. I really want to avoid having to append the packages to $PYTHONPATH or to use some insane method such as loading the modules with imp from the file path.
I read that part of the problem might be that systems.py is in the main package, but that does not explain why the rest of the relative imports do not work either.
P.S. I actually recreated the example from PEP 328 just to test it and it does not work.
You get that when a python file does a relative import, but that file not loaded as a module via import in another module (but e.g. from the commandline). Given this structure:
.
├── main.py
└── test
├── __init__.py
├── a.py
└── b.py
main.py:
from test.a import A
print A
a.py:
from .b import B
A = B
if __name__ == '__main__':
print A
b.py:
B = 'b'
Now try:
python main.py
result is
b
and with
python test/a.py
you get:
Traceback (most recent call last):
File "test/a.py", line 1, in <module>
from .b import B
ValueError: Attempted relative import in non-package
What does work is:
python -m test.a
If you simply add . to your python path, if you run the script from the project_2 folder relative paths such as config.options will work. This requires an update to PYTHONPATH on every machine, unfortunately.
Tested on Python 2.7.14