I'm getting to the point with a project where I need a proper directory structure. I'm trying to arrange this and getting ImportErrors when using my cython extension classes.
The directory structure looks like:
.
├── __init__.py
├── Makefile
├── README.rst
├── setup.py
├── src
│ ├── foo.pxd
│ ├── foo.pyx
│ ├── __init__.py
│ └── metafoo.pyx
└── test
├── test_foo.py
└── test_metafoo.py
The contents of all files can be found (in commit e635617 at time of writing) of this github repo.
My setup.py looks like the following:
from setuptools import setup, Extension, Command
from Cython.Build import cythonize
SRC_DIR = "src"
PACKAGES = [SRC_DIR]
ext_foo = Extension(SRC_DIR + ".foo",
[SRC_DIR + "/foo.pyx"]
)
ext_meta = Extension(SRC_DIR + ".metafoo",
[SRC_DIR + "/metafoo.pyx"]
)
EXTENSIONS = cythonize([ext_foo, ext_meta])
setup(
name = 'minimalcriminal',
packages=PACKAGES,
ext_modules=EXTENSIONS
)
The complexity seems to lie in that extension classes in metafoo.pyx use extension classes from foo.pyx.
After building with python setup.py build_ext --inplace, the test_foo.py program runs ok:
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
import src.foo as foo
somefoo = foo.Foo(2)
somefoo.special_print()
When run from both the cyproj/test and cyproj directories:
/cyproj$ python test/test_foo.py
The value of somefield is: 2
and
/cyproj/test$ python test_foo.py
The value of somefield is: 2
But the test_metafoo.py crashes when run in the cyproj/test directory:
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
import src.foo as foo
import src.metafoo as metafoo
lotsafoo = [foo.Foo(i) for i in range(4)]
mf = metafoo.MetaFoo(lotsafoo)
mf.special_print()
With the message:
ubuntu#ubuntu-UX21E:/projects/cyproj/test$ python test_metafoo.py
Traceback (most recent call last):
File "test_metafoo.py", line 6, in <module>
import src.metafoo as metafoo
File "cyproj/src/foo.pxd", line 6, in init cyproj.src.metafoo (src/metafoo.c:1154)
ImportError: No module named cyproj.src.foo
But runs properly from the parent cyproj directory:
/cyproj$ python test/test_metafoo.py
The value of somefield is: 0
The value of somefield is: 1
The value of somefield is: 2
The value of somefield is: 3
I don't really get what's driving the different behaviour of these errors. If I can't use import src.foo in test_metafoo.py why does it work in test_foo.py?
Similarly if I open up an interactive session in the parent directory and try to import all:
In [1]: from src import *
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-7b8bc2c1dfb9> in <module>()
----> 1 from src import *
/projects/cyproj/cyproj/src/foo.pxd in init cyproj.src.metafoo (src/metafoo.c:1154)()
ImportError: No module named cyproj.src.foo
When src/__init__.py looks like:
__all__ = ["foo", "metafoo"]
Which I thought would allow importing all...
I was able to compile and test your package after removing the __init__.py file from the project root directory and changing test_foo.py and test_metafoo.py.
sys.path.append(os.path.abspath("."))
sys.path.append(os.path.abspath("../"))
Related
I am trying to run a pytest test for filea.py using the following directory structure
test_filea.py
from filea import *
def test_one_p_one():
r = one_p_one()
assert r == 2
filea.py
def one_p_one():
return 1 + 1
When i have to following directory structure every thing works fine.
├── filea.py
├── test_filea.py
but when i move my tests into a sub directory like this
├── filea.py
└── tests
└── test_filea.py
i get the error:
test_filea.py:1: in <module>
from filea import *
E ModuleNotFoundError: No module named 'filea'
My editor seems to indicate the import in the file in the sub directory is ok.. (no read squiggly lines)
but when i run this using "pytest"
i get the error indicated above.
As per pytest documentation about test discovery, try like this:
add an empty __init__.py file in testsdirectory;
make sure that, when you run pytest ., the parent directory of filea.py and tests is the current working directory.
It depends where you run the tests from, and how you invoke pytest. Calling pytest tests is different than calling python -m pytest tests, the later adds the current working directory into the sys.path, which makes filea module importable.
I have trouble creating a package with setuptools. I have a repository which I'm cleaning up to make it a package. The directory structure looks something like this
my-proj
├── setup.py
├── MANIFEST.in
├── MakeFile
├── README.rst
├── setup.py
└── myproj
├── __init__.py
├── my_code.py
├── templates
│ ├── template1.yaml
│ ├── template2.yaml
Initial version of "my_code.py" had code snippet which would directly reference the files withing templates folder to do some processing. If I package this using setup tools, I provide the following information in these files:
MANIFEST.in:
include README.rst
include requirements.txt
include LICENSE.txt
recursive-include myproj/templates *
setup.py:
setup(
name='myproj',
package_dir={'testbed_init': 'testbed_init'},
package_data={'templates': ['templates/*'], 'configs': ['configs/*']},
include_package_data=True,
)
My question is as follows. In "my_Code.py" I used to reference templates directly without any problem as I would run script from the myproj folder. If I package this, how can I make sure, I include the templates as part of package and when script runs, I need to open the templates relative to where the package is installed.
Code snippet from my_code.py:
if _type == "a":
temp_file = f"templates/template1.yaml"
else:
temp_file = f"templates/template2.yaml"
build_config(deploy_esx_file, output_file, data)
Code snippet of what happens in build_config:
def build_config(template_file, output_file, inputs):
templateLoader = jinja2.FileSystemLoader(searchpath="./")
templateEnv = jinja2.Environment(loader=templateLoader)
template = templateEnv.get_template(template_file)
outputText = template.render(inputs)
with open(output_file, 'w') as h:
h.write(outputText)
What am I doing wrong here???
My structure :-
├── tst
│ ├── setup.py
│ └── tst
│ ├── __init__.py
│ ├── mre.py
│ └── start.py
contents of start.py
from mre import mre
def proc1():
mre.more()
return ('ran proc1')
if __name__ == "__main__":
print('test')
print(proc1())
contents of mre.py
class mre(object):
def more():
print('this is some more')
contents of setup.py
from setuptools import setup
setup(name='tst',
version='0.1',
description='just a test',
author='Mr Test',
author_email='test#example.com',
entry_points={'console_scripts': ['tst=tst.start:proc1']},
license='MIT',
packages=['tst'],
zip_safe=False)
nothing in __init__.py
When I run this from the command line all is fine, runs as expected.
However when I package this up using PIP and run using tst I get:-
Traceback (most recent call last):
File "/home/simon/.local/bin/tst", line 5, in <module>
from tst.start import proc1
File "/home/simon/.local/lib/python3.8/site-packages/tst/start.py", line 1, in <module>
from mre import mre
ModuleNotFoundError: No module named 'mre'
I've read numerous posts and I just can't seem to figure this out, if I go into the installed code and change the line
from mre import mre
to
from tst.mre import mre
then it works, but then that doesn't work when running it from the dir for development purposes... I'm obviously missing something obvious :) is it a path issue or am I missing a command in the setup.py?
If someone could point me in the right direction?
edit: do I need to do something different while developing a module thats going to be packaged, perhaps call the code some different way?
cheers
From my point of view, the absolute import from tst.mre import mre is the right thing. You could eventually use from .mre import mre, but the absolute import is safer.
For development purposes:
Use pip's editable mode:
path/to/pythonX.Y -m pip install --editable .
Similar to setuptools develop mode which is slowly going towards deprecation path/to/pythonX.Y setup.py develop.
And run the console script, or the executable module:
tst
path/to/pythonX.Y -m tst.start
Without installation, it is often sill possible to run the executable module:
path/to/pythonX.Y -m tst.start.
Suppose I have the following directory structure:
src/
└── python/
└── generated/
├── __init__.py
├── a.py
└── lib/
├── __init__.py
└── b.py
What does my setup.py need to look like in order to create a dist with a directory layout like:
src/
└── python/
├── __init__.py
├── a.py
└── lib/
├── __init__.py
└── b.py
The goal is to simply eliminate the generated folder. I've tried endless variations with package_dir and can't get anything produced other than the original directory structure.
Your setup.py should be placed in your src directory and should look like this:
#!/usr/bin/env python3
import setuptools
setuptools.setup(
name='Thing',
version='1.2.3',
packages=[
'python',
'python.lib',
],
package_dir={
'python': 'python/generated',
},
)
Note the package_dir setting. It instructs setuptools to get the code for the python package from the directory python/generated. In the built distributions you will then find the right directory structure.
First, here is my solution:
#!/usr/bin/env python
import os, shutil
from setuptools import setup
from setuptools.command.build_py import build_py
class BuildPyCommand(build_py):
"""Custom build command."""
def run(self):
shutil.rmtree('src.tmp', ignore_errors=True)
os.mkdir('src.tmp')
shutil.copytree('src/python/generated', 'src.tmp/python')
build_py.run(self)
setup(cmdclass={ 'build_py': BuildPyCommand },
name='Blabla',
version='1.0',
description='best desc ever',
author='Me',
packages=['python', 'python.lib'],
package_dir={'': 'src.tmp'},
setup_requires=['wheel']
)
And you can generate your distribution with:
python setup.py build bdist_wheel
The idea is perform a two steps build:
I generate a valid source structure
I build this temporary structure
And I deliver it in a wheel because it doesn't require future users to understand my trick. If you give it a try with a source distribution, you will notice that you need to publish the generated files as data (not difficult, but troublesome, and, I guess you will want to hide your tricks from your users).
But, I think that there is a design flaw in your process. The file src/python/generated/__init__.py, assumed to be a module <something>.generated eventually becomes your <something>.python, which is troublesome. It would be much simpler and more robust to generate a valid Python structure: src/generated/python/__init__.py. The setup.py would become trivial and your generator wouldn't be more complex.
This (or similar) question has been asked many times before, but none of the solutions offered work in my case.
My project structure is like this :
| project_2
main.py
__init__.py
systems.py
| config
__init__.py
options.py
| database
__init__.py
database.py
entity.py
| tests
__init__.py
test_systems.py
test_options.py
test_database.py
test_entity.py
Obviously I need to import all the modules in the test modules under the tests package. I tried relative imports with the dot syntax:
from ..systems import System
from ..config import options
from ..database.entity import Entity
Returns a ValueError: Attempt relative import in non-package. I have tried that with a package structure where everything (including systems) is in its own package. It fails with the same message.
What really bothers me is that this is supposed to work: PEP 328, but it does not. I really want to avoid having to append the packages to $PYTHONPATH or to use some insane method such as loading the modules with imp from the file path.
I read that part of the problem might be that systems.py is in the main package, but that does not explain why the rest of the relative imports do not work either.
P.S. I actually recreated the example from PEP 328 just to test it and it does not work.
You get that when a python file does a relative import, but that file not loaded as a module via import in another module (but e.g. from the commandline). Given this structure:
.
├── main.py
└── test
├── __init__.py
├── a.py
└── b.py
main.py:
from test.a import A
print A
a.py:
from .b import B
A = B
if __name__ == '__main__':
print A
b.py:
B = 'b'
Now try:
python main.py
result is
b
and with
python test/a.py
you get:
Traceback (most recent call last):
File "test/a.py", line 1, in <module>
from .b import B
ValueError: Attempted relative import in non-package
What does work is:
python -m test.a
If you simply add . to your python path, if you run the script from the project_2 folder relative paths such as config.options will work. This requires an update to PYTHONPATH on every machine, unfortunately.
Tested on Python 2.7.14