I have a project that needs updating from python2 to python3. It currently uses pkg_resources, which has significant overhead (PyCon 2018 - Get Your Resources Faster with importlib.resources, by Barry Warsaw). It will be packaged into a zip file. The relevant portions of the package structure are:
+-- project/
|__+ __init__.py
|__+ main.py
|__resources/
| |__+ images/
| | |__ # program images
| |__+ logging/
| |__+ custom_logger.py
|__log/
|__+ project_log.log
Intent for Original Question
The intent for the original question was to ask "how does one get the absolute path to a subpackage from within another subpackage", but I did a poor job wording this. I thought since current working directory meant "path to where the cwd command is run from" that this could be used, but I later learned my understanding was incomplete. current working directory means (in layman's terms) "path to where the cwd command is run from on the commandline"). This is why I was having trouble. If I run my program from within the top-level package
C:\Users\usr\Destop\program>py main.py
and use pathlib.Path.cwd() in a .py file in any other subpackage (resources, logging, log), I will get the same answer:
C:\Users\usr\Destop\program
I learned this is proper behavior, but it means I cannot use relative paths to get from one subpackage to another (I need to use relative paths since I don't know where the end user will install this program on his/her machine).
Relevant 2nd Part to Question that was Not Well-Clarified
Since part of my need is to update to python3, both pkgutil and importlib.resources were investigated. Each of these require resources to be treated as packages, so to update to use either of them, I have to add __init__.py to the folders resources, logging, and log. Not a big deal thankfully as this isn't a large program.
Note, for backwards compatibility with minimal code reediting, a namespace renaming has been recommended and used by others (e.g. see (How to use importlib.resources.path(package, resource)?).:
try:
import importlib.resources as pkg_resources
except ImportError:
# Try backported to PY<37 `importlib_resources`.
import importlib_resources as pkg_resources
Neither pgkutil nor importlib.resources has a method that directly returns "this subpackage" (in the way that pathlib.Path.cwd() "directly returns" the cwd without any extra code), but I think importlib.resources has a method I can use that will require the least amount of code and still be understandable to readers. Please note the ask is to get the absolute path to "this subpackage" and nothing else. I do not want to have to use a different approach.
The methods I would use is
try:
import importlib.resources as pkg_resources
except ImportError:
# Try backported to PY<37 `importlib_resources`.
import importlib_resources as pkg_resources
with pkg_resources.path("logging", "__init__.py") as fl:
path = fl.parent.resolve()
Conclusion
Beyond this, I have no questions, because it will probably be judged as an "opinion" question. Please note the __file__ method is unacceptable as this does not work with zip packages (see PyCon 2018 - Get Your Resources Faster with importlib.resources, by Barry Warsaw). If there is anything incorrect in this approach or if it can be improved, I would like to know.
Any python file can access it's own location with file. Hence for example you could simply do
pathlib.Path(__file__).parent
to access the directory of the file that runs that code. This is independent from where you run the program from and/or where you install the program.
Related
I'm trying to use import a package for internal use, but I have been having some issues.
My directory structure looks like this:
app/
model/
file1.go
file2.go
...
main.go
When I try to build the program, I get an error that looks something like this:
/usr/local/go/src/pkg/model (from $GOROOT)
I want to be able to call the model programs in any of my other programs in the app simply using:
import "app/model"
What are my options when it comes to doing this?
You import from GOPATH level .. all of your packages should live there.
For example, assuming your application is here:
$GOPATH/src/dtrinh100/app/
..and your package you wish to import is here:
$GOPATH/src/github.com/other/package
Your import would be:
import "github.com/other/package"
You should review the literature around what the GOPATH environment variable is all about. When beginning Go, it is important you understand its purpose and initially, you should place all of your projects/packages inside of the GOPATH.
When you import a custom package, Go looks for its definition in each workspace listed in the GOPATH environment variable. Your custom package should be defined in a src subdirectory.
If you keep your code in a source repository somewhere, then you should use the root of that source repository as your base path. For instance, if you have a GitHub account at github.com/user, that should be your base path.
Note that you don't need to publish your code to a remote repository before you can build it. It's just a good habit to organize your code as if you will publish it someday. In practice you can choose any arbitrary path name, as long as it is unique to the standard library and greater Go ecosystem.
You should use github.com/user as our base path. Create a directory inside your workspace in which to keep source code:
$ mkdir -p $GOPATH/src/github.com/user
You can look at How to Write Go Code for more details.
I am having difficulty in importing a local go file into another go file.
My project structure is like something below
-samplego
--pkg
--src
---github.com
----xxxx
-----a.go
-----b.go
--bin
I am trying to import a.go inside b.go. I tried the following,
import "a"
import "github.com/xxxx/a"
None of these worked..I understand I have to meddle up with GOPATH but I couldn't get it right. Presently my GOPATH is pointing to samplego(/workspace/samplego).I get the below error
cannot find package "a" in any of:
/usr/local/go/src/pkg/a (from $GOROOT)
/workspace/samplego/src/a (from $GOPATH)
Also, how does GOPATH work when these source files are imported into another project/module? Would the local imports be an issue then? What is the best practice in this case - is it to have just one go file in module(with associated tests)?
Any number of files in a directory are a single package; symbols declared in one file are available to the others without any imports or qualifiers. All of the files do need the same package foo declaration at the top (or you'll get an error from go build).
You do need GOPATH set to the directory where your pkg, src, and bin directories reside. This is just a matter of preference, but it's common to have a single workspace for all your apps (sometimes $HOME), not one per app.
Normally a Github path would be github.com/username/reponame (not just github.com/xxxx). So if you want to have main and another package, you may end up doing something under workspace/src like
github.com/
username/
reponame/
main.go // package main, importing "github.com/username/reponame/b"
b/
b.go // package b
Note you always import with the full github.com/... path: relative imports aren't allowed in a workspace. If you get tired of typing paths, use goimports. If you were getting by with go run, it's time to switch to go build: run deals poorly with multiple-file mains and I didn't bother to test but heard (from Dave Cheney here) go run doesn't rebuild dirty dependencies.
Sounds like you've at least tried to set GOPATH to the right thing, so if you're still stuck, maybe include exactly how you set the environment variable (the command, etc.) and what command you ran and what error happened. Here are instructions on how to set it (and make the setting persistent) under Linux/UNIX and here is the Go team's advice on workspace setup. Maybe neither helps, but take a look and at least point to which part confuses you if you're confused.
No import is necessary as long as you declare both a.go and b.go to be in the same package. Then, you can use go run to recognize multiple files with:
$ go run a.go b.go
./main.go (in package main)
./a/a.go (in package a)
./a/b.go (in package a)
in this case:
main.go import "./a"
It can call the function in the a.go and b.go,that with first letter caps on.
If none of the above answers works,
Just try,
go run .
for production,
go build
This will take care of all the .go files in the folder.
I just wanted something really basic to move some files out of the main folder, like user2889485's reply, but his specific answer didnt work for me. I didnt care if they were in the same package or not.
My GOPATH workspace is c:\work\go and under that I have
/src/pg/main.go (package main)
/src/pg/dbtypes.go (pakage dbtypes)
in main.go I import "/pg/dbtypes"
As people mentioned previously, there is no need to use any imports.
A lot of people mention that using go run is possibe when you mention most files, however when having multiple .go-files in the same dir it can be cumbersome.
Therefore using go run *.go is what I usually do.
As I understand for packages in your project subfolders it's possible now to do, just need to add "." in front of module, like
. "github.com/ilyasf/deadlock-train/common"
where github.com/ilyasf/deadlock-train my main module name and common is just package from /common subfolder inside project.
go1.19.1 version here. I've just had the same issue, discovered that you must simply do: import (a "github.com/xxxx")
You can go to the correct folder and execute: $ go run *.go
As long as the code only 1 main function in all files it works perfect!
And if so, is there way that asdf can import a symbol that is calculated in runtime.
I'm trying to to specify the directory on which the project resides so the test runner can find the input files and also when I run from the repl.
Yes, system-relative-pathname and system-source-directory are your friends. At least if you execute from source.
It depends on what you mean by "execution".
If your code is executed when your file is loaded, take a look at Variable *LOAD-PATHNAME*, *LOAD-TRUENAME*.
If you need the current working directory, asdf has getcwd and chdir.
I want to run my working pydev project python code by double clicking the main module (outside of eclipse): xxx.py
The problem is that due to my imports being in different packages:
from src.apackage.amodule import obj
when xxx.py is double clicked it complains it doesn't know where the imports are (even though when I run xxx.py in pydev it magically knows what I'm importing).
A simple workaround is to remove all of the packages and move all of the modules into one directory (that obviously works but is very inconvenient)
How can I run my code in the file system without doing that work around?
This page answers my question excellently:
http://blog.habnab.it/blog/2013/07/21/python-packages-and-you/
Bottom line is always execute your code from the top, highest level, root directory (e.g. using a minimal main.py file that executes the main script of your program). Then, use absolute imports always and you never have a missing module issue since you start the program from the top directory and all imports are based off that 'home' path.
The problem you encountered is the natural behavior of most languages. A programm only knows about its working path (the path it is started in), the paths which are registered in the environment variables and at least relative paths.
The "magic" of the executable you created is therefore: It collects all scripts/modules needed, and copies/combines them next to/in the executable. The Executable then runs within the directory where all other scripts also reside and voila ...
If you are not happy with your workaround of creating an executable every time you want to run your project without PyDev there are two alternatives.
First but not the one I would suggest is registering the working path into in the environment variables.
Second and the one I think is much better: Create a link to the python executable and alter the calling string of the textfield "Target:". Append the path to your script you would like to run. Then alter the textfield "Start in:" and enter the project directory. After you did this you should be able to start your project with a simple double click.
(If you rely on external libraries which are neither on the path nor in you project you could search for appending paths temporarily to the pythonpath via the sys module.)
I hope I could help a bit.
How does one do a relative import from a parent directory?
From meme/cmd/meme:
import "../../../meme"
This gives an ambiguous error:
matt#stanley:~/gopath/src/bitbucket.org/anacrolix/meme/cmd/meme$ go get bitbucket.org/anacrolix/meme/cmd/meme
can't load package: /home/matt/gopath/src/bitbucket.org/anacrolix/meme/cmd/meme/main.go:8:2: local import "../../../meme" in non-local package
matt#stanley:~/gopath/src/bitbucket.org/anacrolix/meme/cmd/meme$ echo $GOPATH
/home/matt/gopath
How do I import locally from a parent directory?
Edit: Relative import paths are not the way to go in Go. Lack of documentation shows something about popularity of relative paths, and I don't see a reason for using them. Go's recommended code organization works pretty well. Every package should have a unique import path and be imported everywhere using that same import path.
See how a package like github.com/ha/doozerd/peer imports its neighbors. This is a common practice among Go projects and I've seen it a lot of times. Package camlistore.org/pkg/auth (also on GitHub; written by one of the main authors of Go) imports camlistore.org/pkg/netutil by full path.
Even if you are having both commands and libraries in the same project this approach works. In your original questions you wisely asked for best practices. I did my best in explaining best practices on this matter.
Import paths can't be relative in Go. I recommend reading How to Write Go Code, the essential reading on organizing Go projects. Here's a short overview:
Make a directory like ~/go for your Go development. Then say:
$ export GOPATH=~/go
$ mkdir $GOPATH/{src,bin,pkg}
$GOPATH/src holds source code for all your Go packages, even the ones your download with go get. bin and pkg keep output of compilations. Packages with package name main are commands and yield to executable binaries which go to $GOPATH/bin. Other packages are libraries and their compiled object files are put in $GOPATH/pkg.
Now if you put your code in $GOPATH/src/matt/meme, you can import it by import "matt/meme". It's recommended to use a prefix for your package names and leave short package names for standard libraries. That's why I used $GOPATH/src/matt/meme instead of $GOPATH/src/meme.
Organize your code around this idea.
Thanks for adding to your question. First, an answer, then some explanation. I built your code by,
go get, just as you had it. (I ignored the error messages.)
setting the import line in main.go back to "../../../meme", as you wanted to do.
(commenting out a little bit of code containing an unused variable.)
then in the meme/cmd/meme directory, either go run main.go or go build main.go worked.
I was wrong in my comment earlier when I said go install works; I should have said go build.
The key however is that go build alone does not work; you must type go build main.go. This is because the go command does not allow "local imports in non-local packages." You are right that spec is of little help here. It weasels out saying, "The interpretation of the ImportPath is implementation-dependent." The current implementation behavior was set with CL 5787055, which was subsequently debated at length on Go-nuts.
"Local" means indicated with a file system relative path. Obviously a relative path starting with .. is local, so the trick is just getting the go command to treat main as a local package as well. It apparently doesn't do this when you type go build, but does when you type go build main.go.
Relarive imports are supported when manually using the compiler, linker, ... directly. The 'go' (build) tool doesn't support the same (somehow comparable to eg Java).
This may not answer the original question, but I was trying to do the above when I didn't really need to, all I needed to do was update go.mod temporarily with a replace :
module github.com/pselle/foo
replace github.com/pselle/bar => /Users/pselle/Projects/bar
require (
github.com/pselle/bar v1.0.0
)
reference:
https://thewebivore.com/using-replace-in-go-mod-to-point-to-your-local-module/