Importing json resources inside .pex (Python Executable (format by Twitter)) - python-packaging

I'm using a Twitter engineered build tool pants to manage many projects inside my monorepo. It outputs .pex files when I complete a build, this is a binary that packages the bare minimum dependencies I need for each project and makes them a "binary" (actually an archive that's decompressed at runtime), my issue is a utility that my code has used for a long time fails to detect some .json files(now that I'm using pants) I have stored under my environments library. all my other code seems to run fine. I'm pretty sure it has to do with my config, perhaps I'm not storing the resources properly so my code can find it, though when I use unzip my_app.pex the resources I desire are in the package and located in the proper location(dir). Here is the method my utility uses to load the json resources:
if test_env:
file_name = "test_env.json"
elif os.environ["ENVIRONMENT_TYPE"] == "PROD":
file_name = "prod_env.json"
else:
file_name = "dev_env.json"
try:
json_file = importlib.resources.read_text("my_apps.environments", file_name)
except FileNotFoundError:
logger.error(f"my_apps.environments->{file_name} was not found")
exit()
config = json.loads(json_file)
here is the the BUILD file I use for these resource currently:
python_library(
dependencies=[
":dev_env",
":prod_env",
":test_env"
]
)
resources(
name="dev_env",
sources=["dev_env.json"]
)
resources(
name="prod_env",
sources=["prod_env.json"]
)
resources(
name="test_env",
sources=["test_env.json"]
)
and here is the BUILD file for the utility that calls these resources of which the python code above is what you saw:
python_library(
name="environment_handler",
sources=["environment_handler.py"],
dependencies=[
"my_apps/environments:dev_env",
"my_apps/environments:prod_env",
"my_apps/environments:test_env"
]
)
I always get an FileNotFoundError exception and I'm confused because the files are available to the runtime, what's causing these files to not be accessible? and is there a different format I need to set up the JSON resources as?
Also for context here is the decompressed .pex file(actually just the source-code dir):
├── apps
│   ├── __init__.py
│   └── services
│   ├── charts
│   │   ├── crud
│   │   │   ├── __init__.py
│   │   │   └── patch.py
│   │   ├── __init__.py
│   │   └── main.py
│   └── __init__.py
├── environments
│   ├── dev_env.json
│   ├── prod_env.json
│   └── test_env.json
├── __init__.py
├── models
│   ├── charts
│   │   ├── base.py
│   │   └── __init__.py
│   └── __init__.py
└── utils
├── api_queries
│   ├── common
│   │   ├── connections.py
│   │   └── __init__.py
│   └── __init__.py
├── calculations
│   ├── common
│   │   ├── __init__.py
│   │   └── merged_user_management.py
│   └── __init__.py
├── environment_handler.py
├── __init__.py
├── json_response_toolset.py
└── security_toolset.py

I figured it out: I changed the way I access the files within the library and it works perfectly before and after the build to .pex format. I used:
import pkgutil
#json_file = importlib.resources.read_text("my_apps.environments", file_name)
json_file = pkgutil.get_data("my_apps.environments", file_name).decode("utf-8")

Related

Using GitHub Actions in a Single Repository with Multiple Projects

I am fairly competent in using GitHub actions to build a variety of languages, orchestrate deployments, and I've even done cross-repository actions using web-hooks, so I'd say that I'm pretty familiar with working with them.
I often find myself doing a lot of scratch projects to test out an API or making a demo, and these don't usually merit their own repositories, but I'd like to save them for posterity, rather than just making Gists out of them, Gists being largely impossible to search. I'd like to create a scratch repository, with folders per language, like:
.
└── scratch
├── go
│   ├── dancing
│   │   ├── LICENSE-APACHE
│   │   ├── LICENSE-MIT
│   │   ├── main.go
│   │   └── README.md
│   ├── gogettur
│   │   ├── LICENSE-APACHE
│   │   ├── LICENSE-MIT
│   │   ├── main.go
│   │   └── README.md
│   └── streeper
│   ├── LICENSE-APACHE
│   ├── LICENSE-MIT
│   ├── main.go
│   └── README.md
├── node
│   └── javawhat
│   ├── index.js
│   ├── LICENSE-APACHE
│   ├── LICENSE-MIT
│   └── README.md
└── rust
├── logvalanche
│   ├── Cargo.toml
│   ├── LICENSE-APACHE
│   ├── LICENSE-MIT
│   ├── README.md
│   └── src
├── streamini
│   ├── Cargo.toml
│   ├── LICENSE-APACHE
│   ├── LICENSE-MIT
│   ├── README.md
│   └── src
└── zcini
├── Cargo.toml
├── LICENSE-APACHE
├── LICENSE-MIT
├── README.md
└── src
I'd like to generalize GitHub actions per language, for Go, use go test ./... and go build, for Rust cargo test and cargo build, etc.
I know that what I could do is have a workflow for each created project, but this would be tedious, I'd end up copying and pasting most of the time, and every build would run on every change in the entire repository, and I don't want to be building node/javawhat if only rust/zcini has changed.
Therefore I have a few questions:
Is it possible to have a workflow only run when certain files have changed, rather than running everything every single time?
Is there a way to generalize my workflows so that every dir in rust/ uses the same generic workflow, or will I need one workflow per project in the repository?

Install, with CPAN, Perl modules to specific directory when several appear in use [duplicate]

This question already has answers here:
How can I install a CPAN module into a local directory?
(5 answers)
Closed 4 years ago.
Running the following command returns several paths:
perl -e 'print join("\n",#INC,"")'
Every path has modules installed within. I would want to install modules, as root, into the following directory:
/usr/local/share/perl5
What commands would I run to find where cpan,as root, currently installs modules? How would I alter it if it is not the path shown above?
Here's how I have configured cpan to put all new modules in a specific directory:
o conf makepl_arg 'PREFIX=/usr/local/share/perl5 INSTALLMAN3DIR=/usr/local/share/perl5/man/man3'
o conf mbuild_arg '--install_base /usr/local/share/perl5'
o conf mbuild_install_arg '--install_base /usr/local/share/perl5'
o conf mbuildpl_arg '--install-base /usr/local/share/perl5'
[o conf commit]
The first line addresses modules that use ExtUtils::MakeMaker and the next three lines are for modules that use Module::Build.
You can also do this quite easily via App::cpm.
$ cpm install -L my-random-folder Open::This
DONE install Path-Tiny-0.108 (using prebuilt)
DONE install Try-Tiny-0.30 (using prebuilt)
DONE install Module-Build-0.4224 (using prebuilt)
DONE install Module-Runtime-0.016 (using prebuilt)
DONE install Open-This-0.000008 (using prebuilt)
5 distributions installed.
$ tree my-random-folder
my-random-folder
├── bin
│   ├── config_data
│   └── ot
└── lib
└── perl5
├── 5.26.1
│   └── darwin-2level
├── Module
│   ├── Build
│   │   ├── API.pod
│   │   ├── Authoring.pod
│   │   ├── Base.pm
│   │   ├── Bundling.pod
│   │   ├── Compat.pm
│   │   ├── Config.pm
│   │   ├── ConfigData.pm
│   │   ├── Cookbook.pm
│   │   ├── Dumper.pm
│   │   ├── Notes.pm
│   │   ├── PPMMaker.pm
│   │   ├── Platform
│   │   │   ├── Default.pm
│   │   │   ├── MacOS.pm
│   │   │   ├── Unix.pm
│   │   │   ├── VMS.pm
│   │   │   ├── VOS.pm
│   │   │   ├── Windows.pm
│   │   │   ├── aix.pm
│   │   │   ├── cygwin.pm
│   │   │   ├── darwin.pm
│   │   │   └── os2.pm
│   │   └── PodParser.pm
│   ├── Build.pm
│   └── Runtime.pm
├── Open
│   └── This.pm
├── Path
│   └── Tiny.pm
├── Try
│   └── Tiny.pm
└── darwin-2level
└── auto
├── Module
│   ├── Build
│   └── Runtime
├── Open
│   └── This
├── Path
│   └── Tiny
└── Try
└── Tiny

Game_ cannot be resolved to a variable in arquillian persistance example

I was asking on #arquillian Freenode IRC channel about question
Arquillian JPA tutorial: Could not create new instance of class org.jboss.arquillian.test.impl.EventTestRunnerAdaptor
when a user told me that he successfully runned
https://github.com/arquillian/arquillian-examples/tree/master/arquillian-persistence-tutorial
so I did
git clone https://github.com/arquillian/arquillian-examples.git
then in Eclipse, I clicked File->Import->Existing Maven Projects and selecte the subdirectory
arquillian-examples/arquillian-persistence-tutorial
Once Eclipse finished importing the project, I obtain in file
/src/test/java/org/arquillian/example/GamePersistenceTest.java
line 146, the error message
Game_ cannot be resolved to a variable
What can I do? In my attempt at the link at beginning of the page, this did not happpen.
I did not change any file downloaded from git repo.
Folder tree
.
├── pom.xml
├── src
│   ├── main
│   │   ├── java
│   │   │   └── org
│   │   │   └── arquillian
│   │   │   └── example
│   │   │   └── Game.java
│   │   └── resources
│   │   └── META-INF
│   │   └── persistence.xml
│   └── test
│   ├── java
│   │   └── org
│   │   └── arquillian
│   │   └── example
│   │   └── GamePersistenceTest.java
│   ├── resources
│   │   ├── arquillian.launch
│   │   ├── arquillian.xml
│   │   └── jbossas-ds.xml
│   ├── resources-glassfish-embedded
│   │   ├── glassfish-resources.xml
│   │   ├── logging.properties
│   │   └── test-persistence.xml
│   ├── resources-glassfish-remote
│   │   └── test-persistence.xml
│   └── resources-jbossas-managed
│   └── test-persistence.xml
└── target
├── classes
│   ├── META-INF
│   │   ├── MANIFEST.MF
│   │   ├── maven
│   │   │   └── org.arquillian.example
│   │   │   └── arquillian-persistence-tutorial
│   │   │   ├── pom.properties
│   │   │   └── pom.xml
│   │   └── persistence.xml
│   └── org
│   └── arquillian
│   └── example
│   └── Game.class
└── test-classes
├── arquillian.launch
├── arquillian.xml
├── glassfish-resources.xml
├── jbossas-ds.xml
├── logging.properties
├── org
│   └── arquillian
│   └── example
│   └── GamePersistenceTest.class
└── test-persistence.xml
30 directories, 24 files
The _Game class in that example is generated by the Hibernate JPA 2 metamodel generator (hibernate-jpamodelgen) that is defined as a dependency in the project POM. You'll now need to generate the metamodel by employing one the options outlined in the metamodel generator reference guide.
You could use Eclipse itself by configuring the annotation processing phase. Or you could modify the POM to use the maven-compiler-plugin configuration specified in the guide, to run as part of your build.

Which folders are unnecessary while deploying a play! app?

I have finally made a production worthy app with play, and use the play war -o foo command to create an exploded war. However, its size goes on to 30 MB and there are a lot of folders included. I develop in eclipse, so there are some eclipse folders too. Below is a list of the folders I have. Can anyone help me out with the unnecessary folders? I have zeroed in on the app, conf, public and lib folders. Is my assumption correct?
├── app
│   ├── controllers
│   ├── jobs
│   │   ├── daily
│   │   └── monthly
│   ├── models
│   │   ├── encrypt
│   │   ├── file
│   │   ├── login
│   │   └── mail
│   ├── playground
│   └── views
│   ├── Application
│   └── errors
├── conf
├── eclipse
│   └── classes
│   ├── controllers
│   ├── jobs
│   │   ├── daily
│   │   └── monthly
│   ├── models
│   │   ├── encrypt
│   │   ├── file
│   │   ├── login
│   │   └── mail
│   └── playground
├── lib
├── logs
├── public
│   ├── bootstrap
│   │   ├── css
│   │   ├── img
│   │   ├── js
│   │   │   └── tests
│   │   │   ├── unit
│   │   │   └── vendor
│   │   └── less
│   ├── images
│   ├── javascripts
│   └── stylesheets
├── test
└── tmp
├── bytecode
│   └── DEV
└── classes
├── controllers
├── helpers
├── jobs
│   ├── daily
│   └── monthly
├── models
│   ├── encrypt
│   ├── file
│   ├── login
│   └── mail
└── playground
You need to delete your tmp folder as well, try to use play clean on your project.
All folders in app, conf, lib and public should be kept in your war. Maybe should add precompiled

spring-security-facebook, s2-init-facebook not generating the Default Dao

I have installed spring-security-facebook to a test grail application. This was done after installing spring-security-core and running s2-quickstart.
I am just a novice and was trying to integrate the facebook login button into my test app. But, the problem is that when I run s2-init-facebook the plugin is not generating the default Dao as it was told in the documentation in http://grails.org/plugin/spring-security-facebook
Use your own authentication dao
Plugin generates an Dao, after calling s2-init-facebook#, and put it
into your #conf/spring/resources.groovy and #Config.groovy#. Actually
it's an 'dumb' dao implementations, you have to rewrite it to follow
your data structures:
The shortened tree output of my app directory is given below:
.
├── application.properties
├── grails-app
│   ├── conf
│   │   ├── ApplicationResources.groovy
│   │   ├── BootStrap.groovy
│   │   ├── BuildConfig.groovy
│   │   ├── Config.groovy
│   │   ├── DataSource.groovy
│   │   ├── hibernate
│   │   ├── spring
│   │   │   └── resources.groovy
│   │   └── UrlMappings.groovy
│   ├── controllers
│   │   ├── LoginController.groovy
│   │   └── LogoutController.groovy
│   ├── domain
│   │   └── com
│   │   └── fbtest
│   │   └── webapp
│   │   └── auth
│   │   ├── FacebookUser.groovy
│   │   ├── SecRole.groovy
│   │   ├── SecUser.groovy
│   │   └── SecUserSecRole.groovy
│   ├── i18n
│   │   ├── ...
│   ├── services
│   ├── taglib
│   ├── utils
│   └── views
│   ├── error.gsp
│   ├── index.gsp
│   ├── layouts
│   │   └── main.gsp
│   └── login
│   ├── auth.gsp
│   └── denied.gsp
├── lib
├── scripts
├── src
│   ├── groovy
│   └── java
├── test
│   ├── integration
│   └── unit
└── web-app
├── ...
resource.groovy file is also empty.
// Place your Spring DSL code here
beans = {
}
Please, help me in understanding if I am doing any thing wrong.
I am using Grails version: 2.0.4 and spring-security-facebook's version is 0.8.
Thanks in advance.
Oh, it should be removed from documentation. For last versions it's not necessary to have own DAO, now it have default very flexible dao. And now it's not generated during install. Try to run your app, it should works, if everything is configured correctly.
You can still use own DAO, if you have implemented it by yourself. But for versions since 0.7 nearly everything can be extending with optional FacebookAuthService, that you can implement if you wish (it's described in documentation as well).