OS: Windows7, Jython2.7.0FR ("Final Release")
Trying to use Gord Thompson's solution with Jython: Manipulating an Access database from Java without ODBC
I can get the Ucanaccess module to work in Eclipse, but not when I try running from the CLI.
# -*- coding: utf-8 -*-
import java, sys
jars_dir = 'D:\\sysadmin\\Resources\\Java jar files\\'
sys.path.append( jars_dir + 'commons-lang-2.6.jar' )
sys.path.append( jars_dir + 'commons-logging-1.2.jar' )
sys.path.append( jars_dir + 'hsqldb.jar' )
sys.path.append( jars_dir + 'jackcess-2.1.2.jar' )
sys.path.append( jars_dir + 'ucanaccess-3.0.2.jar' )
import net.ucanaccess.jdbc.UcanaccessDriver
import net.ucanaccess
print( '# appear to have imported UcanaccessDriver' )
conn = java.sql.DriverManager.getConnection( 'jdbc:ucanaccess://D:/TESTING.mdb' )
print( '# conn OK...')
This prints out "# conn OK" when I run this in Eclipse.
When I run it from the CLI it prints out "# appear to have imported..." but then an error is thrown:
(NB this output was copied to Eclipse: it is indeed a CLI run)
Anyone got any idea why I might be getting "No suitable driver..."? NB I tried various permutations with backward slashes instead of forward ones in getConnection... to no avail.
By the way, in case this is relevant, this is an extract from the .bat file I'm using to run the thing:
cd "%SOFTWARE_PROJECTS%\workspace\Jython scratchpad\src\jython_scratchpad_root"
REM this is probably irrelevant and doesn't seem to work with Jython2.7.0FR. The jars are being loaded by sys.path.append, obviously
set CLASSPATH=.;"%SYSADMIN%\resources\java jar files/*"
D:\apps\jython2.7.0\bin\jython loading_test.py
It appears that sys.path has more to do with the PATH environment variable than the CLASSPATH environment variable. The following two methods have been tested and do work with Jython 2.7.0 to run a script called "jyTest.py" from the Windows command line
Method 1: Set the CLASSPATH from a batch file (or shell script) before launching Jython
SET CLASSPATH=C:/Users/Public/Downloads/UCanAccess/ucanaccess-3.0.2.jar;C:/Users/Public/Downloads/UCanAccess/lib/commons-lang-2.6.jar;C:/Users/Public/Downloads/UCanAccess/lib/commons-logging-1.1.1.jar;C:/Users/Public/Downloads/UCanAccess/lib/hsqldb.jar;C:/Users/Public/Downloads/UCanAccess/lib/jackcess-2.1.2.jar;.
c:\jython2.7.0\bin\jython jyTest.py
(For Linux et. al. that would be export CLASSPATH ..., and colon separators instead of semicolons.)
Method 2: Use -J-cp to set the CLASSPATH when invoking Jython
c:\jython2.7.0\bin\jython -J-cp C:/Users/Public/Downloads/UCanAccess/ucanaccess-3.0.2.jar;C:/Users/Public/Downloads/UCanAccess/lib/commons-lang-2.6.jar;C:/Users/Public/Downloads/UCanAccess/lib/commons-logging-1.1.1.jar;C:/Users/Public/Downloads/UCanAccess/lib/hsqldb.jar;C:/Users/Public/Downloads/UCanAccess/lib/jackcess-2.1.2.jar;. jyTest.py
(Again, Linux and friends would need the -cp entries to be separated by colons instead of semicolons.)
Related
I want to execute a Python script from Matlab (on a Windows 7 machine). The libraries necessary are installed in an Anaconda virtual environment. When running the script from command line, it runs flawlessly.
When calling the script from Matlab as follows:
[status, commandOut] = system('C:/Users/user/AppData/Local/Continuum/anaconda3/envs/tf/python.exe test.py');
or with shell commands, I get an Import Error:
commandOut =
'Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\core\__init__.py", line 16, in <module>
from . import multiarray
ImportError: DLL load failed: The specified path is invalid.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 2, in <module>
import numpy as np
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\__init__.py", line 142, in <module>
from . import add_newdocs
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\lib\__init__.py", line 8, in <module>
from .type_check import *
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\lib\type_check.py", line 11, in <module>
import numpy.core.numeric as _nx
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\lib\site-packages\numpy\core\__init__.py", line 26, in <module>
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.
Original error was: DLL load failed: The specified path is invalid.
I already changed the default Matlab Python version to the Anaconda env, but no change:
version: '3.5'
executable: 'C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\python.exe'
library: 'C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf\python35.dll'
home: 'C:\Users\user\AppData\Local\Continuum\anaconda3\envs\tf'
isloaded: 1
Just running my test script without importing numpy works. Reloading numpy (py.importlib.import_module('numpy');) didn't work but threw the same error as before.
Does anyone have an idea how to fix this?
So after corresponding with Matlab support I found out that Matlab depends on the path environment (paths which are deliberately not set when using a virtual environment) and therefore numpy fails to find the necessary paths when called from within Matlab (even if the call contains the path to the virtual environment).
The solution is either to call Matlab from within the virtual environment (via command line) or add the missing paths manually in the path environment.
Maybe this information can help someone else.
First Method
You can change the python interpreter with:
pyversion("/home/nibalysc/Programs/anaconda3/bin/python");
And check it with:
pyversion();
You could also do this in a
startup.m
file in your project folder and every time you start MATLAB from this folder the python interpreter will be changed automatically.
Now you can try to use:
py.importlib.import_module('numpy');
Read up the documentation on how to use the integrated python in MATLAB:
Call user defined custom module
Call modified python module
Alternative Method
Alternative method would be to create a
matlab_shell.sh
file with following content, this is basically the appended code from .bashrc when anaconda is installed and asks you if the installer should modify the .bashrc file:
#!/bin/bash
__conda_setup="$(CONDA_REPORT_ERRORS=false '$HOME/path/to/anaconda3/bin/conda' shell.bash hook 2> /dev/null)"
if [ $? -eq 0 ]; then
\eval "$__conda_setup"
else
if [ -f "$HOME/path/to/anaconda3/etc/profile.d/conda.sh" ]; then
CONDA_CHANGEPS1=false conda activate base
else
\export PATH="$HOME/path/to/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda init <<<
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('$HOME/path/to/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "$HOME/path/to/anaconda3/etc/profile.d/conda.sh" ]; then
. "$HOME/path/to/anaconda3/etc/profile.d/conda.sh"
else
export PATH="$HOME/path/to/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
conda activate base
eval $2
Then you need to set the MATLAB_SHELL environment variable either before running MATLAB or in MATLAB itself. The best thing in my opinion would be to do it also in the startup.m file like that:
setenv("MATLAB_SHELL", "/path/to/matlab_shell.sh");
Afterwards you can use the system(...) function to run conda python with all your modules installed like that...
String notation:
system("python -c ""python code goes here"");
Char notation:
system('python -c "python code goes here"');
Hope this helps!
Firstly, if you execute your Python script like a regular system command ([status, commandOut] = system('...python.exe test.py'))
the pyversion (and pyenv, since R2019b) got no effect at all. It only matters if you utilize the py. integration, as in the code below (and, in most cases, this is a way better approach).
Currently (I use R2019b update 5) there's a number of pitfalls, that might cause issues similar to yours. I'd recommend to start from the following:
Create a new clean conda environment:
conda create -n test_py36 python=3.6 numpy
Create the following dummy demo1.py:
def dummy_py_method(x):
return x+1
Create the following run_py_code.m:
function run_py_code()
% explicit module import sometimes show more detailed error messages
py.importlib.import_module('numpy');
% to reloads if there would be any changes:
pymodule = py.importlib.import_module('demo1');
py.importlib.reload(pymodule);
% passing data back and forth
x = rand([3 3]);
x_np = py.numpy.array(x);
y_np=pymodule.dummy_py_method(x_np);
y = double(y_np);
disp(y-x);
Create the following before_first_run.m:
setenv('PYTHONUNBUFFERED','1');
setenv('path',['C:\Users\username\Anaconda3\envs\test_py36\Library\bin;'...
getenv('path')]);
pe=pyenv('Version','C:\users\username\Anaconda3\envs\test_py36\pythonw.exe',...
'ExecutionMode','InProcess'...
);
% add "demo1.py" to path
py_file_path = 'W:\tests\Matlab\python_demos\call_pycode\pycode';
if count(py.sys.path,py_file_path) == 0
insert(py.sys.path,int32(0),py_file_path);
end
Run the before_first_run.m first and run the run_py_code.m next.
Notes:
As already mentioned in this answer, one key point is to add the folder, containing the necessary dll files to the %PATH%, before starting python. This could be achieved with setenv from withing Matlab. Usually, the Library\bin is what should be added.
It might be a good idea to try clean officially-supported CPython distribution (e.g. CPython 3.6.8 ). Only install numpy (python -m pip install numpy). To my experience, the setenv is not necessary in this case.
For me, OutOfProcess mode proved to be buggy. Thus, I'd recommend to explicitly setting InProcess mode (for versions before R2019b, the OutOfProcess option is not present, as well as pyenv).
Do not concatenate the two .m files above into one - the py.importlib statements seem to be pre-executed and thus conflict with pyenv.
I'm working with an Eclipse-based tool which provides an interactive Jython shell for scripting and data analysis against an internal data model.
I'm trying to write a script which exports results to some form of database, so I'm trying to use the built-in com.ziclix.python.sql package in Jython to provide the interface and the Xerial JDBC connector for SQLite (https://github.com/xerial/sqlite-jdbc) to provide the backend.
The script below works perfectly when run outside of the third-party tool using a standard command line Jython interpreter, including relying on the importJar() hack which is commonly used to work around Jython not always using the user CLASSPATH when run using java -jar <blah>:
from com.ziclix.python.sql import zxJDBC
from java.net import URL, URLClassLoader
from java.lang import ClassLoader
from java.io import File
JDBC_URL = "jdbc:sqlite:test.db"
JDBC_DRIVER = "org.sqlite.JDBC"
JDBC_JAR = "E:/sqlite-jdbc-3.21.0.jar"
# Import Jar file into local class path
def importJar(jarFile):
m = URLClassLoader.getDeclaredMethod("addURL", [URL])
m.accessible = 1
m.invoke(ClassLoader.getSystemClassLoader(), [File(jarFile).toURL()])
def main():
try:
importJar(JDBC_JAR)
dbConn = zxJDBC.connect(JDBC_URL, None, None, JDBC_DRIVER)
cursor = dbConn.cursor()
# Do something useful
cursor.close()
dbConn.close()
except zxJDBC.DatabaseError, msg:
print msg
if __name__ == '__main__':
main()
... but fails when run from the plugin inside Eclipse, where the zxJDBC.connect() call errors with:
driver [org.sqlite.JDBC] not found
If I add the Jar file to the JYTHONPATH environment I can do import org.sqlite.JDBC successfully in the Python script, but the connect call still fails in the Java-side of the JDBC driver manager.
For sake of completeness the full path to the Jar file is on the CLASSPATH, PYTHONPATH, and JYTHONPATH environment variables ...
Any ideas?
I have a Plone project which is created by a buildout script and needs a default encoding of utf-8. This is usually done in the sitecustomize.py file of the Python installation. Since there is a virtualenv, I'd like to generate this file automatically, to contain something like:
import sys
sys.setdefaultencoding('utf-8')
After generation I have two empty sitecustomize.py files - one in parts/instance/, and one in parts/buildout; but none of these seems to be used (I couldn't find them in sys.path).
I tried zopepy:
>>> from os.path import join, isfile
>>> from pprint import pprint
>>> import sys
>>> pprint([p for p in sys.path
... if isfile(join(p, 'sitecustomize.py'))
... ])
and found another one in my local lib/python2.7/site-packages/ directory which looks good; but it doesn't work:
>>> import sys
>>> sys.getdefaultencoding()
'ascii'
This directory sits near the end of the sys.path, because I needed to add it by an extra-paths entry (to get another historical package).
Any pointers? Thank you!
System information: CentOS 7, Python 2.7.5
Edit:
I deleted them two empty sitecustomize.py files; now I have a default encoding of utf-8 in the zopepy session but still ascii in Plone; this surprises me, because I have in my buildout script:
[zopepy]
recipe=zc.recipe.egg
eggs = ${instance:eggs}
extra-paths = ${instance:extra-paths}
interpreter = zopepy
scripts = zopepy
To debug this, I created a little function which I added to my code, and which displays a little information about relevant modules in the sys.path:
import sys
from os.path import join, isdir, isfile
def sitecustomize_info():
plen = len(sys.path)
print '-' * 79
print 'sys.path has %(plen)d entries' % locals()
for tup in zip(range(1, plen+1), sys.path):
nr, dname = tup
if isdir(dname):
for fname in ('site.py', 'sitecustomize.py'):
if isfile(join(dname, fname)):
print '%(nr)4d. %(dname)s/%(fname)s' % locals()
spname = join(dname, 'site-packages', 'sitecustomize.py')
if isfile(spname):
print '%(nr)4d. %(spname)s' % locals()
else:
print '? %(dname)s is not a directory' % locals()
print '-' * 79
Output:
sys.path has 303 entries
8. /usr/lib64/python2.7/site-packages/sitecustomize.py
295. /opt/zope/instances/wnzkb/lib/python2.7/site-packages/sitecustomize.py
? /usr/lib64/python27.zip is not a directory
298. /usr/lib64/python2.7/site.py
? /usr/lib64/python2.7/lib-tk is not a directory
? /usr/lib64/python2.7/lib-old is not a directory
303. /usr/lib/python2.7/site-packages/sitecustomize.py
All sitecustomize.py files look the same (switching to utf-8), and I didn't tweak site.py (for now; if everything else fails, I might need to.)
If you really want/need to use the sitecustomize.py trick, you could include this part in your buildout:
[fixencode]
recipe = plone.recipe.command
stop-on-error = yes
update-command = ${fixencode:command}
command =
SITE_PACKAGES=$(${buildout:executable} -c \
'from distutils.sysconfig import get_python_lib;print(get_python_lib())')
cat > $SITE_PACKAGES/../sitecustomize.py << EOF
#!${buildout:executable} -S
import sys
sys.setdefaultencoding('utf-8')
EOF
It will be added into the site-packages folder from your virtualenv.
It looks like sitecustomize.py is not found anymore unless placed in the global lib directories (Discussion "deleting setdefaultencoding in site.py is evil" (2009), Tracker ticket "sitecustomize.py not found") - and this was made on purpose (!).
This is to prevent users from overriding the default encoding which might have been adjusted by some library. Libraries shouldn't do that, however.
Thus, whoever needs to set the default encoding is seduced to do it globally, which looks like a very silly idea to me. I'd consider it much more reasonable to set this in my virtual environment.
Unfortunately the sitecustomize.py modules in a virtualenv seem to be silently ignored; but it is possible to edit the local site.py. Here is a little sed script:
# vv-------1------vv vv---2---vv vv--------3------vv vv-----------4------------vv
sed --in-place -e 's,^\( encoding =\) \("ascii"\) \(# Default value\) \(set by _PyUnicode_Init()\),\1 "utf-8" \3 \2 \4,' lib/python2.7/site.py
I am working on an application that executes a Jython 2.5.3 script from JAVA 1.6.027. The script just open a file using codecs library and it looks like this:
try:
from codecs import open as codecs_open
except ImportError:
print 'ERROR', 'Could not import.'
CODECS_LIST = ['latin-1', 'utf-8', 'utf-16', '1250', '1252']
def open_file(filename, mode):
'''
DOC
'''
for encoding in CODECS_LIST:
try:
f = codecs_open(filename, mode, encoding)
f.read()
f.close()
print 'INFO', "File %s supports encoding %s." % (filename.split("\\")[-1], encoding)
...
except:
...
When I execute this script debugging in Eclipse, everything works OK, but when I execute the part of the JAVA application that invokes this script, I get this error:
Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class sun.nio.ch.FileChannelImpl
at java.io.RandomAccessFile.getChannel(RandomAccessFile.java:253)
at org.python.core.io.FileIO.fromRandomAccessFile(FileIO.java:173)
at org.python.core.io.FileIO.<init>(FileIO.java:79)
at org.python.core.io.FileIO.<init>(FileIO.java:57)
at org.python.core.PyFile.<init>(PyFile.java:135)
at org.python.core.PyTraceback.getLine(PyTraceback.java:65)
at org.python.core.PyTraceback.tracebackInfo(PyTraceback.java:38)
at org.python.core.PyTraceback.dumpStack(PyTraceback.java:109)
at org.python.core.PyTraceback.dumpStack(PyTraceback.java:120)
at org.python.core.Py.displayException(Py.java:1080)
at org.python.core.PySystemState.excepthook(PySystemState.java:1242)
at org.python.core.PySystemStateFunctions.__call__(PySystemState.java:1421)
at org.python.core.Py.printException(Py.java:1053)
at org.python.core.Py.printException(Py.java:1012)
at org.python.util.jython.run(jython.java:264)
at org.python.util.jython.main(jython.java:129)
The JAVA application is able to execute others similar jython scripts. I have detected that the class sun.nio.ch.FileChannelImpl is in the library rt.jar, which is inside /bin/common/ folder and included in my classpath via jvm.cfg file:
...
#LIBRARY PATH
./bin/common;...
...
The same way I do it with other libraries that work fine.
I have been stacked with this problem for a few days, so any help will be appreciated.
Thank you
The problem has been resolved by reinstalling Java Runtime Environment, in my case version jre-6u45
This happened to me because the mysql package that I had installed was installed globally and required root privileges, so when running java, I had to include sudo to get it to work correctly.
Where is ipython's configuration file, which starts with c = get_config(), executed? I'm asking because I want to understand what order things are done in ipython, e.g. why certain commands will not work if included as c.InteractiveShellApp.exec_lines.
This is related to my other question, Log IPython output?, because I want access to a logger attribute, but I can't figure out how to access it in the configuration file, and by the time exec_lines are run, the logger has already started (it's too late).
EDIT: I've accepted a solution based on using a startup file in ipython0.12+. Here is my implementation of that solution:
from time import strftime
import os.path
ip = get_ipython()
#ldir = ip.profile_dir.log_dir
ldir = os.getcwd()
fname = 'ipython_log_' + strftime('%Y-%m-%d') + ".py"
filename = os.path.join(ldir, fname)
notnew = os.path.exists(filename)
try:
ip.magic('logstart -o %s append' % filename)
if notnew:
ip.logger.log_write( u"########################################################\n" )
else:
ip.logger.log_write( u"#!/usr/bin/env python\n" )
ip.logger.log_write( u"# " + fname + "\n" )
ip.logger.log_write( u"# IPython automatic logging file\n" )
ip.logger.log_write( u"# " + '# Started Logging At: '+ strftime('%Y-%m-%d %H:%M:%S\n') )
ip.logger.log_write( u"########################################################\n" )
print " Logging to "+filename
except RuntimeError:
print " Already logging to "+ip.logger.logfname
There are only two subtle differences from the proposed solution linked:
1. saves log to cwd instead of some log directory (though I like that more...)
2. ip.magic_logstart doesn't seem to exist, instead one should use ip.magic('logstart')
The config system sets up a special namespace containing the get_config() function, runs the config file, and collects the values to apply them to the objects as they're created. Referring to your previous question, it doesn't look like there's a configuration value for logging output. You may want to start logging yourself after config, when you can control it more precisely. See this example of starting logging automatically.
Your other question mentions that you're limited to 0.10.2 on one system: that has a completely different config system that won't even look at the same file.