I've built a deep learning model that uses for pre-processing purposes a custom library called Augmentor and I would like to include it when running on Watson Machine learning service on the cloud.
For now I am using a workaround in my code which uses pip to install the library:
import pip
def install(package):
if hasattr(pip, 'main'):
pip.main(['install', package])
else:
pip._internal.main(['install', package])
try:
import Augmentor
except:
install("Augmentor")
import Augmentor
I've been looking at the command line interface bx ml and I noticed that there is a command line bx ml generate-manifest libraries which generates libraries.yml:
name: libraries_custom
description: custom libraries for scoring
version: '1.0'
platform:
name: python
versions:
- '3.5'
I would like to know if this is the right why to declare custom libraries required to run my code on WML ?
I can't find any documentation on this anywhere.
As for as, I am aware of you can create custom components using generate-manifest of ibmcloud CLI
bx ml store libraries thepyfuncpackage-0.1.zip library.json
You can find the complete instructions here
If you want to install additional libraries to support your work, you must create a wheel (.whl) file and include this with the .zip file that you submit as part of your training run or experiment. You can then execute them by running the pip install /.whl command.
Complete instructions here
Hope this is helpful
Related
I am trying to use the psycopg2 library with AWS lambda in my Pycharm IDE on Windows 10.
I have tried all versions of Python listed in this repository and I still always get the following error:
{"errorMessage": "Unable to import module 'app': No module named 'psycopg2._psycopg'", "errorType": "Runtime.ImportModuleError", "stackTrace": []}
I have tried this solution where we install aws-psycopg2 but it did not work. As per this solution, I have ensured that my local Python runtime and the Lambda runtime are the same and I still get the error.
I have made sure that my Python version is 64 bit as per this answer
Hey incase it still didn't work, i faced the same issue here's what i did:
ran this command:
pip install --platform=manylinux1_x86_64 --only-binary=:all: psycopg2-binary --target psycopg-binary/python/lib/python3.8/site-packages
created a lambda layer with the above package (pyscopg-binary) and added it to the lambda function
make sure runtimes are correct ( at first i had the lamda layer runtime 3.9 and the function 3.9, then i switched both to 3.8 and it worked)
Im not absolutely sure aside from the runtimes which of the packages caused it to work (since i did multiple things in one step: added aws-psycopg2 to the requirements.txt in addition to changing runtime, hence im not sure if the aws-psycopg2 had anything to do with it working :), but i think that it has nothing to do with it, so it should work without it).
hope this helps.
I have installed Robocorp Code as well as Robot Framework Language Server and have configured them. However, I am still having errors when trying to run the tests via the code lens options.
Repo - A webapi repo with a specific folder containing all tests. Lets call it regression.
RF - 4.1.3
Python - 3.8
This is what happens when I click on Run on the code lens for any of the tests -
`PS C:\git\xxxx\regression> C:; cd 'C:\git\xxxx\regression'; &
'C:\Users\xxxx\AppData\Local\Temp\rf-ls-run\run_env_00_smh5defr.bat'
'-u'
'c:\Users\xxxx.vscode\extensions\robocorp.robotframework-lsp-0.47.2\src\robotframework_debug_adapter\run_robot__main__.py'
'--port' '54331' '--no-debug' '--argumentfile'
'C:\git\xxxx\regression\args-local.txt' '--pythonpath'
'c:\git\xxxx\regression\common\lib' '--variable'
'EXECDIR:C:/git/xxxx/regression'
'--prerunmodifier=robotframework_debug_adapter.prerun_modifiers.FilteringTestsSuiteVisitor'
'c:\git\xxxx\regression\api\api_Test.robot'
[ ERROR ] Parsing'--pythonpath' failed: File or directory to execute does not exist.
However, the test starts if I remove the argumentfile parameter but it, of course, fails because its missing arguments from the file.
Do note that the folder specified in pythopath exists and has some python libraries needed for the tests.
Say I develop a package with a limited set of dependencies (for example, LinearAlgebra).
In the Unit testing part, I might need additional dependencies (for instance, CSV to load a file). I can configure that in the Project.toml all good.
Now from there and in VS Code, how can I debug the Unit tests? I tried running the "runtests.jl" in the debugger; however, it unsurprisingly complains that the CSV package is unavailable.
I could add the CSV package (as a temporary solution), but I would prefer that the debugger run with the configuration for the unit testing; how can I achieve that?
As requested, here is how it can be reproduced (it is not quite minimal, but instead I used a commonly used package as it give confidence the package is not the problem). We will use DataFrames and try to execute the debugger for its unit tests.
Make a local version of DataFrames for the purpose of developing a feature in it. I execute dev DataFrames in a new REPL.
Select the correct environment (in .julia/dev/DataFrames) through the VS-code user interface.
Execute the "proper" unit testing by executing test DataFrames at the pkg prompt. Everything should go smoothly.
Try to execute the tests directly (open the runtests.jl and use the "Run" button in vs-code). I see some errors of the type:
LoadError: ArgumentError: Package CategoricalArrays not found in current path:
- Run `import Pkg; Pkg.add("CategoricalArrays")` to install the CategoricalArrays package.
which is consistent with CategoricalArrays being present in the [extras] section of the Project.toml but not present in the [deps].
Finally, instead of the "Run" command, execute the "Run and Debug". I encounter similar errors here is the first one:
Test Summary: | Pass Total
merge | 19 19
PASSED: index.jl
FAILED: dataframe.jl
LoadError: ArgumentError: Package DataStructures not found in current path:
- Run `import Pkg; Pkg.add("DataStructures")` to install the DataStructures package.
So I can't debug the code after the part requiring the extras packages.
After all that I delete this package with the command free DataFrames at the pkg prompt.
I see the same behavior in my package.
I'm not certain I understand your question, but I think you might be looking for the TestEnv package. It allows you to activate a temporary environment containing the [extras] dependencies. The discourse announcement contains a good description of the use cases.
Your runtest.jl file should contain all necessary imports to run tests.
Hence you are expected to have in your runtests.jl file lines such as:
using YourPackageName
using CSV
# the lines with tests now go here.
This is a standard in Julia package layout. For an example have a look at any mature Julia such as DataFrames.jl (https://github.com/JuliaData/DataFrames.jl/blob/main/test/runtests.jl).
I'm creating Conda create environment from yml I generated on Windows' Miniconda install. I need to create same environment on OS X. Following the advise found here on SO I used the --no-builds option.
Also, the names of some packages under section ResolvePackageNotFound are clearly (many if not all) specific to Windows:
- m2w64-gmp=6.1.0
- m2w64-gcc-libs-core=5.3.0
- m2w64-gcc-libs=5.3.0
- vc=14.1
- vs2015_runtime=15.5.2
- msys2-conda-epoch=20160418
- menuinst=1.4.14
- icc_rt=2019.0.0
- m2w64-libwinpthread-git=5.0.0.4634.697f757
- pywinpty=0.5.5
- wincertstore=0.2
- m2w64-gcc-libgfortran=5.3.0
- win_inet_pton=1.1.0
- winpty=0.4.3
I removed all of these from the yml file. Even then it's stalled at the following screen:
(base) MacBook-Air:Anaconda.d xtian$ conda env create -f 32b-qb-2019-10-05.yml
Collecting package metadata (repodata.json): done
Solving environment: \
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abor|
Examining openssl: 10%|█████████▍ | 29/279 [00:00<00:00, 3729.87it- ]
Comparing specs that have this dependency: 16%|██████████▉ | 16/101 [05:53<31:19, 22.11s/it]
Finding shortest conflict path for openssl[version='>=1.0.2p,<1.0.3a']: 38%|███████████████▊ | 6/16 [02:39<06:23, 38.32s/it]
This process is progressing at an astonishingly slow pace, and hasn't got past openssl ... 29/279. Should I wait and trust Conda can figure this all out?
Or,
Do I need another strategy--
I'm wondering if I can't remove the offending packages, each in turn, and create a series of yml files to install in order using, $ conda env update --prefix ./env --file environment.yml --prune, because whatever finally works here I know I'm going to need to use it on another machine so I can share the project env with a colleague.
Any other suggestions?
Short answer: Try deleting the packages that your system is getting stuck on from the .yml file. i.e., remove "openssl" from .yml file.
I have been running into the same issue trying to install a .yml file created in a Windows system to a Mac system. I basically followed the same procedure you did:
-Created yml file using the --no-builds option.
-Attempted to create environment on Mac system and had several windows specific packages left under ResolvePackageNotFound section (listed below)
m2w64-libwinpthread-git=5.0.0.4634.697f757
pyreadline=2.1
pywinpty=0.5.5
m2w64-gcc-libgfortran=5.3.0
vc=14
m2w64-gcc-libs-core=5.3.0
m2w64-gmp=6.1.0
wincertstore=0.2
icc_rt=2019.0.0
m2w64-gcc-libs=5.3.0
vs2015_runtime=14.15.26706
winpty=0.4.3
msys2-conda-epoch=20160418
-Deleted those from the yml file
-Attempted to create environment from updated yml file and received the following conflicts:
- Found conflicts! Looking for incompatible packages.
My system also got stuck trying to solve the "openssl" conflict along with a "_tflow_select". I ended up deleting those and was able to create my environment and run the code without too much trouble.
I have a whole bunch of working C++ code that I want to write Python bindings for. I'm trying to use Boost.Python since it seems to be the easiest way to get this working, but it isn't cooperating. Here's part of the code for the extension module I'm trying to build:
BOOST_PYTHON_MODULE(libpcap_ext) {
using namespace boost::python;
class_<PacketEngine>("PacketEngine")
.def("getAvailableDevices", &PacketEngine_getAvailableDevices);
}
Bjam seems to be a pain and refuses to recognize my Pythonpath or allow me to link with libpcap, so I'm using CMake. Here's my CMakeLists file, which can import and build everything just fine (outputs libpcap.so as expected):
CMAKE_MINIMUM_REQUIRED(VERSION 2.8)
IF(NOT CMAKE_BUILD_TYPE)
SET(CMAKE_BUILD_TYPE "DEBUG")
#SET(CMAKE_BUILD_TYPE "RELEASE")
#SET(CMAKE_BUILD_TYPE "RELWITHDEBINFO")
#SET(CMAKE_BUILD_TYPE "MINSIZEREL")
ENDIF()
FIND_PACKAGE(Boost 1.55.0)
find_package(PythonLibs REQUIRED)
IF(Boost_FOUND)
INCLUDE_DIRECTORIES("${Boost_INCLUDE_DIRS}" "${PYTHON_INCLUDE_DIRS}")
SET(Boost_USE_STATIC_LIBS OFF)
SET(Boost_USE_MULTITHREADED ON)
SET(Boost_USE_STATIC_RUNTIME OFF)
FIND_PACKAGE(Boost 1.55.0 COMPONENTS python)
ADD_LIBRARY(pcap_ext MODULE PacketWarrior/pcap_ext.cc PacketWarrior/PacketEngine.h PacketWarrior/PacketEngine.cc PacketWarrior/Packet.h PacketWarrior/Packet.cc)
TARGET_LINK_LIBRARIES(pcap_ext pcap)
TARGET_LINK_LIBRARIES(pcap_ext ${Boost_LIBRARIES} ${PYTHON_LIBRARIES})
ELSEIF(NOT Boost_FOUND)
MESSAGE(FATAL_ERROR "Unable to find correct Boost version. Did you set BOOST_ROOT?")
ENDIF()
ADD_DEFINITIONS("-Wall")
And my pcap.py file that attempts to utilize the module:
import libpcap_ext
engine = libpcap_ext.PacketEngine()
print engine.getAvailableDevices()
But whenever I try to run the module, I get the following error:
Traceback (most recent call last):
File "../pcap.py", line 2, in <module>
engine = libpcap_ext.PacketEngine()
TypeError: __init__() should return None, not 'NoneType
I'm assuming it's because Boost.Python is trying to use Python 3 and my system default is Python 2.7.3. I've tried changing my user-config.jam file (in my boost_1_55_0 directory) to point to Python 2.7 and tried building:
# Configure specific Python version.
# using python : 2.7 : /usr/bin/python2.7 : /usr/include/python2.7 : /usr/lib ;
Boost.Python's installation instructions [0] seem to fail for me when I try to build quickstart with bjam (lots of warnings), so I tried following the Boost Getting Started instructions [1] to build a Python header binary, which is I think what is causing this problem. Any recommendations as to how to fix this would be amazing, I've spent hours on this.
This error is probably due to linking against the wrong Python library. Make sure your extension as well as the Boost Python library are linked against the Python installation you are using to import the module.
On Linux you can check against which libraries you've linked with ldd. On OS X otool -L does the same thing. So, for example
otool -L libpcap_ext.so
otool -L /path/to/libboost_python-mt.dylib
should list the Python library they are linked against.
With CMake you can use the variable PYTHON_LIBRARY to change which Python library is used. As an example, on the command line you can set it with
cmake -DPYTHON_LIBRARY="/path/to/libpython2.7.dylib" source_dir
Lastly, on OS X a quick and dirty way (i.e. without recompiling) to change the dynamically linked libraries is install_name_tool -change.