I am trying to use the psycopg2 library with AWS lambda in my Pycharm IDE on Windows 10.
I have tried all versions of Python listed in this repository and I still always get the following error:
{"errorMessage": "Unable to import module 'app': No module named 'psycopg2._psycopg'", "errorType": "Runtime.ImportModuleError", "stackTrace": []}
I have tried this solution where we install aws-psycopg2 but it did not work. As per this solution, I have ensured that my local Python runtime and the Lambda runtime are the same and I still get the error.
I have made sure that my Python version is 64 bit as per this answer
Hey incase it still didn't work, i faced the same issue here's what i did:
ran this command:
pip install --platform=manylinux1_x86_64 --only-binary=:all: psycopg2-binary --target psycopg-binary/python/lib/python3.8/site-packages
created a lambda layer with the above package (pyscopg-binary) and added it to the lambda function
make sure runtimes are correct ( at first i had the lamda layer runtime 3.9 and the function 3.9, then i switched both to 3.8 and it worked)
Im not absolutely sure aside from the runtimes which of the packages caused it to work (since i did multiple things in one step: added aws-psycopg2 to the requirements.txt in addition to changing runtime, hence im not sure if the aws-psycopg2 had anything to do with it working :), but i think that it has nothing to do with it, so it should work without it).
hope this helps.
Related
I'm working on a pybind11 extension written in C++ but I'm having a hard time understanding how should it be distributed.
The project links to a number of third party libraries (e.g. libpng, glew etc.).
The project builds fine with CMAKE and it generates a .so file. Now I am not sure what is the right way of installing this extension. The extension seems to work, as if I try copy the file into the python lib directories it is picked up (I can import it, and it works correctly). However, this is clearly not the way to go I think.
I also tried the setuptools route (from https://pybind11.readthedocs.io/en/stable/compiling.html) by creating a setup.py files like this:
import sys
# Available at setup time due to pyproject.toml
from pybind11 import get_cmake_dir
from pybind11.setup_helpers import Pybind11Extension, build_ext
from setuptools import setup
from glob import glob
files = sorted(glob("*.cpp"))
__version__ = "0.0.1"
ext_modules = [
Pybind11Extension("mylib",
files,
# Example: passing in the version to the compiled code
define_macros = [('VERSION_INFO', __version__)],
),
]
setup(
name="mylib",
version=__version__,
author="fab",
author_email="fab#fab",
url="https://github.com/pybind/python_example",
description="mylib",
long_description="",
ext_modules=ext_modules,
extras_require={"test": "pytest"},
cmdclass={"build_ext": build_ext},
zip_safe=False,
python_requires=">=3.7",
)
and now I can build the extension by simply calling
pip3 install
however it looks like all the links are broken because whenever I try importing the extension in Python I get linkage errors, as if setuptools does not link correctly the extension with the 3rd party libs. For instance errors in linking with libpng as in:
>>> import mylib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /home/fabrizio/.local/lib/python3.8/site-packages/mylib.cpython-38-x86_64-linux-gnu.so: undefined symbol: png_sig_cmp
However I have no clue how to add this link info to setuptools, and don't even know if that's possible (it should be the setuptools equivalent of CMAKE's target_link_libraries).
I am really at a loss after weeks of reading documentation, forum threads and failed attempts. If anyone is able to point me in the right way or to clear some of the fog it would be really appreciated!
Thanks!
Fab
/home/fabrizio/.local/lib/python3.8/site-packages/mylib.cpython-38-x86_64-linux-gnu.so: undefined symbol: png_sig_cmp
This line pretty much says it clearly. Your local shared object file .so can't find the libpng.so against which it is linked.
You can confirm this by running:
ldd /home/fabrizio/.local/lib/python3.8/site-packages/mylib.cpython-38-x86_64-linux-gnu.so
There is no equivalent of target_link_libraries() in setuptools. Because that wouldn't make any sense. The library is already built and you've already linked it. This is your system more or less telling you that it can't find the libraries it needs. And those most likely need to be installed.
This is also one of the reasons why Linux distributions provide their own package managers and why you should use the developer packages provided by said distributions.
So how do you fix this? Well your .so file needs to find the other .so files against which you linked to understand how this works I will refer you to this link.
My main guess is based on the fact that when you manually copy the files it works - That during the build process you probably specify the rpath to a local directory. Hence what you most likely need to do is specify to your setuptools that it needs to copy those files when installing.
I'm trying to create a layer and use it in Lambda function for PostgreSQL but getting the issue:
"errorMessage": "Unable to import module 'lambda_function': No module named 'psycopg2'",
"errorType": "Runtime.ImportModuleError"
I've tried uploading the Psycopg2 package from https://github.com/jkehler/awslambda-psycopg2 but still no progress. I've tried every version from 3.6 to 3.9. I'm using Windows OS.
What is the simplest way or the best solution for it?
I'm new to Purescript and am following the tutorial for installation. Purescript itself is working and I can start the CLI using pulp psci, but installing purescript-list runs into trouble.
Having entered the command bower install purescript-lists --save, I get a long list of package names, but when it gets to purescript-eff and purescript-prelude I run into some version conflicts:
bower purescript-eff#^2.0.0 cached https://github.com/purescript/purescript-eff.git#2.0.0
bower purescript-eff#^2.0.0 validate 2.0.0 against https://github.com/purescript/purescript-eff.git#^2.0.0
Unable to find a suitable version for purescript-eff, please choose one by typing one of the numbers below:
1) purescript-eff#^1.0.0 which resolved to 1.0.0 and is required by purescript-console#1.0.0
2) purescript-eff#^2.0.0 which resolved to 2.0.0 and is required by purescript-st#2.0.0
Prefix the choice with ! to persist it to bower.json
? Answer
A similar message is shown for purescript-prelude. No matter which options I choose, both pulp build and pulp run fail with:
$ pulp build
* Building project in /Developer/purescript/training1
Error found:
in module PSCI.Support
at /Developer/purescript/training1/bower_components/purescript-psci-support/src/PSCI/Support.purs line 10, column 34 - line 10, column 53
Cannot import value unsafeInterleaveEff from module Control.Monad.Eff.Unsafe
It either does not exist or the module does not export it.
See https://github.com/purescript/purescript/wiki/Error-Code-UnknownImport for more information,
or to contribute content related to this error.
Compiling PSCI.Support
* ERROR: Subcommand terminated with exit code 1
What have I missed here?
Thanks
Chris W
If you are using psc version 0.10.* you should go with prelude, lists and eff v2*.
If you are using psc version 0.9.* you should go with prelude, lists and eff v1*.
If you are using psc 0.10.* you might want to update pulp to version 9.1.0
The problem occurs due to breaking changes between psc 0.9 and 0.10 and the relevant libraries. by writing bower install purescript-lists --save you are asking bower for the latest dependencies which conflict with the dependency versions specified in your bower.json.
I have a whole bunch of working C++ code that I want to write Python bindings for. I'm trying to use Boost.Python since it seems to be the easiest way to get this working, but it isn't cooperating. Here's part of the code for the extension module I'm trying to build:
BOOST_PYTHON_MODULE(libpcap_ext) {
using namespace boost::python;
class_<PacketEngine>("PacketEngine")
.def("getAvailableDevices", &PacketEngine_getAvailableDevices);
}
Bjam seems to be a pain and refuses to recognize my Pythonpath or allow me to link with libpcap, so I'm using CMake. Here's my CMakeLists file, which can import and build everything just fine (outputs libpcap.so as expected):
CMAKE_MINIMUM_REQUIRED(VERSION 2.8)
IF(NOT CMAKE_BUILD_TYPE)
SET(CMAKE_BUILD_TYPE "DEBUG")
#SET(CMAKE_BUILD_TYPE "RELEASE")
#SET(CMAKE_BUILD_TYPE "RELWITHDEBINFO")
#SET(CMAKE_BUILD_TYPE "MINSIZEREL")
ENDIF()
FIND_PACKAGE(Boost 1.55.0)
find_package(PythonLibs REQUIRED)
IF(Boost_FOUND)
INCLUDE_DIRECTORIES("${Boost_INCLUDE_DIRS}" "${PYTHON_INCLUDE_DIRS}")
SET(Boost_USE_STATIC_LIBS OFF)
SET(Boost_USE_MULTITHREADED ON)
SET(Boost_USE_STATIC_RUNTIME OFF)
FIND_PACKAGE(Boost 1.55.0 COMPONENTS python)
ADD_LIBRARY(pcap_ext MODULE PacketWarrior/pcap_ext.cc PacketWarrior/PacketEngine.h PacketWarrior/PacketEngine.cc PacketWarrior/Packet.h PacketWarrior/Packet.cc)
TARGET_LINK_LIBRARIES(pcap_ext pcap)
TARGET_LINK_LIBRARIES(pcap_ext ${Boost_LIBRARIES} ${PYTHON_LIBRARIES})
ELSEIF(NOT Boost_FOUND)
MESSAGE(FATAL_ERROR "Unable to find correct Boost version. Did you set BOOST_ROOT?")
ENDIF()
ADD_DEFINITIONS("-Wall")
And my pcap.py file that attempts to utilize the module:
import libpcap_ext
engine = libpcap_ext.PacketEngine()
print engine.getAvailableDevices()
But whenever I try to run the module, I get the following error:
Traceback (most recent call last):
File "../pcap.py", line 2, in <module>
engine = libpcap_ext.PacketEngine()
TypeError: __init__() should return None, not 'NoneType
I'm assuming it's because Boost.Python is trying to use Python 3 and my system default is Python 2.7.3. I've tried changing my user-config.jam file (in my boost_1_55_0 directory) to point to Python 2.7 and tried building:
# Configure specific Python version.
# using python : 2.7 : /usr/bin/python2.7 : /usr/include/python2.7 : /usr/lib ;
Boost.Python's installation instructions [0] seem to fail for me when I try to build quickstart with bjam (lots of warnings), so I tried following the Boost Getting Started instructions [1] to build a Python header binary, which is I think what is causing this problem. Any recommendations as to how to fix this would be amazing, I've spent hours on this.
This error is probably due to linking against the wrong Python library. Make sure your extension as well as the Boost Python library are linked against the Python installation you are using to import the module.
On Linux you can check against which libraries you've linked with ldd. On OS X otool -L does the same thing. So, for example
otool -L libpcap_ext.so
otool -L /path/to/libboost_python-mt.dylib
should list the Python library they are linked against.
With CMake you can use the variable PYTHON_LIBRARY to change which Python library is used. As an example, on the command line you can set it with
cmake -DPYTHON_LIBRARY="/path/to/libpython2.7.dylib" source_dir
Lastly, on OS X a quick and dirty way (i.e. without recompiling) to change the dynamically linked libraries is install_name_tool -change.
The following modules appear to be missing
email.Generator
email.Iterators
email.Utils
win32api
win32con
w in32pipe
wx
My setup file looks like this:
from distutils.core import setup
import py2exe
setup(console=['fwsm_migration.py'])
i'm using Python 2.5.4 and the py2exe 0.6.8
Looked here and outside for a peculiar solution but have not found one!!
read about using "optoins: but being new to python itself failing to know where to do it.
Please HELP!
Try cx_freeze. After having a hell of a time with py2exe cx_freeze compiled my script without any configuration. In the same environment Py2exe claimed I'd missed nine packages.
For simple scripts you only need to do:
cxfreeze hello.py --target-dir dist