using boto3 in a python3 virtual env in AWS Lambda - virtualenv

I am trying to use Python3.4 and boto3 to walk an S3 bucket and publish some file locations to an RDS instance. The part of this effort I am having trouble with is when using boto3. My lambda function looks like the following:
import subprocess
def lambda_handler(event, context):
args = ("venv/bin/python3.4", "run.py")
popen = subprocess.Popen(args, stdout=subprocess.PIPE)
popen.wait()
output = popen.stdout.read()
print(output)
and, in my run.py file I have some lines:
import boto3
s3c = boto3.client('s3')
which cause an exception. The run.py file is not relevant for this question however, so in order make this post more concise, I've found that the cause of this error is generated with executing the lambda function:
import subprocess
def lambda_handler(event, context):
args = ("python3.4", "-c", "import boto3; print(boto3.client('s3'))")
popen = subprocess.Popen(args, stdout=subprocess.PIPE)
popen.wait()
output = popen.stdout.read()
print(output)
My logstream reports the error:
Event Data
START RequestId: 2b65421a-664d-11e6-81db-974c7c09d283 Version: $LATEST
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/var/runtime/boto3/__init__.py", line 79, in client
return _get_default_session().client(*args, **kwargs)
File "/var/runtime/boto3/session.py", line 250, in client
aws_session_token=aws_session_token, config=config)
File "/var/runtime/botocore/session.py", line 818, in create_client
client_config=config, api_version=api_version)
File "/var/runtime/botocore/client.py", line 63, in create_client
cls = self._create_client_class(service_name, service_model)
File "/var/runtime/botocore/client.py", line 85, in _create_client_class
base_classes=bases)
File "/var/runtime/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/var/runtime/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/var/runtime/boto3/utils.py", line 61, in _handler
module = import_module(module)
File "/var/runtime/boto3/utils.py", line 52, in import_module
__import__(name)
File "/var/runtime/boto3/s3/inject.py", line 13, in <module>
from boto3.s3.transfer import S3Transfer
File "/var/runtime/boto3/s3/transfer.py", line 135, in <module>
from concurrent import futures
File "/var/runtime/concurrent/futures/__init__.py", line 8, in <module>
from concurrent.futures._base import (FIRST_COMPLETED,
File "/var/runtime/concurrent/futures/_base.py", line 357
raise type(self._exception), self._exception, self._traceback
^
SyntaxError: invalid syntax
END RequestId: 2b65421a-664d-11e6-81db-974c7c09d283
REPORT RequestId: 2b65421a-664d-11e6-81db-974c7c09d283 Duration: 2673.45 ms Billed Duration: 2700 ms Memory Size: 1024 MB Max Memory Used: 61 MB
I need to use boto3 downstream of run.py. Any ideas on how to resolve this are much appreciated. Thanks!

Related

Python Imports Keep Failing

I recently wrote a package which is on 1.2.9 but when trying to import it using pip install rolimons Python tries to import 1.2.9 which fails and moves on until version 1.1.3 which is the only version that doesn't error out.
Here is the error:
ERROR: Command errored out with exit status 1:
command: /home/runner/rolimonspy-test/venv/bin/python3 /home/runner/rolimonspy-test/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp89bhulpc
cwd: /tmp/pip-install-baj2a_r_/rolimons_41a3df9c7f414766a57996a21dc8cd26
Complete output (16 lines):
Traceback (most recent call last):
File "/home/runner/rolimonspy-test/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 280, in <module>
main()
File "/home/runner/rolimonspy-test/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 263, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/runner/rolimonspy-test/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 133, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-029qi198/overlay/lib/python3.8/site-packages/poetry/core/masonry/api.py", line 41, in prepare_metadata_for_build_wheel
builder = WheelBuilder(poetry)
File "/tmp/pip-build-env-029qi198/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 56, in __init__
super().__init__(poetry, executable=executable)
File "/tmp/pip-build-env-029qi198/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/builder.py", line 83, in __init__
self._module = Module(
File "/tmp/pip-build-env-029qi198/overlay/lib/python3.8/site-packages/poetry/core/masonry/utils/module.py", line 69, in __init__
raise ModuleOrPackageNotFound(
poetry.core.masonry.utils.module.ModuleOrPackageNotFound: No file/folder found for package python-template
More information about the structure and package here:

JSON error while using Pandas output format

I am using alpha_vantage Timeseries API like below:
-----------------------------------------code------------------------------------
import pandas as pd
from alpha_vantage.timeseries import TimeSeries
from alpha_vantage.techindicators import TechIndicators
from matplotlib.pyplot import figure
import matplotlib.pyplot as plt
from pprint import pprint
#my key
key = 'mykey'
ts = TimeSeries(key, output_format='pandas')
def processMyBatch(batch, FD):
for i in batch:
df, meta_data = ts.get_quote_endpoint(i)
FD=FD.append(df)
return(FD)
main code...
for i in batches:
DF2=processMyBatch(i, DF)
DF=DF2
While the API worked for few symbols (see error log below), somewhere in between going through the list of symbols, I suddenly got the following JSONDecoder error ... but I am using output_format as pandas. Could you please throw some light on why this error occurred?
thank you
================error===============
/opt/scripts
starting now. fileName is: /mnt/NAS/Documents/../../../dailyquote2020-03-03.xlsx
completed the batch: ['AAPL', 'ABBV', 'AMZN', 'BAC', 'BNDX']
Waiting to honor API requirement: for 1 min
Waited: 65 sec
completed the batch: ['C', 'CNQ', 'CTSH', 'EEMV', 'FBGRX']
Waiting to honor API requirement: for 1 min
Waited: 65 sec
completed the batch: ['FDVV', 'FFNOX', 'FSMEX', 'FXAIX', 'GE']
Waiting to honor API requirement: for 1 min
Waited: 65 sec
Traceback (most recent call last):
File "getQuotes.py", line 55, in <module>
DF2=processMyBatch(i, DF)
File "getQuotes.py", line 29, in processMyBatch
df, meta_data = ts.get_quote_endpoint(i)
File "/home/username/.local/lib/python3.6/site-packages/alpha_vantage/alphavantage.py", line 174, in _format_wrapper
self, *args, **kwargs)
File "/home/username/.local/lib/python3.6/site-packages/alpha_vantage/alphavantage.py", line 159, in _call_wrapper
return self._handle_api_call(url), data_key, meta_data_key
File "/home/username/.local/lib/python3.6/site-packages/alpha_vantage/alphavantage.py", line 287, in _handle_api_call
json_response = response.json()
File "/home/username/.local/lib/python3.6/site-packages/requests/models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 518, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Added on 3/4/2020
..
..
completed the batch: ['FDVV', 'FFNOX', 'FSMEX', 'FXAIX', 'GE']
Waiting to honor API requirement: for 1 min
Waited: 65 sec
completed the batch: ['GOOGL', 'IGEB', 'IJH', 'IJR', 'IMTB']
Waiting to honor API requirement: for 1 min
Waited: 65 sec
Traceback (most recent call last):
File "getQuotes.py", line 55, in <module>
DF2=processMyBatch(i, DF)
..
..
Well I was getting an error like that today and it turns out the Alpha Vantage site is down!

"InterfaceError: connection already closed" when using multiprocessing.Pool on black box function that queries PostgreSQL database

I've been given a Python (2.7) function that takes 3 strings as arguments, and returns a list of dictionaries. Due to the nature of the project, I can't alter the function, which is quite complex, calling several other non-standard Python modules and querying a PostgreSQL database using psychopg2. I think that it's the Postgres functionality that's causing me problems.
I want to use the multiprocessing module to speed up calling the function hundreds of times. I've written a "helper" function so that I can use multiprocessing.Pool (which takes only 1 argument) with my function:
from function_script import function
def function_helper(args):
return function(*args)
And my main code looks like this:
from helper_script import function_helper
from multiprocessing import Pool
argument_a = ['a0', 'a1', ..., 'a99']
argument_b = ['b0', 'b1', ..., 'b99']
argument_c = ['c0', 'c1', ..., 'c99']
input = zip(argument_a, argument_b, argument_c)
p = Pool(4)
results = p.map(function_helper, input)
print results
What I'm expecting is a list of lists of dictionaries, however I get the following errors:
Traceback (most recent call last):
File "/local/python/2.7/lib/python2.7/site-packages/variantValidator/variantValidator.py", line 898, in validator
vr.validate(input_parses)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/validator.py", line 33, in validate
return self._ivr.validate(var, strict) and self._evr.validate(var, strict)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/validator.py", line 69, in validate
(res, msg) = self._ref_is_valid(var)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/validator.py", line 89, in _ref_is_valid
var_x = self.vm.c_to_n(var) if var.type == "c" else var
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/variantmapper.py", line 223, in c_to_n
tm = self._fetch_TranscriptMapper(tx_ac=var_c.ac, alt_ac=var_c.ac, alt_aln_method="transcript")
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/decorators/lru_cache.py", line 176, in wrapper
result = user_function(*args, **kwds)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/variantmapper.py", line 372, in _fetch_TranscriptMapper
self.hdp, tx_ac=tx_ac, alt_ac=alt_ac, alt_aln_method=alt_aln_method)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/transcriptmapper.py", line 69, in __init__
self.tx_identity_info = hdp.get_tx_identity_info(self.tx_ac)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/decorators/lru_cache.py", line 176, in wrapper
result = user_function(*args, **kwds)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/dataproviders/uta.py", line 353, in get_tx_identity_info
rows = self._fetchall(self._queries['tx_identity_info'], [tx_ac])
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/dataproviders/uta.py", line 216, in _fetchall
with self._get_cursor() as cur:
File "/local/python/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/dataproviders/uta.py", line 529, in _get_cursor
cur.execute("set search_path = " + self.url.schema + ";")
File "/local/python/2.7/lib/python2.7/site-packages/psycopg2/extras.py", line 144, in execute
return super(DictCursor, self).execute(query, vars)
DatabaseError: SSL error: decryption failed or bad record mac
And:
Traceback (most recent call last):
File "/local/python/2.7/lib/python2.7/site-packages/variantValidator/variantValidator.py", line 898, in validator
vr.validate(input_parses)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/validator.py", line 33, in validate
return self._ivr.validate(var, strict) and self._evr.validate(var, strict)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/validator.py", line 69, in validate
(res, msg) = self._ref_is_valid(var)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/validator.py", line 89, in _ref_is_valid
var_x = self.vm.c_to_n(var) if var.type == "c" else var
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/variantmapper.py", line 223, in c_to_n
tm = self._fetch_TranscriptMapper(tx_ac=var_c.ac, alt_ac=var_c.ac, alt_aln_method="transcript")
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/decorators/lru_cache.py", line 176, in wrapper
result = user_function(*args, **kwds)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/variantmapper.py", line 372, in _fetch_TranscriptMapper
self.hdp, tx_ac=tx_ac, alt_ac=alt_ac, alt_aln_method=alt_aln_method)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/transcriptmapper.py", line 69, in __init__
self.tx_identity_info = hdp.get_tx_identity_info(self.tx_ac)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/decorators/lru_cache.py", line 176, in wrapper
result = user_function(*args, **kwds)
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/dataproviders/uta.py", line 353, in get_tx_identity_info
rows = self._fetchall(self._queries['tx_identity_info'], [tx_ac])
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/dataproviders/uta.py", line 216, in _fetchall
with self._get_cursor() as cur:
File "/local/python/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/local/python/2.7/lib/python2.7/site-packages/hgvs/dataproviders/uta.py", line 526, in _get_cursor
conn.autocommit = True
InterfaceError: connection already closed
Does anybody know what might cause the Pool function to behave like this, when it seems so simple to use in other examples that I've tried? If this isn't enough information to go on, can anyone advise me on a way of getting to the bottom of the problem (this is the first time I've worked with someone else's code)? Alternatively, are there any other ways that I could use the multiprocessing module to call the function hundreds of times?
Thanks
I think what may be happening is that your connection object is used across all workers and when 1 worker has completed all its tasks it closes the connection and meanwhile the other workers are still working and the connection is closed so when one of those workers tries to use the db it is already closed.

Send csv-file as attachment via MIMEBase - python

I don't get it. I try to send an e-mail with a csv-file as attachment. The first run with the script was fine, I recieved the e-mail with the csv-file as attachment. But after running it once, the script crashes with an odd Traceback message.
Heres my code so far:
import smtplib
import email.utils
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
from email.mime.text import MIMEText
from email import Encoders
msg = MIMEMultipart('foo')
msg['To'] = email.utils.formataddr(('foo', 'foo#foobar.de'))
msg['From'] = email.utils.formataddr(('foo', 'foo#foo'))
msg['Subject'] = 'foo'
# Attach a file
mail_file = file('foo.csv').read()
mail_file = MIMEBase('application', 'csv')
mail_file.set_payload(mail_file)
mail_file.add_header('Content-Disposition', 'attachment', filename='foo.csv')
Encoders.encode_base64(mail_file)
msg.attach(mail_file)
# Define SMTP server
server = smtplib.SMTP('localhost')
server.set_debuglevel(True) # show communication with the server
# Send the mail
try:
server.sendmail('foo#foo', ['foo#foobar.de'], msg.as_string())
finally:
server.quit()
Now I get the following traceback:
Traceback (most recent call last):
File "mail.py", line 27, in <module>
Encoders.encode_base64(mail_file)
File "/usr/lib/python2.7/email/encoders.py", line 45, in encode_base64
encdata = _bencode(orig)
File "/usr/lib/python2.7/email/encoders.py", line 31, in _bencode
hasnewline = (s[-1] == '\n')
File "/usr/lib/python2.7/email/message.py", line 294, in __getitem__
return self.get(name)
File "/usr/lib/python2.7/email/message.py", line 360, in get
name = name.lower()
AttributeError: 'int' object has no attribute 'lower'
When I uncomment line 27 (Encoding Line), I get the following:
Traceback (most recent call last):
File "mail.py", line 33, in <module>
server.sendmail('foo#foo', ['foo#foobar.de'], msg.as_string())
File "/usr/lib/python2.7/email/message.py", line 137, in as_string
g.flatten(self, unixfrom=unixfrom)
File "/usr/lib/python2.7/email/generator.py", line 83, in flatten
self._write(msg)
File "/usr/lib/python2.7/email/generator.py", line 108, in _write
self._dispatch(msg)
File "/usr/lib/python2.7/email/generator.py", line 134, in _dispatch
meth(msg)
File "/usr/lib/python2.7/email/generator.py", line 203, in _handle_multipart
g.flatten(part, unixfrom=False)
File "/usr/lib/python2.7/email/generator.py", line 83, in flatten
self._write(msg)
File "/usr/lib/python2.7/email/generator.py", line 108, in _write
self._dispatch(msg)
File "/usr/lib/python2.7/email/generator.py", line 134, in _dispatch
meth(msg)
File "/usr/lib/python2.7/email/generator.py", line 177, in _handle_text
raise TypeError('string payload expected: %s' % type(payload))
TypeError: string payload expected: <type 'instance'>
Any ideas?
Strange, when I do
mail_file.set_payload(open('foo.csv', 'rb').read())
instead of
mail_file = file('foo.csv').read()
mail_file.set_payload(mail_file)
the Code works.. Still wondering why...!?

Error while start openerp

i have installed openerp. i have created postgres database. While i open ¨localhost:8060¨ in browser, display the followin error:
Client Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/addons/web/http.py", line 204, in dispatch
response["result"] = method(self, **self.params)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/addons/web/controllers/main.py", line 761, in get_list
monodb = db_monodb(req)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/addons/web/controllers/main.py", line 129, in db_monodb
return db_redirect(req, True)[0]
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/addons/web/controllers/main.py", line 109, in db_redirect
dbs = db_list(req, True)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/addons/web/controllers/main.py", line 90, in db_list
dbs = proxy.list(force)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/addons/web/session.py", line 30, in proxy_method
result = self.session.send(self.service_name, method, *args)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/addons/web/session.py", line 103, in send
raise xmlrpclib.Fault(openerp.tools.ustr(e), formatted_info)
Server Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/addons/web/session.py", line 89, in send
return openerp.netsvc.dispatch_rpc(service_name, method, args)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/netsvc.py", line 292, in dispatch_rpc
result = ExportService.getService(service_name).dispatch(method, params)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/service/web_services.py", line 122, in dispatch
return fn(*params)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/service/web_services.py", line 359, in exp_list
cr = db.cursor()
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/sql_db.py", line 484, in cursor
return Cursor(self._pool, self.dbname, serialized=serialized)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/sql_db.py", line 182, in __init__
self._cnx = pool.borrow(dsn(dbname))
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/sql_db.py", line 377, in _locked
return fun(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/sql_db.py", line 440, in borrow
result = psycopg2.connect(dsn=dsn, connection_factory=PsycoConnection)
File "/usr/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
connection_factory=connection_factory, async=async)
OperationalError: FATAL: role "bala" does not exist
otherwise i started like following ¨openerp-server start¨, error following like
Traceback (most recent call last):
File "/usr/local/bin/openerp-server", line 5, in <module>
pkg_resources.run_script('openerp==7.0-20140110-002122', 'openerp-server')
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 499, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1235, in run_script
execfile(script_filename, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/EGG-INFO/scripts/openerp-server", line 5, in <module>
openerp.cli.main()
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/cli/__init__.py", line 51, in main
__import__(m)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/modules/module.py", line 133, in load_module
mod = imp.load_module('openerp.addons.' + module_part, f, path, descr)
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/addons/l10n_si/__init__.py", line 22, in <module>
import account_wizard
File "/usr/local/lib/python2.7/dist-packages/openerp-7.0_20140110_002122-py2.7.egg/openerp/addons/l10n_si/account_wizard.py", line 22, in <module>
import tools
ImportError: No module named tools
Anyone help for error correction. i cannot clear this error even two days i spend for this.
OperationalError: FATAL: role "bala" does not exist
A user that you have told OpenERP to use does not exist in the database. You probably need to create it. I expect you missed one or more installation steps.
Look for a CREATE USER, CREATE ROLE, or createuser command in the instructions.
GRANT commands or a database ownership assignment may also be nececssary depending on what the user is supposed to be able to do.
(It might also be assumed that you'll know to create the user to connect to the DB as).