I need to create a small WebDAV client that just upload files on the server.
I've found "requests" library that seems to be very easy to be used but I'm not able to use it properly.
The client should transfer binary files - so I've used the example bellow:
>>> url = 'http://IPADDR/webdav'
>>> files = {'report.xls': open('report.xls', 'rb')}
>>> r = requests.post(url, files=files)
from http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file.
For me it's not working, I have the following error:
File ".../site-packages/requests/packages/urllib3/connectionpool.py", line 260, in _make_request
conn.request(method, url, **httplib_request_kw)
File ".../httplib.py", line 941, in request
self._send_request(method, url, body, headers)
File ".../httplib.py", line 975, in _send_request
self.endheaders(body)
File ".../httplib.py", line 937, in endheaders
self._send_output(message_body)
File ".../httplib.py", line 795, in _send_output
msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 147: ordinal not in range(128)
Should be the input file somehow encoded? (I didn't found anything related in the "requests" documentation).
After some debugging, I've actually found what's happening.
I was able to fix the issue by removing the following import in my script:
from __future__ import unicode_literals
This import seems to cause unwanted string conversions in urllib3 (which requests relies on).
As requests' author explained, this issue should be filed against urllib3.
Related
i have problem with pyinstaller i use library opcua for comunication, with normal run in pycharm everything works correctly, after create .exe with pyinstaller after few second or minute the comunication stops.
from opcua import Client
from opcua.crypto import security_policies
from opcua.ua import MessageSecurityMode
from opcua import ua
def connect(self):
try:
self.client.set_security(policy=security_policies.SecurityPolicyBasic256Sha256,
certificate_path=self.root_path+CERT_RELATIVE_PATH,
private_key_path=self.root_path+PRIVATE_KEY_RELATIVE_PATH,
mode=MessageSecurityMode.SignAndEncrypt
)
self.client.application_uri = "urn:script:opcua"
# self.client.secure_channel_timeout = 10000
self.client.session_timeout = 60000
self.client.connect()
self.root_node = self.client.get_root_node()
self.call_method('ns=2;s=/Methods/GiveUserAccess', 'OpcUaClient', 'GudWrite')
return True
except Exception:
self.main_log.info('unable to connect to nc: {}'.format(traceback.format_exc()))
raise custom_exceptions.UnableToConnect()
def get_battery_100(self):
return self.client.get_node('ns=2;s=/NC/_N_NC_GD2_ACX/DG_BATTERY_TIME').get_value()
This is the error after some data exchange:
WARNING:opcua.uaprotocol:Received an error: MessageAbort(error:StatusCode(BadSecurityChecksFailed), reason:None)
CRITICAL:opcua.client.ua_client.Socket:Received an error: MessageAbort(error:StatusCode(BadSecurityChecksFailed), reason:None)
ERROR:opcua.client.ua_client.Socket:Protocol Error
..
File "opcua\client\ua_client.py", line 101, in _run
File "opcua\client\ua_client.py", line 121, in _receive
File "opcua\client\ua_client.py", line 129, in _call_callback
opcua.ua.uaerrors._base.UaError: No future object found for request: 0, callbacks in list are dict_keys([8612, 8613])
..
..
..
File "opcua\common\node.py", line 155, in get_value
File "opcua\common\node.py", line 164, in get_data_value
File "opcua\common\node.py", line 275, in get_attribute
File "opcua\client\ua_client.py", line 347, in read
File "opcua\client\ua_client.py", line 83, in send_request
File "concurrent\futures_base.py", line 443, in result
concurrent.futures._base.CancelledError
do you know if there is any modification to make to make it work in pyinstaller for comunication0?
(class with this module is call with a thread)
BadSecurityChecksFailed related to security, are you shure you are destribute the correct certificates? Or if you are using your client on another device, are you matching the correct hostname/ipaddress in the certificates?
I solved. when i created the .exe file with pyinstaller i copied the .der and .pem certificates (all in the same pc) but if i delete them and with python I recreate them automatically, the program works correctly, instead with old certificates, the connection is unstable.
[LATEST UPDATE] Thanks to Jack's enormous help!!! I managed to connect to the Cloud SQL postgres DB and read/write my dataframes to the database. However, I am still experiencing the same error that I experienced previously, which is...
struct.error: 'h' format requires -32768 <= number <= 32767
This error doesnt happen when the dataframes are small, compact and columns do not have too many NaN values in them. However, when there are many NaN values in the columns, the program throws the following error.
Separately I have tried using df = df.fillna(0) to fill the NaN values with 0. But it did not work as well, and the same error surfaced. Please help!
Traceback (most recent call last):
File "...\falcon_vbackup\STEP5_SavetoDB_and_SendEmail.py", line 81, in <module>
main_SavetoDB_and_SendEmail(
File "...\falcon_vbackup\STEP5_SavetoDB_and_SendEmail.py", line 37, in main_SavetoDB_and_SendEmail
Write_Dataframe_to_SQLTable(
File "...\falcon_vbackup\APPENDIX_Database_ReadWrite_v2.py", line 143, in Write_Dataframe_to_SQLTable
df_Output.to_sql(sql_tablename, con=conn, schema='public', index=False, if_exists=if_exists, method='multi', chunksize=1000)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\core\generic.py", line 2963, in to_sql
return sql.to_sql(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 697, in to_sql
return pandas_sql.to_sql(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 1739, in to_sql
total_inserted = sql_engine.insert_records(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 1322, in insert_records
return table.insert(chunksize=chunksize, method=method)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 950, in insert
num_inserted = exec_insert(conn, keys, chunk_iter)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 873, in _execute_insert_multi
result = conn.execute(stmt)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 1289, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\sql\elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 1481, in _execute_clauseelement
ret = self._execute_context(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 1845, in _execute_context
self._handle_dbapi_exception(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 2030, in _handle_dbapi_exception
util.raise_(exc_info[1], with_traceback=exc_info[2])
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\util\compat.py", line 207, in raise_
raise exception
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 1802, in _execute_context
self.dialect.do_execute(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pg8000\dbapi.py", line 455, in execute
self._context = self._c.execute_unnamed(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pg8000\core.py", line 627, in execute_unnamed
self.send_PARSE(NULL_BYTE, statement, oids)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pg8000\core.py", line 601, in send_PARSE
val.extend(h_pack(len(oids)))
struct.error: 'h' format requires -32768 <= number <= 32767
Exception ignored in: <function Connector.__del__ at 0x00000213190D8700>
Traceback (most recent call last):
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\google\cloud\sql\connector\connector.py", line 167, in __del__
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\concurrent\futures\_base.py", line 447, in result
concurrent.futures._base.TimeoutError:
I have setup a postgresql in GCP's Cloud SQL. I am trying to connect to it using
google.cloud.sql.connector. I have created a Service Account from the GCP Console, and downloaded the json keys.
I want to use a service account , credentials/ keys (in the format of reading a .json file placed in the same directory as my main.py code) to authenticate access to cloud_sql.
I am trying to authenticate, but I keep getting an error that says that the service account json file was not found.
Can anyone help to figure out how to fix this error? Thank you!
import pandas as pd
from google.cloud.sql.connector import connector
import os
import pandas as pd
import pandas as pd
import sqlalchemy
import os
# configure Cloud SQL Python Connector properties
def getconn():
conn = connector.connect(
os.environ['LL_DB_INSTANCE_CONNECTION_NAME'],
"pg8000",
user=os.environ['LL_DB_USER'],
password=os.environ['LL_DB_PASSWORD'],
db=os.environ['LL_DB_NAME'])
return conn
# Show existing SQLTables within database
def Show_SQLTables_in_Database(conn):
if conn!=None:
# Show what tables remain in database
results = conn.execute("""SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'""").fetchall()
for table in results:
print(table)
if __name__=="__main__":
# Set the Google Application Credentials as environment variable
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = os.path.join(os.getcwd(),"Google-Credentials-LL-tech2.json")
# create connection pool to re-use connections
pool = sqlalchemy.create_engine("postgresql+pg8000://", creator=getconn)
with pool.connect() as db_conn:
# Show what tables remain in database
results = db_conn.execute("""SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'""").fetchall()
for table in results:
print(table)
The error you are seeing means that the .json file is not being found. This is most likely being caused by os.getcwd() which gets the path of the current working directory from where main.py is being called. This leads to errors if you are calling the file from anywhere other than the parent directory.
Working case: python main.py
Error case: python folder/main.py
Change the line where you set credentials to the following:
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = os.path.join(os.path.dirname(os.path.abspath(__file__)),"Google-Credentials-LL-tech2.json")
This will allow the credentials path to be properly set for all cases of where your main.py is called from.
Responding to your latest update of the error.
First, make sure that your service account has the Cloud SQL Client role applied to it.
Secondly, try executing the following basic script prior to your custom configuration, this will help isolate the error to the Python Connector or the service account/implementation.
The following should just connect to your database and print the time.
from google.cloud.sql.connector import connector
import sqlalchemy
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = os.path.join(os.path.dirname(os.path.abspath(__file__)),"GSheet-Credentials-LL-tech2.json")
# build connection for db using Python Connector
def getconn():
conn = connector.connect(
os.environ['LL_DB_INSTANCE_CONNECTION_NAME'],
"pg8000",
user=os.environ['LL_DB_USER'],
password=os.environ['LL_DB_PASSWORD'],
db=os.environ['LL_DB_NAME'],
)
return conn
# create connection pool
pool = sqlalchemy.create_engine("postgresql+pg8000://", creator=getconn)
def db_connect():
with pool.connect() as conn:
current_time = conn.execute(
"SELECT NOW()").fetchone()
print(f"Time: {str(current_time[0])}")
db_connect()
If that still gives the error, please provide the full stacktrace of the error so that I can try and debug it further with more info.
I'm new to pybliometrics. I opened the python pybliometrics documentation and ran this example:
from pybliometrics.scopus import AffiliationSearch
query = "AFFIL(Max Planck Institute for Innovation and Competition Munich)"
s = AffiliationSearch(query)
print(s)
But it throws a pybliometrics.scopus.exception.Scopus401Error error.
Traceback (most recent call last):
File "C:/Users/Professional/PycharmProjects/firstScopus/main.py", line 3, in <module>
s = AffiliationSearch(query)
File "C:\Users\Professional\PycharmProjects\firstScopus\venv\lib\site-packages\pybliometrics\scopus\affiliation_search.py", line 106, in __init__
Search.__init__(self, query=query, api="AffiliationSearch",
File "C:\Users\Professional\PycharmProjects\firstScopus\venv\lib\site-packages\pybliometrics\scopus\superclasses\search.py", line 76, in __init__
Base.__init__(self, qfile, refresh, params=params, url=SEARCH_URL[api],
File "C:\Users\Professional\PycharmProjects\firstScopus\venv\lib\site-packages\pybliometrics\scopus\superclasses\base.py", line 70, in __init__
resp = get_content(url, params, *args, **kwds)
File "C:\Users\Professional\PycharmProjects\firstScopus\venv\lib\site-packages\pybliometrics\scopus\utils\get_content.py", line 88, in get_content
raise errors[resp.status_code](reason)
pybliometrics.scopus.exception.Scopus401Error
Process finished with exit code 1
As I understand it, I need to specify the API keys. How to do it?
You're right, according to https://pybliometrics.readthedocs.io/en/stable/access.html#error-messages 401 is most likely due to a missing key.
As the documentation on the configuration states, pybliometrics starts a process in the beginning to create the config. It asks for the API keys, and more. It seems you skipped this.
To manually create the config, go to your home directory (~/ on *nix systems) and create the folder .scopus (the dot is important). In there, create the file config.ini with a normal text editor. It needs to have at least those two sections, Directories and Authentication as shown on the documentation on the configuration.
Or simply reinstall pybliometrics and don't skip the set-up prompt in the beginning.
I am using the Anaconda 2.76 version. It was working fine until today. The notebook page was not loaded properly. Noe of the feature was responsive. After I did some research, I think it is some coding error, but since I am really not a computing kind of guy, I don't know where exactly went wrong and how to fix it. Below is the error message I received. please lend me a hand. Thanks a lot.
HTTPRequest (protocol=’http’, host =’127.0.0.1:8888;,method=’GET’,uri=’/static/base/images/favicon.ico’,version=’HTTP/1.1’,remote_ip-‘127.0.0.1’,headers={‘connection’:’keep-alive’,’Accept-Language’:’zh-CN,zh;q=0.8,en;q=0.6,zh-TW;q=0.4’,’Accept-Encoding’:’gzip,deflate,sdch’,'host’:'127.0.0.1:8888’,Accept':'*/*','User-Agent':'Mozilla/5.0(windows NT 6.1; WOW64)AppleWebKit/537.36(KHTML,like Gecko)Chrome/34.0.1847.131 Safari/537.36'})
Traceback (most recent call last):
File "D:|Anaconda\lib\site-packages\tornado\web.py", line 1218, in _when_complete
callback()
File "D:|Anaconda\lib\site-packages\tornado\web.py", Line 1239, in _execute_method
self._when_coplete(method(*self.path_args,**self.path_kwargs),
File "D:|Anaconda\lib\site-packages\IPython\html\base\handlers.py", line 318, in get
mime_type, encoding=mimetypes.guess_type(abspath)
File "D:\Anaconda\lib\mimetypes.py", line 297, in guess_type
init()
File "D:\Anaconda\lib\mimetypes.py", line 358,in init
db.read_windows_registry()
File "D:\Anaconda\lib\mimetypes.py", line 258,in read_windows_registry
for subkeyname in enum_types(hkcr):
File "D:\Anaconda\lib\mimetypes.py", line 249,in enum_types
ctype=ctype.encode(default_encoding)#omit in 3.X!
UnicodeDecodeError: "ascii" codec can't decode byte 0*b0 in position 1:ordinal not in range(128)
2014-5-12 16:43:45.456 [tornado.access] ERROR |500 GET /static/base/images/favicon.ico (127.0.0.1) 97.00ms`
This is a known issue.
I've solved the same problem using the following temporary modification of Anaconda/Lib/mimetypes.py, lines 252-253 (as proposed here).
try:
ctype = ctype.encode(default_encoding) # omit in 3.x!
except UnicodeEncodeError:
pass
except Exception: #<--
pass #<--
else:
yield ctype
My webpages are served by a script that dynamically imports a bunch of files with
try:
with open (filename, 'r') as f:
exec(f.read())
except IOError: pass
(actually, can you suggest a better method of importing a file? I'm sure there is one.)
Sometimes the files have strings in different languages, like
# contents of language.ru
title = "Название"
Those were all saved as UTF-8 files. Python has no problem running the script in command line or serving a page from my MacBook:
OK: [server command line] python3.0 page.py /index.ru
OK: http://whitebox.local/index.ru
but it throws an error when trying to serve a page from a server we just moved to:
157 try:
158 with open (filename, 'r') as f:
159 exec(f.read())
160 except IOError: pass
161
/usr/local/lib/python3.0/io.py in read(self=, n=-1)
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 627: ordinal not in range(128)
All the files were copied from my laptop where they were perfectly served by Apache. What is the reason?
Update: I found out the default encoding for open() is platform-dependent so it was utf8 on my laptop and ascii on server. I wonder if there is a per-program function to set it in Python 3 (sys.setdefaultencoding is used in site module and then deleted from the namespace).
Use open(filename, 'r', encoding='utf8').
See Python 3 docs for open.
Use codecs library, I'm using python 2.6.6 and I do not use the usual open with encoding argument:
import codecs
codecs.open('filename','r',encoding='UTF-8')
You can use something like
with open(fname, 'r', encoding="ascii", errors="surrogateescape") as f:
data = f.read()
# make changes to the string 'data'
with open(fname + '.new', 'w',
encoding="ascii", errors="surrogateescape") as f:
f.write(data)
more information is on python unicode documents