I'm using MongoDB for my database.
db = DAL('mongodb://localhost:27017/app')
When I attempt to register a new user, I get the following error:
Traceback (most recent call last):
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/restricted.py", line 209, in restricted
exec ccode in environment
File "/home/ecosystem/web2py/applications/app/controllers/default.py", line 146, in <module>
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/globals.py", line 187, in <lambda>
self._caller = lambda f: f()
File "/home/ecosystem/web2py/applications/app/controllers/default.py", line 108, in user
return dict(form=auth())
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/tools.py", line 1205, in __call__
return getattr(self,args[0])()
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/tools.py", line 2180, in register
self.add_membership(group_id, form.vars.id)
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/tools.py", line 2963, in add_membership
record = membership(user_id = user_id,group_id = group_id)
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/dal.py", line 7656, in __call__
else:
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/dal.py", line 8787, in select
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/dal.py", line 5092, in select
row=[]
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1058, in next
if len(self.__data) or self._refresh():
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1002, in _refresh
self.__uuid_subtype))
OverflowError: MongoDB can only handle up to 8-byte ints
If I look in my database tables, everything seems okay except a couple fields.
In auth_user (not sure about this one):
registration_id:""
In auth_event (I'm assuming this should not be null):
user_id:null
What is the best way to resolve this? If possible, I'd like to use MongoDB for all database transactions (including access control / authentication).
UPDATE: You can avoid this by upgrading to the latest version of Web2py. This is now fully supported so you do not need to use two different databases.
I wasn't able to solve this problem directly, but I was able to create a work-around. In db.py, I define two separate database connections:
db = DAL('sqlite://storage.sqlite',pool_size=1,check_reserved=['all'])
db1 = DAL('mongodb://localhost:27017/collection')
I call db (eventually a mysql database) on all auth functions (registration, login, is logged in checks), while handling everything else (including detailed member profiles) in mongo. I often bypass db1 DAL entirely by using the pymongo API.
Related
[LATEST UPDATE] Thanks to Jack's enormous help!!! I managed to connect to the Cloud SQL postgres DB and read/write my dataframes to the database. However, I am still experiencing the same error that I experienced previously, which is...
struct.error: 'h' format requires -32768 <= number <= 32767
This error doesnt happen when the dataframes are small, compact and columns do not have too many NaN values in them. However, when there are many NaN values in the columns, the program throws the following error.
Separately I have tried using df = df.fillna(0) to fill the NaN values with 0. But it did not work as well, and the same error surfaced. Please help!
Traceback (most recent call last):
File "...\falcon_vbackup\STEP5_SavetoDB_and_SendEmail.py", line 81, in <module>
main_SavetoDB_and_SendEmail(
File "...\falcon_vbackup\STEP5_SavetoDB_and_SendEmail.py", line 37, in main_SavetoDB_and_SendEmail
Write_Dataframe_to_SQLTable(
File "...\falcon_vbackup\APPENDIX_Database_ReadWrite_v2.py", line 143, in Write_Dataframe_to_SQLTable
df_Output.to_sql(sql_tablename, con=conn, schema='public', index=False, if_exists=if_exists, method='multi', chunksize=1000)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\core\generic.py", line 2963, in to_sql
return sql.to_sql(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 697, in to_sql
return pandas_sql.to_sql(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 1739, in to_sql
total_inserted = sql_engine.insert_records(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 1322, in insert_records
return table.insert(chunksize=chunksize, method=method)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 950, in insert
num_inserted = exec_insert(conn, keys, chunk_iter)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 873, in _execute_insert_multi
result = conn.execute(stmt)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 1289, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\sql\elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 1481, in _execute_clauseelement
ret = self._execute_context(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 1845, in _execute_context
self._handle_dbapi_exception(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 2030, in _handle_dbapi_exception
util.raise_(exc_info[1], with_traceback=exc_info[2])
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\util\compat.py", line 207, in raise_
raise exception
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 1802, in _execute_context
self.dialect.do_execute(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pg8000\dbapi.py", line 455, in execute
self._context = self._c.execute_unnamed(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pg8000\core.py", line 627, in execute_unnamed
self.send_PARSE(NULL_BYTE, statement, oids)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pg8000\core.py", line 601, in send_PARSE
val.extend(h_pack(len(oids)))
struct.error: 'h' format requires -32768 <= number <= 32767
Exception ignored in: <function Connector.__del__ at 0x00000213190D8700>
Traceback (most recent call last):
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\google\cloud\sql\connector\connector.py", line 167, in __del__
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\concurrent\futures\_base.py", line 447, in result
concurrent.futures._base.TimeoutError:
I have setup a postgresql in GCP's Cloud SQL. I am trying to connect to it using
google.cloud.sql.connector. I have created a Service Account from the GCP Console, and downloaded the json keys.
I want to use a service account , credentials/ keys (in the format of reading a .json file placed in the same directory as my main.py code) to authenticate access to cloud_sql.
I am trying to authenticate, but I keep getting an error that says that the service account json file was not found.
Can anyone help to figure out how to fix this error? Thank you!
import pandas as pd
from google.cloud.sql.connector import connector
import os
import pandas as pd
import pandas as pd
import sqlalchemy
import os
# configure Cloud SQL Python Connector properties
def getconn():
conn = connector.connect(
os.environ['LL_DB_INSTANCE_CONNECTION_NAME'],
"pg8000",
user=os.environ['LL_DB_USER'],
password=os.environ['LL_DB_PASSWORD'],
db=os.environ['LL_DB_NAME'])
return conn
# Show existing SQLTables within database
def Show_SQLTables_in_Database(conn):
if conn!=None:
# Show what tables remain in database
results = conn.execute("""SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'""").fetchall()
for table in results:
print(table)
if __name__=="__main__":
# Set the Google Application Credentials as environment variable
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = os.path.join(os.getcwd(),"Google-Credentials-LL-tech2.json")
# create connection pool to re-use connections
pool = sqlalchemy.create_engine("postgresql+pg8000://", creator=getconn)
with pool.connect() as db_conn:
# Show what tables remain in database
results = db_conn.execute("""SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'""").fetchall()
for table in results:
print(table)
The error you are seeing means that the .json file is not being found. This is most likely being caused by os.getcwd() which gets the path of the current working directory from where main.py is being called. This leads to errors if you are calling the file from anywhere other than the parent directory.
Working case: python main.py
Error case: python folder/main.py
Change the line where you set credentials to the following:
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = os.path.join(os.path.dirname(os.path.abspath(__file__)),"Google-Credentials-LL-tech2.json")
This will allow the credentials path to be properly set for all cases of where your main.py is called from.
Responding to your latest update of the error.
First, make sure that your service account has the Cloud SQL Client role applied to it.
Secondly, try executing the following basic script prior to your custom configuration, this will help isolate the error to the Python Connector or the service account/implementation.
The following should just connect to your database and print the time.
from google.cloud.sql.connector import connector
import sqlalchemy
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = os.path.join(os.path.dirname(os.path.abspath(__file__)),"GSheet-Credentials-LL-tech2.json")
# build connection for db using Python Connector
def getconn():
conn = connector.connect(
os.environ['LL_DB_INSTANCE_CONNECTION_NAME'],
"pg8000",
user=os.environ['LL_DB_USER'],
password=os.environ['LL_DB_PASSWORD'],
db=os.environ['LL_DB_NAME'],
)
return conn
# create connection pool
pool = sqlalchemy.create_engine("postgresql+pg8000://", creator=getconn)
def db_connect():
with pool.connect() as conn:
current_time = conn.execute(
"SELECT NOW()").fetchone()
print(f"Time: {str(current_time[0])}")
db_connect()
If that still gives the error, please provide the full stacktrace of the error so that I can try and debug it further with more info.
I'm new to pybliometrics. I opened the python pybliometrics documentation and ran this example:
from pybliometrics.scopus import AffiliationSearch
query = "AFFIL(Max Planck Institute for Innovation and Competition Munich)"
s = AffiliationSearch(query)
print(s)
But it throws a pybliometrics.scopus.exception.Scopus401Error error.
Traceback (most recent call last):
File "C:/Users/Professional/PycharmProjects/firstScopus/main.py", line 3, in <module>
s = AffiliationSearch(query)
File "C:\Users\Professional\PycharmProjects\firstScopus\venv\lib\site-packages\pybliometrics\scopus\affiliation_search.py", line 106, in __init__
Search.__init__(self, query=query, api="AffiliationSearch",
File "C:\Users\Professional\PycharmProjects\firstScopus\venv\lib\site-packages\pybliometrics\scopus\superclasses\search.py", line 76, in __init__
Base.__init__(self, qfile, refresh, params=params, url=SEARCH_URL[api],
File "C:\Users\Professional\PycharmProjects\firstScopus\venv\lib\site-packages\pybliometrics\scopus\superclasses\base.py", line 70, in __init__
resp = get_content(url, params, *args, **kwds)
File "C:\Users\Professional\PycharmProjects\firstScopus\venv\lib\site-packages\pybliometrics\scopus\utils\get_content.py", line 88, in get_content
raise errors[resp.status_code](reason)
pybliometrics.scopus.exception.Scopus401Error
Process finished with exit code 1
As I understand it, I need to specify the API keys. How to do it?
You're right, according to https://pybliometrics.readthedocs.io/en/stable/access.html#error-messages 401 is most likely due to a missing key.
As the documentation on the configuration states, pybliometrics starts a process in the beginning to create the config. It asks for the API keys, and more. It seems you skipped this.
To manually create the config, go to your home directory (~/ on *nix systems) and create the folder .scopus (the dot is important). In there, create the file config.ini with a normal text editor. It needs to have at least those two sections, Directories and Authentication as shown on the documentation on the configuration.
Or simply reinstall pybliometrics and don't skip the set-up prompt in the beginning.
Admittedly, I am fairly new to postgresql and pgadmin, but after spending a day learning it, with pgadmin4.4.1 I was able to backup tables and restore them in various different formats, currently, the format of choice is plain text, so that I can actually read the backup file to see what it is doing and also my database is tiny, but large enough to not want to type in every entry by hand.
Since updating to pgadmin4.4.2, the restore function does not work. I provides a green box saying restore job created, yet no data is added to the database. This was not the case yesterday, it would just restore the data.
Does anyone know what defaults might possibly have changed in the update that will be preventing my restore job from actually running? I am assuming that it is created and waiting in a queue somewhere that I can't find in order to actually run the backup.
Here is the last error from the log file. Apologies for the delay in replying.
019-02-08 17:59:06,942: ERROR pgadmin: 'NoneType' object has no attribute 'replace'
Traceback (most recent call last):
File "/Applications/pgAdmin 4.app/Contents/Resources/venv/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/Applications/pgAdmin 4.app/Contents/Resources/venv/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Applications/pgAdmin 4.app/Contents/Resources/venv/lib/python3.6/site-packages/flask_login.py", line 792, in decorated_view
return func(*args, **kwargs)
File "/Applications/pgAdmin 4.app/Contents/Resources/web/pgadmin/misc/bgprocess/__init__.py", line 61, in index
return make_response(response=BatchProcess.list())
File "/Applications/pgAdmin 4.app/Contents/Resources/web/pgadmin/misc/bgprocess/processes.py", line 610, in list
desc = desc.message
File "/Applications/pgAdmin 4.app/Contents/Resources/web/pgadmin/tools/backup/__init__.py", line 145, in message
port = html.safe_str(port)
File "/Applications/pgAdmin 4.app/Contents/Resources/web/pgadmin/utils/html.py", line 35, in safe_str
return cgi.escape(x)
File "/Applications/pgAdmin 4.app/Contents/Resources/venv/lib/python3.6/cgi.py", line 1056, in escape
s = s.replace("&", "&") # Must be done first!
AttributeError: 'NoneType' object has no attribute 'replace'
I was using PyMongo without any transaction and sessions before and was inserting documents successfully as:
try:
_ = db[collection].insert_many(dataset, ordered=False)
except:
err = filter(lambda x: x['code'] != 11000, e.details['writeErrors'])
if len(err) > 0:
raise
The code above was successfully ignoring the errors about duplicate keys, which is what I wanted.
Now, I upgraded to MongoDB 4.0 and tried the new transactions API and tried to do this in a session as:
def do_insert(db, dataset, session):
try:
_ = db[collection].insert_many(dataset, ordered=False, session=session)
except pymongo.errors.DuplicateKeyError as e:
pass
However, the operation also generates an OperationFailure error and I get something like:
ERROR: test_insert_duplicate_categories (__main__.TestDefaultAnnotations)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/xargon/Dropbox/infermatica/code/alchera/altrack/altrack/tests/test_mongodb_default.py", line 152, in test_insert_duplicate_categories
insert_dataset(db, ds)
File "/Users/xargon/Dropbox/infermatica/code/alchera/altrack/altrack/data/default.py", line 269, in insert_dataset
session.commit_transaction()
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/client_session.py", line 393, in commit_transaction
self._finish_transaction_with_retry("commitTransaction")
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/client_session.py", line 457, in _finish_transaction_with_retry
return self._finish_transaction(command_name)
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/client_session.py", line 452, in _finish_transaction
parse_write_concern_error=True)
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/database.py", line 514, in _command
client=self.__client)
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/pool.py", line 579, in command
unacknowledged=unacknowledged)
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/network.py", line 150, in command
parse_write_concern_error=parse_write_concern_error)
File "/Users/xargon/anaconda/envs/deep/lib/python3.6/site-packages/pymongo/helpers.py", line 155, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Transaction 1 has been aborted.
The call is as:
with db.client.start_session() as session:
try:
session.start_transaction()
do_insert(db, dataset, session)
session.commit_transaction()
except Exception as e:
session.abort_transaction()
raise
How can I ignore this duplicate key error in a transactional setting? The problem is that even though I ignore the duplicate key exception, the transaction now seems to be in an inconsistent state. So when I commit, it throws up that exception.
So, my use case is that I can have users trying to insert duplicates and the database should silently ignore the insert if a record already exists
How can I ignore this duplicate key error in a transactional setting?
As you're aware, as of current stable version of MongoDB (v4.0) a DuplicatedKey error will abort transactions.
This is because DuplicatedKey is only checked after the data write, and the convention is abort the WiredTiger Storage Engine transaction.
This applies to any index constraints will result in the same issue, e.g. inserting invalid location format into geo index.
I upgraded to MongoDB 4.0 and tried the new transactions API and tried to do this in a session.
Please note that upgrading to MongoDB 4.0, doesn't mean that you have to also update your code to use multi-document transactions. Only if the use case requires atomicity for updates to multiple documents or consistency between reads to multiple documents.
I'm trying to use mongoDB with cuckoo but i get this error message :
2016-12-16 06:58:01,632 [lib.cuckoo.core.plugins] ERROR: Failed to run the reporting module "MongoDB":
Traceback (most recent call last):
File "/home/ziv/Documents/cuckoo/lib/cuckoo/core/plugins.py", line 533, in process
current.run(self.results)
File "/home/ziv/Documents/cuckoo/modules/reporting/mongodb.py", line 89, in run
if "cuckoo_schema" in self.db.collection_names():
File "/usr/lib/python2.7/dist-packages/pymongo/database.py", line 520, in collection_names
results = self._list_collections(sock_info, slave_okay)
File "/usr/lib/python2.7/dist-packages/pymongo/database.py", line 492, in _list_collections
cursor = self._command(sock_info, cmd, slave_okay)["cursor"]
File "/usr/lib/python2.7/dist-packages/pymongo/database.py", line 393, in _command
allowable_errors)
File "/usr/lib/python2.7/dist-packages/pymongo/pool.py", line 211, in command
read_concern)
File "/usr/lib/python2.7/dist-packages/pymongo/network.py", line 100, in command
helpers._check_command_response(response_doc, msg, allowable_errors)
File "/usr/lib/python2.7/dist-packages/pymongo/helpers.py", line 196, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
OperationFailure: command SON([('listCollections', 1), ('cursor', {})]) on namespace cuckoo.$cmd failed: not authorized on cuckoo to execute command { listCollections: 1, cursor: {} }
this is the DB's i have:
show dbs
admin 0.078GB
cuckoo 0.078GB
local 0.078GB
i used this guide to install mongo db https://www.howtoforge.com/tutorial/install-mongodb-on-ubuntu-16.04/
i used this guide to install cuckoo
http://mostlyaboutsecurity.com/?p=15&i=1
update
ithink i dont have permissions but i dont know how to set up what i need,
this is the cuckoo code that uses mongo DB:
(on this line "self.db.collection_names():")
def run(self, results):
"""Writes report.
#param results: analysis results dictionary.
#raise CuckooReportError: if fails to connect or write to MongoDB.
"""
if not HAVE_MONGO:
raise CuckooDependencyError(
"Unable to import pymongo (install with "
"`pip install pymongo`)"
)
self.connect()
# Set mongo schema version.
# TODO: This is not optimal becuase it run each analysis. Need to run
# only one time at startup.
if "cuckoo_schema" in self.db.collection_names():
if self.db.cuckoo_schema.find_one()["version"] != self.SCHEMA_VERSION:
CuckooReportError("Mongo schema version not expected, check data migration tool")
else:
self.db.cuckoo_schema.save({"version": self.SCHEMA_VERSION})
def connect(self):
"""Connects to Mongo database, loads options and set connectors.
#raise CuckooReportError: if unable to connect.
"""
host = self.options.get("host", "127.0.0.1")
port = int(self.options.get("port", 27017))
db = self.options.get("db", "cuckoo")
try:
self.conn = MongoClient(host, port)
self.db = self.conn[db]
self.fs = GridFS(self.db)
except TypeError:
raise CuckooReportError("Mongo connection port must be integer")
except ConnectionFailure:
raise CuckooReportError("Cannot connect to MongoDB")
I don't want to edit this code (add connection string to it)
I have a clean installation of MongoDB, how do i create a DB named cuckoo
that this code can acsess and use?
I couldn't find any reference in all the guides I read. its like it should work automaticly but it doesn't
The error message is:
OperationFailure: command SON([('listCollections', 1), ('cursor', {})]) on namespace cuckoo.$cmd failed: not authorized on cuckoo to execute command { listCollections: 1, cursor: {} }
This indicates that your app is attempting to execute a command in the MongoDB database for which it does not have permissions.
Does your connection string to the database include the authentication credentials (username/password)?
Does this user have necessary permissions to execute this command?