Web2py - Auth with MongoDB - mongodb

Good day,
I'm trying to use MongoDB with web2py, and for that I started with authentication, but this appeared some errors that I do not understand.
In a relational database, the web2py creates the authentication tables, MongoDB in the collections are not created automatically.
Below is the code and the error when trying to log me:
db.py
db = DAL("mongodb://localhost/primer", check_reserved=["mongodb_nonreserved",], adapter_args={"safe":False})
from gluon.tools import Auth, Service, PluginManager
auth = Auth(db)
service = Service()
plugins = PluginManager()
auth.settings.remember_me_form = False
auth.settings.actions_disabled=['register','change_password','request_reset_password','retrieve_username','profile']
auth.define_tables(username=True)
from gluon.contrib.login_methods.ldap_auth import ldap_auth
auth.settings.login_methods = [ldap_auth(server='localhost', port='10389', base_dn='ou=people,o=empresa,dc=com,dc=br')]
The authentication is by LDAP, and works perfectly in a relational database, which has the AUTH_USER table.
However, the loging using MongoDB, this appearing the following error:
Traceback (most recent call last):
File "C:\Users\Rafa\Desktop\web2py-10-06-2015p4\applications\contrato\controllers/appadmin.py", line 249, in select
nrows = db(query, ignore_common_filters=True).count()
File "C:\Users\Rafa\Desktop\web2py-10-06-2015p4\gluon\packages\dal\pydal\objects.py", line 2016, in count
return db._adapter.count(self.query,distinct)
File "C:\Users\Rafa\Desktop\web2py-10-06-2015p4\gluon\packages\dal\pydal\adapters\mongo.py", line 200, in count
count=True,snapshot=snapshot)['count'])
File "C:\Users\Rafa\Desktop\web2py-10-06-2015p4\gluon\packages\dal\pydal\adapters\mongo.py", line 319, in select
sort=mongosort_list, snapshot=snapshot).count()}
File "C:\Python27\lib\site-packages\pymongo\collection.py", line 929, in find
return Cursor(self, *args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'snapshot'
The database "primer" is created and only has two collections "posts" and "system.indexes"
Could someone help me with this error to be able to use MongoDB with the web2py?
Thank You!

Found it.
From pymongo's changelog There are a lot of breaking changes in pymongo 3.0 compared to 2.8
The following find/find_one options have been removed:
snapshot (use the new modifiers option instead)
So uninstall pymongo and try the latest before 3.0:
pip install pymongo==2.8.1
Here's my attempt:
>>> from pydal import *
No handlers could be found for logger "web2py"
>>> db = DAL('mongodb://localhost/connect_test')
>>> db.define_table('some',Field('key'),Field('value'))
<Table some (id,key,value)>
>>> db.define_table('some2',Field('ref','reference some'),Field('value'))
<Table some2 (id,ref,value)>
>>> db(db.some).select()
<Rows (1)>
>>> db(db.some).select().first()
<Row {'value': 'pir', 'key': 'bla', 'id': 26563964102769618087622556519L}>
>>>
[edit]
There's more to it. This worked at least with pydal 15.03. Googling some code i found the following in the mongo.py adapter :
from pymongo import version
if 'fake_version' in driver_args:
version = driver_args['fake_version']
if int(version.split('.')[0]) < 3:
raise Exception(
"pydal requires pymongo version >= 3.0, found '%s'"
% version)
Which was like good soil for a big frown...
After updating pydal to 15.07 it apears to brake indeed:
RuntimeError: Failure to connect, tried 5 times:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\pydal\base.py", line 437, in __init__
self._adapter = ADAPTERS[self._dbname](**kwargs)
File "C:\Python27\lib\site-packages\pydal\adapters\base.py", line 57, in __call__
obj = super(AdapterMeta, cls).__call__(*args, **kwargs)
File "C:\Python27\lib\site-packages\pydal\adapters\mongo.py", line 82, in __init__
% version)
Exception: pydal requires pymongo version >= 3.0, found '2.8.1'
So it's back to upgrading pymongo :)
With pymongo at 3.0.3 and pydal at 15.07 it works like a charm again.

Related

Error from Google Authentication in cloud sql connection

[LATEST UPDATE] Thanks to Jack's enormous help!!! I managed to connect to the Cloud SQL postgres DB and read/write my dataframes to the database. However, I am still experiencing the same error that I experienced previously, which is...
struct.error: 'h' format requires -32768 <= number <= 32767
This error doesnt happen when the dataframes are small, compact and columns do not have too many NaN values in them. However, when there are many NaN values in the columns, the program throws the following error.
Separately I have tried using df = df.fillna(0) to fill the NaN values with 0. But it did not work as well, and the same error surfaced. Please help!
Traceback (most recent call last):
File "...\falcon_vbackup\STEP5_SavetoDB_and_SendEmail.py", line 81, in <module>
main_SavetoDB_and_SendEmail(
File "...\falcon_vbackup\STEP5_SavetoDB_and_SendEmail.py", line 37, in main_SavetoDB_and_SendEmail
Write_Dataframe_to_SQLTable(
File "...\falcon_vbackup\APPENDIX_Database_ReadWrite_v2.py", line 143, in Write_Dataframe_to_SQLTable
df_Output.to_sql(sql_tablename, con=conn, schema='public', index=False, if_exists=if_exists, method='multi', chunksize=1000)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\core\generic.py", line 2963, in to_sql
return sql.to_sql(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 697, in to_sql
return pandas_sql.to_sql(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 1739, in to_sql
total_inserted = sql_engine.insert_records(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 1322, in insert_records
return table.insert(chunksize=chunksize, method=method)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 950, in insert
num_inserted = exec_insert(conn, keys, chunk_iter)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pandas\io\sql.py", line 873, in _execute_insert_multi
result = conn.execute(stmt)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 1289, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\sql\elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 1481, in _execute_clauseelement
ret = self._execute_context(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 1845, in _execute_context
self._handle_dbapi_exception(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 2030, in _handle_dbapi_exception
util.raise_(exc_info[1], with_traceback=exc_info[2])
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\util\compat.py", line 207, in raise_
raise exception
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\base.py", line 1802, in _execute_context
self.dialect.do_execute(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\sqlalchemy\engine\default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pg8000\dbapi.py", line 455, in execute
self._context = self._c.execute_unnamed(
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pg8000\core.py", line 627, in execute_unnamed
self.send_PARSE(NULL_BYTE, statement, oids)
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\pg8000\core.py", line 601, in send_PARSE
val.extend(h_pack(len(oids)))
struct.error: 'h' format requires -32768 <= number <= 32767
Exception ignored in: <function Connector.__del__ at 0x00000213190D8700>
Traceback (most recent call last):
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\site-packages\google\cloud\sql\connector\connector.py", line 167, in __del__
File "c:\Users\ng_yj\.conda\envs\venv_falcon\lib\concurrent\futures\_base.py", line 447, in result
concurrent.futures._base.TimeoutError:
I have setup a postgresql in GCP's Cloud SQL. I am trying to connect to it using
google.cloud.sql.connector. I have created a Service Account from the GCP Console, and downloaded the json keys.
I want to use a service account , credentials/ keys (in the format of reading a .json file placed in the same directory as my main.py code) to authenticate access to cloud_sql.
I am trying to authenticate, but I keep getting an error that says that the service account json file was not found.
Can anyone help to figure out how to fix this error? Thank you!
import pandas as pd
from google.cloud.sql.connector import connector
import os
import pandas as pd
import pandas as pd
import sqlalchemy
import os
# configure Cloud SQL Python Connector properties
def getconn():
conn = connector.connect(
os.environ['LL_DB_INSTANCE_CONNECTION_NAME'],
"pg8000",
user=os.environ['LL_DB_USER'],
password=os.environ['LL_DB_PASSWORD'],
db=os.environ['LL_DB_NAME'])
return conn
# Show existing SQLTables within database
def Show_SQLTables_in_Database(conn):
if conn!=None:
# Show what tables remain in database
results = conn.execute("""SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'""").fetchall()
for table in results:
print(table)
if __name__=="__main__":
# Set the Google Application Credentials as environment variable
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = os.path.join(os.getcwd(),"Google-Credentials-LL-tech2.json")
# create connection pool to re-use connections
pool = sqlalchemy.create_engine("postgresql+pg8000://", creator=getconn)
with pool.connect() as db_conn:
# Show what tables remain in database
results = db_conn.execute("""SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'""").fetchall()
for table in results:
print(table)
The error you are seeing means that the .json file is not being found. This is most likely being caused by os.getcwd() which gets the path of the current working directory from where main.py is being called. This leads to errors if you are calling the file from anywhere other than the parent directory.
Working case: python main.py
Error case: python folder/main.py
Change the line where you set credentials to the following:
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = os.path.join(os.path.dirname(os.path.abspath(__file__)),"Google-Credentials-LL-tech2.json")
This will allow the credentials path to be properly set for all cases of where your main.py is called from.
Responding to your latest update of the error.
First, make sure that your service account has the Cloud SQL Client role applied to it.
Secondly, try executing the following basic script prior to your custom configuration, this will help isolate the error to the Python Connector or the service account/implementation.
The following should just connect to your database and print the time.
from google.cloud.sql.connector import connector
import sqlalchemy
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = os.path.join(os.path.dirname(os.path.abspath(__file__)),"GSheet-Credentials-LL-tech2.json")
# build connection for db using Python Connector
def getconn():
conn = connector.connect(
os.environ['LL_DB_INSTANCE_CONNECTION_NAME'],
"pg8000",
user=os.environ['LL_DB_USER'],
password=os.environ['LL_DB_PASSWORD'],
db=os.environ['LL_DB_NAME'],
)
return conn
# create connection pool
pool = sqlalchemy.create_engine("postgresql+pg8000://", creator=getconn)
def db_connect():
with pool.connect() as conn:
current_time = conn.execute(
"SELECT NOW()").fetchone()
print(f"Time: {str(current_time[0])}")
db_connect()
If that still gives the error, please provide the full stacktrace of the error so that I can try and debug it further with more info.

Web2py Access Control & MongoDB

I'm using MongoDB for my database.
db = DAL('mongodb://localhost:27017/app')
When I attempt to register a new user, I get the following error:
Traceback (most recent call last):
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/restricted.py", line 209, in restricted
exec ccode in environment
File "/home/ecosystem/web2py/applications/app/controllers/default.py", line 146, in <module>
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/globals.py", line 187, in <lambda>
self._caller = lambda f: f()
File "/home/ecosystem/web2py/applications/app/controllers/default.py", line 108, in user
return dict(form=auth())
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/tools.py", line 1205, in __call__
return getattr(self,args[0])()
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/tools.py", line 2180, in register
self.add_membership(group_id, form.vars.id)
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/tools.py", line 2963, in add_membership
record = membership(user_id = user_id,group_id = group_id)
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/dal.py", line 7656, in __call__
else:
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/dal.py", line 8787, in select
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/gluon/dal.py", line 5092, in select
row=[]
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1058, in next
if len(self.__data) or self._refresh():
File "/home/ecosystem/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1002, in _refresh
self.__uuid_subtype))
OverflowError: MongoDB can only handle up to 8-byte ints
If I look in my database tables, everything seems okay except a couple fields.
In auth_user (not sure about this one):
registration_id:""
In auth_event (I'm assuming this should not be null):
user_id:null
What is the best way to resolve this? If possible, I'd like to use MongoDB for all database transactions (including access control / authentication).
UPDATE: You can avoid this by upgrading to the latest version of Web2py. This is now fully supported so you do not need to use two different databases.
I wasn't able to solve this problem directly, but I was able to create a work-around. In db.py, I define two separate database connections:
db = DAL('sqlite://storage.sqlite',pool_size=1,check_reserved=['all'])
db1 = DAL('mongodb://localhost:27017/collection')
I call db (eventually a mysql database) on all auth functions (registration, login, is logged in checks), while handling everything else (including detailed member profiles) in mongo. I often bypass db1 DAL entirely by using the pymongo API.

django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet. tastypie

Continuing my search for an answer to get oauth2.0 to work on pythonanywhere.
i am following this tutorial: http://ianalexandr.com/blog/building-a-true-oauth-20-api-with-django-and-tasty-pie.html
im using django 1.6 : https://www.pythonanywhere.com/wiki/VirtualEnvForNewerDjango
when i get to this line of codes:
from provider.oauth2.models import Client
# from django.contrib.auth.models import User
from django.contrib.auth import get_user_model
User = get_user_model()
u = User.objects.get(id=1)
c = Client(user=u, name="mysite client", client_type=1, url="http://pythonx00x.pythonanywhere.com")
c.save()
c.client_id
'd63f53a7a6cceba04db5'
c.client_secret
'afe899288b9ac4127d57f2f12ac5a49d839364dc'
it seems that i got an error at line:
User = get_user_model()
and it raise an error:
raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
here is the full stack trace:
Traceback (most recent call last):
File "addClient.py", line 9, in <module>
User = get_user_model()
File "/home/python2006/.virtualenvs/django16/local/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 136, in get_user_model
return django_apps.get_model(settings.AUTH_USER_MODEL)
File "/home/python2006/.virtualenvs/django16/local/lib/python2.7/site-packages/django/apps/registry.py", line 200, in get_model
self.check_models_ready()
File "/home/python2006/.virtualenvs/django16/local/lib/python2.7/site-packages/django/apps/registry.py", line 132, in check_models_ready
raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
I can't seem to find out how to get the models load.
if I'm getting the idea right.
I think you may not be using the version of Django that you think you are. AppRegistryNotReady was introduced in Django 1.7. I would guess that, if you pinned your DJango version to 1.6, your code would work.

Iron Python and PyMongo Error

I'm getting this error when I try to connect to mongodb (using pymongo) with Iron Python...
Traceback (most recent call last):
File "test.py", line 3, in <module>
File "c:\Program Files (x86)\IronPython 2.7\lib\site-packages\pymongo\connecti
on.py", line 179, in __init__
File "c:\Program Files (x86)\IronPython 2.7\lib\site-packages\pymongo\mongo_cl
ient.py", line 269, in __init__
pymongo.errors.ConnectionFailure: Specified cast is not valid.
Code is pretty simple, I have replaced the db name.
import pymongo
c = pymongo.Connection('mongodb://testuser:test123#linus.mongohq.com:10021/sometestdb')
It works fine with regular python. Any ideas?
Ironpython isn't supported by pymongo - so I wouldn't advise trying to use it. You can see on the pypi page a list of supported implementations: http://pypi.python.org/pypi/pymongo
Please also see the answer here: Working with PTVS, IronPython and MongoDB
You may not be able to use pymongo with IronPython, but you can use the C#/.NET driver for MongoDB from IronPython.
Information on the driver is here. As explained in this link, you can install with nuget (PM> Install-Package mongocsharpdriver), or just download the dlls.
Once installed, you can use the assemblies in the normal way in IronPython:
# Add reference to the Mongo C# driver
import clr
clr.AddReferenceToFileAndPath("MongoDB.Bson.dll")
clr.AddReferenceToFileAndPath("MongoDB.Driver.dll")
Then use according to the MongoDB C# Driver API, for example:
# Get the MongoDB database
from MongoDB.Driver import MongoClient
client = MongoClient("mongodb://localhost")
server = client.GetServer()
database = server.GetDatabase("test")
# Get a collection
collection = database.GetCollection("users")
# Add a document
from MongoDB.Bson import BsonDocument
user = BsonDocument({'first_name':'John', 'last_name':'Smith'})
collection.Insert(user)
See the the MongoDB C# Driver API for more information.

Bottle web framework: any way to run a console/shell and get it to work with Werkzeug?

I searched but couldn't find an easy way to run a console or shell akin to Django's manage.py shell or Rail's rails console
Since I just started using Bottle for an existing project, I just wanted to play around with the existing models and managers in the console. The closest I came up with was using ipdb's set_trace() and go from there, but that's not ideal by any means.
Also, I tried integrating Bottle with Werkzeug, but when I follow the instructions:
import bottle
app = bottle.Bottle()
werkzeug = bottle.ext.werkzeug.Plugin()
app.install(werkzeug)
I get the following traceback error:
Traceback (most recent call last):
File "mysite.py", line 62, in <module>
werkzeug = bottle.ext.werkzeug.Plugin()
AttributeError: 'module' object has no attribute 'werkzeug'
Try adding importing bottle.ext.werkzeug by adding this at the beginning of your source:
import bottle.ext.werkzeug