Iron Python and PyMongo Error - mongodb

I'm getting this error when I try to connect to mongodb (using pymongo) with Iron Python...
Traceback (most recent call last):
File "test.py", line 3, in <module>
File "c:\Program Files (x86)\IronPython 2.7\lib\site-packages\pymongo\connecti
on.py", line 179, in __init__
File "c:\Program Files (x86)\IronPython 2.7\lib\site-packages\pymongo\mongo_cl
ient.py", line 269, in __init__
pymongo.errors.ConnectionFailure: Specified cast is not valid.
Code is pretty simple, I have replaced the db name.
import pymongo
c = pymongo.Connection('mongodb://testuser:test123#linus.mongohq.com:10021/sometestdb')
It works fine with regular python. Any ideas?

Ironpython isn't supported by pymongo - so I wouldn't advise trying to use it. You can see on the pypi page a list of supported implementations: http://pypi.python.org/pypi/pymongo

Please also see the answer here: Working with PTVS, IronPython and MongoDB
You may not be able to use pymongo with IronPython, but you can use the C#/.NET driver for MongoDB from IronPython.
Information on the driver is here. As explained in this link, you can install with nuget (PM> Install-Package mongocsharpdriver), or just download the dlls.
Once installed, you can use the assemblies in the normal way in IronPython:
# Add reference to the Mongo C# driver
import clr
clr.AddReferenceToFileAndPath("MongoDB.Bson.dll")
clr.AddReferenceToFileAndPath("MongoDB.Driver.dll")
Then use according to the MongoDB C# Driver API, for example:
# Get the MongoDB database
from MongoDB.Driver import MongoClient
client = MongoClient("mongodb://localhost")
server = client.GetServer()
database = server.GetDatabase("test")
# Get a collection
collection = database.GetCollection("users")
# Add a document
from MongoDB.Bson import BsonDocument
user = BsonDocument({'first_name':'John', 'last_name':'Smith'})
collection.Insert(user)
See the the MongoDB C# Driver API for more information.

Related

pyspark: IOError: [Errno 20] Not a directory

I am running a pyspark job on AWS-EMR and I got the following error:
IOError: [Errno 20] Not a directory: '/mnt/yarn/usercache/hadoop/filecache/12/my_common-9.0-py2.7.egg/my_common/data_tools/myData.yaml'
Does anyone know what I might have missed? Thanks!
I've run into this recently when I switched my Python Spark application from Client deploy mode to Cluster deploy mode.
My workaround is to locate the ZIP file (the artifact that I fed to spark-submit using --py-files):
CURRENT_FILE_PATH = os.path.dirname(__file__)
print("[DEBUG] CURRENT_FILE_PATH=" + CURRENT_FILE_PATH)
It comes out something like this:
/mnt2/yarn/usercache/task/appcache/application_1638998214637_0019/container_1638998214637_0019_02_000001/something.zip
then I can use something like:
import zipfile
archive = zipfile.ZipFile(CURRENT_FILE_PATH, 'r')
json_bytes = archive.read('myfile.json')
json_string = json.loads(json_bytes)
Note: I first tried using pkg_resources but couldn't read in the
resulting JSON due to TypeError from json.loads():
import pkg_resources
json_data = pkg_resources.resource_stream(__name__, 'myfile.json')
See also PySpark: how to resolve path of a resource file present inside the dependency zip file
As the error states my_common-9.0-py2.7.egg is not a directory.
Are you missing space in your path?
/mnt/yarn/usercache/hadoop/filecache/12/my_common-9.0-py2.7.egg /my_common/data_tools/myData.yaml

make packages installed in virtualenv visibile to sphinx

I am using sphinx to document my software. and I am using a virtualenv for the installation. now some packages are only installed in the virtual environment, and sphinx does not see them.
I have this code in my conf.py:
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
p = os.path.abspath('..')
sys.path.insert(0, p)
if 'VIRTUAL_ENV' in os.environ:
q = os.sep.join([os.environ['VIRTUAL_ENV'],
'lib', 'python2.7', 'site-packages'])
sys.path.insert(0, q)
p = p + ":" + q
os.environ['PYTHONPATH'] = p
yet if I make html, I get this sort of warnings:
/home/mario/Local/github/Bauble/bauble.classic/doc/api.rst:358: WARNING: autodoc: failed to import class u'TagItemGUI' from module u'bauble.plugins.tag'; the following exception was raised:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/sphinx/ext/autodoc.py", line 385, in import_object
__import__(self.modname)
File "/home/mario/Local/github/Bauble/bauble.classic/bauble/plugins/tag/__init__.py", line 30, in <module>
from sqlalchemy import *
ImportError: No module named sqlalchemy
my $VIRTUAL_ENV/lib/python2.7/site-packages contains SQLAlchemy-1.0.4-py2.7-linux-x86_64.egg.
definitely related to question Sphinx autodoc dies on ImportError of third party package, but the description of the procedure I chose to follow is in a broken link.
The problem is that packages are not directly included in virtualenv's site-packages dir, you would need to specify the full path to be able to import package from there. I use the following hack:
if 'VIRTUAL_ENV' in os.environ:
site_packages_glob = os.sep.join([
os.environ['VIRTUAL_ENV'],
'lib', 'python2.7', 'site-packages', 'projectname-*py2.7.egg'])
site_packages = glob.glob(site_packages_glob)[-1]
sys.path.insert(0, site_packages)
Where projectname is the name of the python module I would like to import.
Note that this is error prone, especially when you have multiple versions
of the module, but so far it works for me.

Web2py - Auth with MongoDB

Good day,
I'm trying to use MongoDB with web2py, and for that I started with authentication, but this appeared some errors that I do not understand.
In a relational database, the web2py creates the authentication tables, MongoDB in the collections are not created automatically.
Below is the code and the error when trying to log me:
db.py
db = DAL("mongodb://localhost/primer", check_reserved=["mongodb_nonreserved",], adapter_args={"safe":False})
from gluon.tools import Auth, Service, PluginManager
auth = Auth(db)
service = Service()
plugins = PluginManager()
auth.settings.remember_me_form = False
auth.settings.actions_disabled=['register','change_password','request_reset_password','retrieve_username','profile']
auth.define_tables(username=True)
from gluon.contrib.login_methods.ldap_auth import ldap_auth
auth.settings.login_methods = [ldap_auth(server='localhost', port='10389', base_dn='ou=people,o=empresa,dc=com,dc=br')]
The authentication is by LDAP, and works perfectly in a relational database, which has the AUTH_USER table.
However, the loging using MongoDB, this appearing the following error:
Traceback (most recent call last):
File "C:\Users\Rafa\Desktop\web2py-10-06-2015p4\applications\contrato\controllers/appadmin.py", line 249, in select
nrows = db(query, ignore_common_filters=True).count()
File "C:\Users\Rafa\Desktop\web2py-10-06-2015p4\gluon\packages\dal\pydal\objects.py", line 2016, in count
return db._adapter.count(self.query,distinct)
File "C:\Users\Rafa\Desktop\web2py-10-06-2015p4\gluon\packages\dal\pydal\adapters\mongo.py", line 200, in count
count=True,snapshot=snapshot)['count'])
File "C:\Users\Rafa\Desktop\web2py-10-06-2015p4\gluon\packages\dal\pydal\adapters\mongo.py", line 319, in select
sort=mongosort_list, snapshot=snapshot).count()}
File "C:\Python27\lib\site-packages\pymongo\collection.py", line 929, in find
return Cursor(self, *args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'snapshot'
The database "primer" is created and only has two collections "posts" and "system.indexes"
Could someone help me with this error to be able to use MongoDB with the web2py?
Thank You!
Found it.
From pymongo's changelog There are a lot of breaking changes in pymongo 3.0 compared to 2.8
The following find/find_one options have been removed:
snapshot (use the new modifiers option instead)
So uninstall pymongo and try the latest before 3.0:
pip install pymongo==2.8.1
Here's my attempt:
>>> from pydal import *
No handlers could be found for logger "web2py"
>>> db = DAL('mongodb://localhost/connect_test')
>>> db.define_table('some',Field('key'),Field('value'))
<Table some (id,key,value)>
>>> db.define_table('some2',Field('ref','reference some'),Field('value'))
<Table some2 (id,ref,value)>
>>> db(db.some).select()
<Rows (1)>
>>> db(db.some).select().first()
<Row {'value': 'pir', 'key': 'bla', 'id': 26563964102769618087622556519L}>
>>>
[edit]
There's more to it. This worked at least with pydal 15.03. Googling some code i found the following in the mongo.py adapter :
from pymongo import version
if 'fake_version' in driver_args:
version = driver_args['fake_version']
if int(version.split('.')[0]) < 3:
raise Exception(
"pydal requires pymongo version >= 3.0, found '%s'"
% version)
Which was like good soil for a big frown...
After updating pydal to 15.07 it apears to brake indeed:
RuntimeError: Failure to connect, tried 5 times:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\pydal\base.py", line 437, in __init__
self._adapter = ADAPTERS[self._dbname](**kwargs)
File "C:\Python27\lib\site-packages\pydal\adapters\base.py", line 57, in __call__
obj = super(AdapterMeta, cls).__call__(*args, **kwargs)
File "C:\Python27\lib\site-packages\pydal\adapters\mongo.py", line 82, in __init__
% version)
Exception: pydal requires pymongo version >= 3.0, found '2.8.1'
So it's back to upgrading pymongo :)
With pymongo at 3.0.3 and pydal at 15.07 it works like a charm again.

django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet. tastypie

Continuing my search for an answer to get oauth2.0 to work on pythonanywhere.
i am following this tutorial: http://ianalexandr.com/blog/building-a-true-oauth-20-api-with-django-and-tasty-pie.html
im using django 1.6 : https://www.pythonanywhere.com/wiki/VirtualEnvForNewerDjango
when i get to this line of codes:
from provider.oauth2.models import Client
# from django.contrib.auth.models import User
from django.contrib.auth import get_user_model
User = get_user_model()
u = User.objects.get(id=1)
c = Client(user=u, name="mysite client", client_type=1, url="http://pythonx00x.pythonanywhere.com")
c.save()
c.client_id
'd63f53a7a6cceba04db5'
c.client_secret
'afe899288b9ac4127d57f2f12ac5a49d839364dc'
it seems that i got an error at line:
User = get_user_model()
and it raise an error:
raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
here is the full stack trace:
Traceback (most recent call last):
File "addClient.py", line 9, in <module>
User = get_user_model()
File "/home/python2006/.virtualenvs/django16/local/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 136, in get_user_model
return django_apps.get_model(settings.AUTH_USER_MODEL)
File "/home/python2006/.virtualenvs/django16/local/lib/python2.7/site-packages/django/apps/registry.py", line 200, in get_model
self.check_models_ready()
File "/home/python2006/.virtualenvs/django16/local/lib/python2.7/site-packages/django/apps/registry.py", line 132, in check_models_ready
raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
I can't seem to find out how to get the models load.
if I'm getting the idea right.
I think you may not be using the version of Django that you think you are. AppRegistryNotReady was introduced in Django 1.7. I would guess that, if you pinned your DJango version to 1.6, your code would work.

Bottle web framework: any way to run a console/shell and get it to work with Werkzeug?

I searched but couldn't find an easy way to run a console or shell akin to Django's manage.py shell or Rail's rails console
Since I just started using Bottle for an existing project, I just wanted to play around with the existing models and managers in the console. The closest I came up with was using ipdb's set_trace() and go from there, but that's not ideal by any means.
Also, I tried integrating Bottle with Werkzeug, but when I follow the instructions:
import bottle
app = bottle.Bottle()
werkzeug = bottle.ext.werkzeug.Plugin()
app.install(werkzeug)
I get the following traceback error:
Traceback (most recent call last):
File "mysite.py", line 62, in <module>
werkzeug = bottle.ext.werkzeug.Plugin()
AttributeError: 'module' object has no attribute 'werkzeug'
Try adding importing bottle.ext.werkzeug by adding this at the beginning of your source:
import bottle.ext.werkzeug