I have a small flask application which I am deploying to Heroku.
My local configuration looks like this:
from flask import Flask
from flask.ext.mongoengine import MongoEngine
app = Flask(__name__)
app.debug = True
app.config["MONGODB_SETTINGS"] = {'DB': "my_app"}
app.config["SECRET_KEY"] = "secretpassword"
db = MongoEngine(app)
So, I know that I need to configure the app to use the Mongo URI method of connection, and I have my connection info:
mongodb://<user>:<password>#alex.mongohq.com:10043/app12345678
I am just a little stuck as to the syntax for modifying my app to connect through the URI.
So I got it working (finally):
from flask import Flask
from mongoengine import connect
app = Flask(__name__)
app.config["MONGODB_DB"] = 'app12345678'
connect(
'app12345678',
username='heroku',
password='a614e68b445d0d9d1c375740781073b4',
host='mongodb://<user>:<password>#alex.mongohq.com:10043/app12345678',
port=10043
)
Though I anticipate that various other configurations will work.
When you look at the flask-mongoengine code, you can see what configuration variables are available
So this should work:
app.config["MONGODB_HOST"] = 'alex.mongohq.com/app12345678'
app.config["MONGODB_PORT"] = 10043
app.config["MONGODB_DATABASE"] = 'dbname'
app.config["MONGODB_USERNAME"] = 'user'
app.config["MONGODB_PASSWORD"] = 'password'
db = MongoEngine(app)
I'm not sure, if app123 is the app or the database name. You might have to fiddle arround a little to get the connection. I had the same problem with Mongokit + MongoLab on Heroku :)
Also you could use the URI like this.
app.config["MONGODB_SETTINGS"] = {'DB': "my_app", "host":'mongodb://<user>:<password>#alex.mongohq.com:10043/app12345678'}
I have actually no idea, at what point "MONGODB_SETTINGS" is read, but it seemed to work, when I tried it in the shell.
I figured out how to use the flask.ext.mongoengine.MongoEngine wrapper class to do this rather than mongoengine.connect():
from flask import Flask
from flask.ext.mongoengine import MongoEngine
app = Flask(__name__)
HOST = '<hostname>' # ex: 'oceanic.mongohq.com'
db_settings = {
'MONGODB_DB': '<database>',
'MONGODB_USERNAME': '<username>',
'MONGODB_PASSWORD': '<password>',
'MONGODB_PORT': <port>,
}
app.config = dict(list(app.config.items()) + list(db_settings.items()))
app.config["MONGODB_HOST"] = ('mongodb://%(MONGODB_USERNAME)s:%(MONGODB_PASSWORD)s#'+
HOST +':%(MONGODB_PORT)s/%(MONGODB_DB)s') % db_settings
db = MongoEngine(app)
if __name__ == '__main__':
app.run()
If you're using mongohq, app.config["MONGODB_HOST"] should match the Mongo URI under Databases->Admin->Overview.
You can then follow MongoDB's tumblelog tutorial using this setup to write your first app called tumblelog.
Using python's nifty object introspection (python oh how I love you so), you can see how the MongoEngine wrapper class accomplishes this:
from flask.ext.mongoengine import MongoEngine
import inspect
print(inspect.getsource(MongoEngine))
...
conn_settings = {
'db': app.config.get('MONGODB_DB', None),
'username': app.config.get('MONGODB_USERNAME', None),
'password': app.config.get('MONGODB_PASSWORD', None),
'host': app.config.get('MONGODB_HOST', None),
'port': int(app.config.get('MONGODB_PORT', 0)) or None
}
...
self.connection = mongoengine.connect(**conn_settings)
...
self.app = app
Related
I am struggeling to return a file e.g. a pdf which was uploaded to a mongodb.
I am able to upload a file to the database but I am not able to retrieve the file again.
How shoud my endpoint (view) look like to return the file?
I am using django rest framework v3.12.4 with djongo v1.3.6. I use drf-yasg v1.20.0 for the documentation of the API.
Here are my settings, models.py, serializers.py, views.py and urls.py:
# app settings.py
DATABASES = {
'default': {
'ENGINE': 'djongo',
'NAME': 'TAST_DB2',
'CLIENT': {
'host': 'localhost',
'port': 27017,
'username': 'root',
'password': 'reallychangeme', # :-)
'authSource': 'admin',
'authMechanism': 'SCRAM-SHA-1'
}
}
}
DEFAULT_FILE_STORAGE = 'mongo_storage.storage.GridFSStorage'
GRIDFS_DATABASE = 'myfiles'
BASE_URL = 'http://localhost:8085/'
UPLOADED_FILES_USE_URL = True
# models.py
from django.db import models
from djongo.storage import GridFSStorage
grid_fs_storage = GridFSStorage(collection='myfiles', base_url=''.join([settings.BASE_URL, 'myfiles/']))
class TestStandardFile(models.Model):
myfile = models.FileField(upload_to='teststandards1', storage=grid_fs_storage)
# serializers.py
from rest_framework import serializers
from teststandards.models import TestStandardFile
class TestStandardFileSerializer(serializers.ModelSerializer):
class Meta:
model = TestStandardFile
fields = '__all__'
# views.py
from rest_framework.generics import ListCreateAPIView
from .models import TestStandard
from .serializers import TestStandardFileSerializer
from rest_framework.parsers import MultiPartParser
# used for the upload
class FileView(ListCreateAPIView):
parser_classes = ( MultiPartParser,)
serializer_class = TestStandardFileSerializer
queryset = TestStandardFile.objects.all()
<---------- !ENDPOINT FOR FILE RETRIEVAL MISSING HERE???
# urls.py
urlpatterns = [
re_path(r'^api/teststandards/file', api.FileView.as_view(), name='teststandard-file'),
re_path(r'^myfiles/(?P<pk>[0-9]+)$', api.myfiles.as_view(), name='get-file'),
]
I can see my file properly uploaded to mongodb with mongodb compass.
There are three collections in it:
TAST_DB2.teststandardsFiles
TAST_DB2.myfiles.teststandards1.files
TAST_DB2.myfiles.teststandards1.chunks
I assume that I need an endpoint which gives the file back as a response.
I tried to overwrite the 'get'-function of my 'myfiles' endpoint. But I don't know
how to get the file handle from the requested file. And I do not know how to
return the file as a HttpResponse.
Any help is appreciated!
I finally got it to work. I created an RetrieveAPIView for the retrieval of one entry and overwrote the retrieve function. THis is how my views.py looked like:
# for upload
class FileView(ListCreateAPIView):
parser_classes = ( MultiPartParser,)
serializer_class = TestStandardFileSerializer
queryset = TestStandardFile.objects.all()
# download
class myfiles(RetrieveAPIView):
parser_classes = ( MultiPartParser,)
serializer_class = TestStandardFileSerializer
queryset = TestStandardFile.objects.all()
def retrieve(self, request, *args, **kwargs):
obj = self.get_object()
response = HttpResponse(obj.myfile, content_type='application/octet-stream')
response['Content-Disposition'] = 'attachment; filename=%s' % obj.myfile
return response
"myfile" is the name of the FileField from my model. With using drf_spectacular for the swagger documentation. This generated a nice download button for the file retrieval. It helped a lot during testing the upload / download functionality.
I'm trying to connect to a postgres-DB, which unfortunately has a name with a whitespace in it:
%load_ext sql
from sqlalchemy import create_engine
%sql postgresql://postgres:dbpass#localhost/Test DB
(psycopg2.OperationalError) FATAL: database "Test DB" does not exist
I've tried to follow some tipps on the internet and used:
import urllib.parse
urllib.parse.quote_plus("Test DB")
which simply results in a string "Test+DB" (this does not work).
How can I adress the database, without changing its name?
Best regards!
I was able to solve it by using sqlalchemy's create_engine(), therefore being able to simply save the string (with its whitespace) as a variable (eg database_name):
import sqlalchemy as db
database_name = 'Test DB'
engine = db.create_engine('postgresql://' + 'user_name' + ':' + 'password' + '#localhost/' + database_name)
connection = engine.connect()
s = 'SELECT id FROM user'
df = pd.read_sql_query(s, engine)
df.head()
Hope it helps, best regards.
I'm trying to create a connection and add a document with mongoengine through an SSH tunnel.
A successful attempt with pymongo can be seen below, I simply want something similar with mongoengine. :-)
from auth import *
import pymongo
from sshtunnel import SSHTunnelForwarder
server = SSHTunnelForwarder(
(HOST_IP, HOST_PORT),
ssh_username = SSH_USER,
ssh_password = SSH_PASS,
remote_bind_address = ('localhost', 27017)
)
server.start()
client = pymongo.MongoClient('127.0.0.1', server.local_bind_port)
db = client[MONGO_DB]
db.authenticate(MONGO_USER, MONGO_PASS)
coll = db.queue_db
coll.insert({"testFile42":43})
server.stop()
mongoengine.connect(
db=DB_NAME,
host="127.0.0.1",
port=server.local_bind_port
)
i've made a local server using flask and mongoDB which works great on windows, but when i moved my code to the raspberry pi, i've got an error which i couldn't figure out why it occurs.
the code im using:
1) for the flask server
from flask import Flask
from flask import jsonify
from flask import request
import pymongo
import time
import datetime
import json
app = Flask(__name__)
client = pymongo.MongoClient("localhost", 27017)
db = client['mqtt-db']
obs_collection = db['mqtt-collection']
#app.route("/obs")
def obs():
data_str = request.args.get("data")
print data_str
data = json.loads(data_str)
print data
data["date"] = datetime.datetime.now()
obs_collection.save(data)
return "success"
#app.route("/get_obs")
def get_obs():
res = []
for row in obs_collection.find():
del row['_id']
res.append(row)
return jsonify(res)
#app.route("/delete_all")
def delete_all():
res = obs_collection.delete_many({})
return jsonify({"deleted": res.deleted_count})
if __name__ == "__main__":
app.run(host="0.0.0.0", debug=True)
2) script for inserting messages into db , using mqtt protocol:
import paho.mqtt.client as mqtt
import pymongo
import json
import datetime
topic = "sensor"
host = "10.0.0.6"
client = pymongo.MongoClient("localhost", 27017)
db = client['mqtt-db']
mqtt_collection = db['mqtt-collection']
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe(topic)
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
data_str = str(msg.payload)
data = json.loads(data_str)
print data_str
print data
data["date"] = datetime.datetime.now()
mqtt_collection.save(data)
print(msg.topic+" "+str(msg.payload))
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect(host, 1883, 60)
# Blocking call that processes network traffic, dispatches callbacks and
# handles reconnecting.
# Other loop*() functions are available that give a threaded interface and a
# manual interface.
client.loop_forever()
the error occurs when i try to retrieve data from the server using "get_obs" function.
the error is: "Value Error: dictionary update sequence element #0 has length 4; 2 is required"
appreciate your help.
as #davidism suggested, the solution was to update to the latest version of Flask
I have an application with Blueprints and Celery
the code is here:
config.py
import os
from celery.schedules import crontab
basedir = os.path.abspath(os.path.dirname(__file__))
class Config:
SECRET_KEY = os.environ.get('SECRET_KEY') or ''
SQLALCHEMY_COMMIT_ON_TEARDOWN = True
RECORDS_PER_PAGE = 40
SQLALCHEMY_DATABASE_URI = ''
CELERY_BROKER_URL = ''
CELERY_RESULT_BACKEND = ''
CELERY_RESULT_DBURI = ''
CELERY_TIMEZONE = 'Europe/Kiev'
CELERY_ENABLE_UTC = False
CELERYBEAT_SCHEDULE = {}
#staticmethod
def init_app(app):
pass
class DevelopmentConfig(Config):
DEBUG = True
WTF_CSRF_ENABLED = True
APP_HOME = ''
SQLALCHEMY_DATABASE_URI = 'mysql+mysqldb://...'
CELERY_BROKER_URL = 'sqla+mysql://...'
CELERY_RESULT_BACKEND = "database"
CELERY_RESULT_DBURI = 'mysql://...'
CELERY_TIMEZONE = 'Europe/Kiev'
CELERY_ENABLE_UTC = False
CELERYBEAT_SCHEDULE = {
'send-email-every-morning': {
'task': 'app.workers.tasks.send_email_task',
'schedule': crontab(hour=6, minute=15),
},
}
class TestConfig(Config):
DEBUG = True
WTF_CSRF_ENABLED = False
TESTING = True
SQLALCHEMY_DATABASE_URI = 'mysql+mysqldb://...'
class ProdConfig(Config):
DEBUG = False
WTF_CSRF_ENABLED = True
SQLALCHEMY_DATABASE_URI = 'mysql+mysqldb://...'
CELERY_BROKER_URL = 'sqla+mysql://...celery'
CELERY_RESULT_BACKEND = "database"
CELERY_RESULT_DBURI = 'mysql://.../celery'
CELERY_TIMEZONE = 'Europe/Kiev'
CELERY_ENABLE_UTC = False
CELERYBEAT_SCHEDULE = {
'send-email-every-morning': {
'task': 'app.workers.tasks.send_email_task',
'schedule': crontab(hour=6, minute=15),
},
}
config = {
'development': DevelopmentConfig,
'default': ProdConfig,
'production': ProdConfig,
'testing': TestConfig,
}
class AppConf:
"""
Class to store current config even out of context
"""
def __init__(self):
self.app = None
self.config = {}
def init_app(self, app):
if hasattr(app, 'config'):
self.app = app
self.config = app.config.copy()
else:
raise TypeError
init.py:
import os
from flask import Flask
from celery import Celery
from config import config, AppConf
def create_app(config_name):
app = Flask(__name__)
app.config.from_object(config[config_name])
config[config_name].init_app(app)
app_conf.init_app(app)
# Connect to Staging view
from staging.views import staging as staging_blueprint
app.register_blueprint(staging_blueprint)
return app
def make_celery(app=None):
app = app or create_app(os.getenv('FLASK_CONFIG') or 'default')
celery = Celery(__name__, broker=app.config.CELERY_BROKER_URL)
celery.conf.update(app.conf)
TaskBase = celery.Task
class ContextTask(TaskBase):
abstract = True
def __call__(self, *args, **kwargs):
with app.app_context():
return TaskBase.__call__(self, *args, **kwargs)
celery.Task = ContextTask
return celery
tasks.py:
from app import make_celery, app_conf
cel = make_celery(app_conf.app)
#cel.task
def send_realm_to_fabricdb(realm, form):
some actions...
and here is the problem:
The Blueprint "staging" uses task send_realm_to_fabricdb, so it makes: from tasks import send_realm_to_fabricdb
than, when I just run application, everything goes ok
BUT, when I'm trying to run celery: celery -A app.tasks worker -l info --beat, it goes to cel = make_celery(app_conf.app) in tasks.py, got app=None and trying to create application again: registering a blueprint... so I've got cycle import here.
Could you tell me how to break this cycle?
Thanks in advance.
I don't have the code to try this out, but I think things would work better if you move the creation of the Celery instance out of tasks.py and into the create_app function, so that it happens at the same time the app instance is created.
The argument you give to the Celery worker in the -A option does not need to have the tasks, Celery just needs the celery object, so for example, you could create a separate starter script, say celery_worker.py that calls create_app to create app and cel and then give it to the worker as -A celery_worker.cel, without involving the blueprint at all.
Hope this helps.
What I do to solve this error is that I create two Flask instance which one is for Web app, and another is for initial Celery instance.
Like #Miguel said, I have
celery_app.py for celery instance
manager.py for Flask instance
And in these two files, each module has it's own Flask instance.
So I can use celery.task in Views. And I can start celery worker separately.
Thanks Bob Jordan, you can find the answer from https://stackoverflow.com/a/50665633/2794539,
Key points:
1. make_celery do two things at the same time: create celery app and run celery with flask content, so you can create two functions to do make_celery job
2. celery app must init before blueprint register
Having the same problem, I ended up solving it very easily using shared_task (docs), keeping a single app.py file and not having to instantiate the flask app multiple times.
The original situation that led to the circular import:
from src.app import celery # src.app is ALSO importing the blueprints which are importing this file which causes the circular import.
#celery.task(bind=True)
def celery_test(self):
sleep(5)
logger.info("Task processed by Celery.")
The current code that works fine and avoids the circular import:
# from src.app import celery <- not needed anymore!
#shared_task(bind=True)
def celery_test(self):
sleep(5)
logger.info("Task processed by Celery.")
Please mind that I'm pretty new to Celery so I might be overseeing important stuff, it would be great if someone more experienced can give their opinion.