FastAPI with MongoDB - mongodb

Well im kind of new with all the Back-end stuff so pardon if i ask silly questions or my code makes no sense ;).
so what im trying to do is to transfer data between my api and the database. right now focusing on get-all and post methods. the data that im transferring is Als files (abelton live set).
i have a main.py file which is where the routes sit, database.py containing the engine and the functions and file_model.py which is the pydantic model.
im able to post a file and indeed i can see it in my DB, it has the default mongo _id : objectid
and the file.
when i try the get_all_files_db() function the errors start to show, well there was one problem with the _id which python cant read properly so i tried to fix it with some web searching.
its hard to define my problem because I probably have many, so specifying the error will not be relevant.
so this is where you guys come in, your answers could be code recommendations or a link to some docs that may help me understand better or even using some different libraries/dependencies because i see that fastapi and mongoDB is annoying when dealing with bigger files and not regular json.
those are my 2 routes in main.py (used Uploadfile because of the file type)
`#app.post("/api/files" , response_model=Als)
async def post_file(title : str = Form(...), file : UploadFile = File(...)):
Als(title=title)
response = await upload_file(title,file.file)
if response :
return response
raise HTTPException(400, "Something went wrong")
#app.get("/api/files" , response_model= Als )
async def get_all_files():
response =await get_all_files_db()
return response`
here are my functions in the database.py and the connection to database.
client = motor.motor_asyncio.AsyncIOMotorClient('localhost', 27017)
db = client['Files']
collection = db['ALS_Files']
async def upload_file(title,file):
await collection.insert_one({title:file.read()})
return file
async def get_all_files_db():
files = []
cursor = collection.find({})
async for document in cursor:
#doc_copy = document.copy()
files.append(document)
return files()
and my file_model.py
class PyObjectId(ObjectId):
""" Custom Type for reading MongoDB IDs """
#classmethod
def __get_validators__(cls):
yield cls.validate
#classmethod
def validate(cls, v):
if not ObjectId.is_valid(v):
raise ValueError("Invalid object_id")
return ObjectId(v)
#classmethod
def __modify_schema__(cls, field_schema):
field_schema.update(type="string")
class Als(BaseModel) :
id: PyObjectId = Field(default_factory=PyObjectId, alias="_id")
title: str
class Config:
allow_population_by_field_name = True
arbitrary_types_allowed = True
json_encoders = {ObjectId: str}

Related

Is it possible to pull data from Heroku Postgresql?

Currently I am working on a telegram bot using Heroku and Flask to deploy it. I wanted to store some data into a database then I came across Heroku PostgreSQL. I managed to store data into it but I am having trouble retrieving these data from Heroku PostgreSQL. I have research quite a bit but couldn't find much information about the process on pulling the data. Can someone enlighten me or suggest what are the steps to retrieve/pull these data from Heroku PostgreSQL with Flask?
The Telegram Bot is basically a random food generator. I have stored a list of food into the data and wanted to pull these data and random generator whenever user trigger the bot/action.
Screenshot of the current heroku postgresql database
from os import environ
from flask import Flask, request, jsonify
from flask_sqlalchemy import SQLAlchemy
from flask_cors import CORS
from invokes import invoke_http
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = environ.get("DB_URL") or "postgres://database-url"
app.config["SQLALCHEMY_TRACK_MODIFICATION"] = false
db = SQLAlchemy(app)
CORS(app)
class Cusine(db.Model):
__tablename__ = "cusines"
id = db.Column(db.String(255), primary_key =True)
cusine = db.Column(db.String(255), nullable=False)
def __init__(self, id, cusine):
self.id = id
self.cusine = cusine
def getId(self):
return self.id
def getCusine(self):
return self.cusine
def json(self):
return {"id": self.id,"cusine":self.cusine}
db.create_all()
#app.route("/getAllCusine")#
def getAllCusine():
try:
all_cusine = Cusine.query.all()
cusines = []
for row in all_cusine:
cusines.append(row.cusine)
return jsonify(
{
"code": 200,
"message": cusines
}
),200
except:
return jsonify(
{
"code": 500,
"message": "error in getting all course"
}
),500
So, how can I pull the data from Heroku PostgreSQL using Flask?
Thank you in advance.
I have managed to extract the data from Heroku PostgreSQL.
I have changed the above code with the following.
import os
import psycopg2
DATABASE_URL = os.environ['DATABASE_URL']
def retrieveData():
try:
conn = psycopg2.connect(DATABASE_URL, sslmode='require')
cursor = conn.cursor()
cursor.execute("select * from table") #some sql statement
conn.commit()
result = cursor.fetchall()
cursor.close()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
return result
It will return a list of tuple.
Remember to pip install psycopg2-binary
It is my very first time working with Heroku and PostgreSQL, which took me quite awhile to figure out. But thanks guy for teaching me how to properly use stackoverflow, appreciate it. Hopefully this can help someone out there (like me), who is new to PostgreSQL. Cheers :)

Cannot find_one record in MongoDB with custom _id's

I have a MongoDB database where I overwrote the _id field manually using some generated tags (duplicates are not an issue).
Trouble is, I am trying to find_one searching by _id but it continuously returns None and I cannot figure it out for the life of me. Any help would be appreciated.
Here is the code I have so far:
async def update(self):
#self.input is part of my class variables and comes in as a string
client = AsyncIOMotorClient(sensitive_data.mongoDB_server_address) #sensitive data is another python file with the server address/usernae/password
db = client.Database
accounts = db.accounts
print(type(self.input)) #this prints string
account = await accounts.find_one({'_id':self.input})
print(account) #this prints none
return account
Edit:
I found some information on pyMongo documentation about ObjectID. I tried implementing it with this code:
from bson.objectid import ObjectId
async def update(self):
#self.input is part of my class variables and comes in as a string
client = AsyncIOMotorClient(sensitive_data.mongoDB_server_address) #sensitive data is another python file with the server address/usernae/password
db = client.Database
accounts = db.accounts
print(type(self.input)) #this prints string
account = await accounts.find_one({'_id':ObjectID(self.input)})
print(account) #this prints none
return account
But get this traceback:
bson.errors.InvalidId: 'mystring' is not a valid ObjectId, it must be a 12-byte input or a 24-character hex string
I figured it out. I was trying to find an ID in a database that didn't exist. Oops.

Best tools for creating a REST API with static data [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
I need some Tips for building a REST API with about 35000 static (non-changing) JSON data.
It's my first time building a REST API seriously, so I need some design decision advice.
First, I was planning to use Flask to build the API since I am familiar with it and MongoDB to store the data. But I've heard that MongoDB is not a good choice for data that do not change.
What I would like to know are:
Which DB is suitable for this kind of data?
Is Flask a good choice if I am expecting many users using the API at the same time?
What are the brief steps for doing this? What I have in my mind right now is something like below:
Steps:
1) Upload my data to DB
2) Create a REST API that helps the user fetch the data
3) Upload the REST API to some server
4) Test with Postman to see if it works
Is my overall thought correct?
Any advice would be great. Thanks in advance.
If you are unsure about what DB to use I would just go with PostgreSQL. It's scalable so if you ever need to build on your dataset it will work just fine. In terms of performance it just depends on how many requests it gets, but I bet it could handle whatever you throw at it.
Regarding the API, if you are set with Flask then I recommend the package Flask-Restful. Outline your db using an ORM in a file called models.py. In a folder called resources, make files that serve as your API resources. Example would be blogposts.py, which would have a get request for all or a single post, post, put, and delete for single posts. Here is something I have for a really lightweight blog. Using peewee as an ORM and another package called Flask-HTTPAuth for authentication.
# blogposts.py
import json
from flask import jsonify, Blueprint, abort, make_response
from flask_restful import (Resource, Api, reqparse, inputs, fields,
url_for, marshal, marshal_with)
from auth import auth
import models
blogpost_fields = {
'id': fields.Integer,
'title': fields.String,
'content': fields.String,
'created': fields.DateTime
}
def blogpost_or_404(id):
try:
blogpost = models.BlogPost.get(models.BlogPost.id==id)
except models.BlogPost.DoesNotExist:
abort(404)
else:
return blogpost
class BlogPostList(Resource):
def __init__(self):
self.reqparse = reqparse.RequestParser()
self.reqparse.add_argument(
'title',
required=True,
help='No title provided',
location=['form', 'json']
)
self.reqparse.add_argument(
'content',
required=False,
nullable=True,
location=['form', 'json'],
default=''
)
super().__init__()
def get(self):
blogpost = [marshal(blogpost, blogpost_fields)
for blogpost in models.BlogPost.select()]
return {'BlogPosts': blogpost}
#marshal_with(blogpost_fields)
#auth.login_required
def post(self):
args = self.reqparse.parse_args()
blogpost = models.BlogPost.create(**args)
return (blogpost, 201, {
'Location': url_for('resources.blogposts.blogpost', id=blogpost.id)
})
class BlogPost(Resource):
def __init__(self):
self.reqparse = reqparse.RequestParser()
self.reqparse.add_argument(
'title',
required=False,
help='No title provided',
location=['form', 'json']
)
self.reqparse.add_argument(
'content',
required=False,
nullable=True,
location=['form', 'json'],
default=''
)
super().__init__()
#marshal_with(blogpost_fields)
def get(self, id):
return (blogpost_or_404(id))
#marshal_with(blogpost_fields)
#auth.login_required
def put(self, id):
args = self.reqparse.parse_args()
try:
blogpost = models.BlogPost.select().where(
models.BlogPost.id==id).get()
except models.BlogPost.DoesNotExist:
return make_response(json.dumps(
{'error': 'That blogpost does not exist or is not editable'}
), 403)
else:
query = blogpost.update(**args).where(models.BlogPost.id==id)
query.execute()
blogpost = (blogpost_or_404(id))
return (blogpost, 200, {
'Location': url_for('resources.blogposts.blogpost', id=id)
})
#auth.login_required
def delete(self, id):
try:
blogpost = models.BlogPost.select().where(
models.BlogPost.id==id).get()
except models.BlogPost.DoesNotExist:
return make_response(json.dumps(
{'error': 'That blogpost does not exist or is not editable'}
), 403)
else:
query = blogpost.delete().where(models.BlogPost.id==id)
query.execute()
return '', 204, {'Location': url_for('resources.blogposts.blogposts')}
blogposts_api = Blueprint('resources.blogposts', __name__)
api = Api(blogposts_api)
api.add_resource(
BlogPostList,
'/blogposts',
endpoint='blogposts'
)
api.add_resource(
BlogPost,
'/blogposts/<int:id>',
endpoint='blogpost'
)
Resource classes have methods with the http method name, this is what sets which methods are allowed. For instance, if I tried to delete to /blogposts without an ID, it would respond with method not allowed. Delete is only defined for a single post. Marshaling determines what information is in the response, you define it with blogpost_fields at the top. In the init of each class, we define the Request Parser which is what determines the information the API needs. In this example we only need a title and the post content. In a users resource you would add in things like email, username, password, verify password, admin status etc.
# models.py
import datetime
import jwt
from argon2 import PasswordHasher
from peewee import *
import config
DATABASE = PostgresqlDatabase('blogdb', user=config.DB['USER'], password=config.DB['PW'], host=config.DB['HOST'])
HASHER = PasswordHasher()
class User(Model):
username = CharField(unique=True)
email = CharField(unique=True)
password = CharField()
class Meta:
database = DATABASE
#classmethod
def create_user(cls, username, email, password, **kwargs):
email = email.lower()
try:
cls.select().where(
(cls.email==email)|(cls.username**username)
).get()
except cls.DoesNotExist:
user = cls(username=username, email=email)
user.password = user.set_password(password)
user.save()
return user
else:
raise Exception("User with that email or username already exists")
#staticmethod
def verify_auth_token(token):
try:
payload = jwt.decode(token, config.SECRET_KEY)
return payload['sub']
except jwt.ExpiredSignatureError:
return 'Signature expired. Please log in again.'
except jwt.InvalidTokenError:
return 'Invalid token. Please log in again.'
#staticmethod
def set_password(password):
return HASHER.hash(password)
def verify_password(self, password):
return HASHER.verify(self.password, password)
def generate_auth_token(self, id):
try:
payload = {
'exp': datetime.datetime.utcnow() + datetime.timedelta(days=0, seconds=5),
'iat': datetime.datetime.utcnow(),
'sub': id
}
return jwt.encode(
payload,
config.SECRET_KEY,
algorithm='HS256'
)
except Exception as e:
return e
class BlogPost(Model):
title = CharField(default='', unique=True)
content = TextField(default='')
created = DateTimeField(default=datetime.datetime.now)
class Meta:
database = DATABASE
def initialize():
DATABASE.connect()
DATABASE.create_tables([User, BlogPost], safe=True)
DATABASE.close()
# auth.py
from flask import g
from flask_httpauth import HTTPTokenAuth
import models
auth = HTTPTokenAuth(scheme='Bearer')
#auth.verify_token
def verify_token(token):
user = models.User.verify_auth_token(token)
if user is not None:
g.user = user
return True
return False
Models is pretty self explanatory if you've ever worked with an ORM like SQLAlchemy. I would recommend that package since your dataset is far far far larger than the one for this example. HTTPAuth allows you to decorate your API resource methods with a required authentication method. In my example, logging in will generate a JWT which needs to be sent with each request as a Bearer token.
Once all of that is set up you register your API blueprints in app.py
# app.py
app = Flask(__name__)
app.register_blueprint(users_api, url_prefix='/api/v1')
app.register_blueprint(blogposts_api, url_prefix='/api/v1')
app.register_blueprint(login_api)
That's it!

Jenkins: Active Choices Parameter + Groovy to build a list based on REST responde

I have a REST client that returns me a list of systems.
I need this list to be as a parameter for a jenkins job.
I think I need Actice Choices Parameter plugin with Groovy and HTTPBuilder in order to do that.
What do you guys think?
I did not find a way to install HTTPBuilder into Jenkins.
Is there any other way that you guys think it is possible?
I have run into the same problem trying to parse parameters via groovy script. Arun's answer did not work for me. However, I have managed to get it to work using the following:
import java.io.BufferedReader
import java.io.InputStreamReader
import java.io.OutputStreamWriter
import java.net.URL
import java.net.URLConnection
import groovy.json.JsonSlurper
def choices = []
def url = new URL("some.data.url")
def conn = url.openConnection()
conn.setDoOutput(true)
def reader = new BufferedReader(new InputStreamReader(conn.getInputStream()))
def results = new JsonSlurper().parseText(reader.getText());
reader.close()
results.each { data -> choices.push(data.field) }
return choices.sort()
First paste the JSON body snapshot output -or whatever your REST client is going to return. That'll help.
For ex: if it'll return a JSON object then you can use Active Choice Parameter's Groovy script step - OR Scriptler script (within the Active Choice Parameter plugin). PS: Scriptler script runs in the same JVM of Jenkins process so it has access to Jenkins/etc object for free. You don't need HTTPBuilder or anything. See the code sample below.
Assuming if your REST client is returning a JSON object and from that object if you want to list hostname of the system or some field name then replace the following variable with that and you'll get it listed while doing "Build with parameters" from Jenkins job's dashboard.
import groovy.json.JsonSlurper
//this will be your URL which will return something, tweak it if you want to pass parameters or username/password acc.
def SOME_URL = "https://koba.baby.com/some_url"
// now connect to the URL and create a connection variable 'conn'
def conn = SOME_URL.toURL().openConnection()
// create a list variable 'servernames'
def servernames = []
// if connection response was successful i.e. http protocol return code was 200, then do the following
if( conn.responseCode == 200 ) {
// get the results / output of the URL connection in a variable 'results'
def results = new JsonSlurper().parseText(conn.content.text)
// to see results variable output uncomment the next line
//println results
// now read each element in the 'results' variable and pick servername/somefield variable into the list variable 'servernames'
results.each { id, data -> servernames.push(data.someField_or_HostName) }
}
return servernames.sort().unique()
// return servernames.sort()

PlayFramework 2.3.x: Access public folder using URL with Play and Scala

I am uploading a videos and images using web-service and save the images in our application. When i save the files, the files are save on root of application folder. I want to access those images and videos with localhost url, like: I upload the file and save under app-root/upload/image.jpg. In my route mapping file, i declare routing as below:
GET /uploads/ staticDir:/upload
As define in Play Documentation. But still getting an compile time error: Controller method call expected. I want to access image like this http://localhost:9999/uploads/image.jpg
Well... One way of doing this is by adding following routes,
GET /uploads/*file controllers.Assets.at(path="/uploads", file)
But, it will interfere with the reverse-routing of already existing route which is,
GET /assets/*file controllers.Assets.at(path="/public", file)
And then you will have to use your these two assets routes as - #route.Assets.at("public", filename) and #route.Assets.at("uploads", filename) which means all your templates which use you public assets route as - #route.Assets.at(filename) will have to be changed. Which can be a hassle in an existing big project.
You can avoid this by using following method,
Create another controller as,
package controllers
object FileServer extends Controller {
def serveUploadedFiles1 = controllers.Assets.at( dicrectoryPath, file, false )
// Or... following is same as above
def serveUploadedFiles2( file: String ) = Action.async {
implicit request => {
val dicrectoryPath = "/uploads"
controllers.Assets.at( dicrectoryPath, file, false ).apply( request )
}
}
}
The above should have worked... but seems like play does a lot of meta-data checking on the requested "Assets" which somehow results in empty results for all /uploads/filename requests. I tried to look into the play-source code to check, but it seems like it may take sometime to figure it out.
So I think we can make do with following simpler method ( It can be refined further in so many ways.).
object FileServer extends Controller {
import play.api.http.ContentTypes
import play.api.libs.MimeTypes
import play.api.libs.iteratee.Enumerator
import play.api.libs.concurrent.Execution.Implicits.defaultContext
def serveUploadedFiles(file: String) = Action { implicit request =>
val fileResUri = "uploads/"+file
val mimeType: String = MimeTypes.forFileName( fileResUri ).fold(ContentTypes.BINARY)(addCharsetIfNeeded)
val serveFile = new java.io.File(fileResUri)
if( serveFile.exists() ){
val fileContent: Enumerator[Array[Byte]] = Enumerator.fromFile( serveFile )
//Ok.sendFile(serveFile).as( mimeType )
val response = Result(
ResponseHeader(
OK,
Map(
CONTENT_LENGTH -> serveFile.length.toString,
CONTENT_TYPE -> mimeType
)
),
fileContent
)
response
}
else {
NotFound
}
}
def addCharsetIfNeeded(mimeType: String): String =
if (MimeTypes.isText(mimeType)) s"$mimeType; charset=$defaultCharSet" else mimeType
lazy val defaultCharSet = config(_.getString("default.charset")).getOrElse("utf-8")
def config[T](lookup: Configuration => Option[T]): Option[T] = for {
app <- Play.maybeApplication
value <- lookup(app.configuration)
} yield value
}
But this method will cause some troubles in case of packaged-build deployments.
Which means, using the Play's Asset thing would be wiser choice. So looking again, the controllers.Assets.at which is actually controllers.Assets.assetAt uses this method at one place,
def resource(name: String): Option[URL] = for {
app <- Play.maybeApplication
resource <- app.resource(name)
} yield resource
Which means, it tries to locate the resource in the directories which are part of application's classpath and our uploads folder sure is not one of them. So... we can make play's Assets.at thingy work by adding uploads to classpath.
But... thinking again... If I recall all folders in the classpath are supposed to be packaged in the package to be deployed in-case of packaged-build deployments. And uploaded things will be created by the users, which means they should not be a part of package. Which again means... we should not be trying to access our uploaded things using Play's Assets.at thingy.
So... I think we are better off using our own simpler rudimentary implementation of serveUploadedFiles.
Now add a route in route file as,
GET /uploads/*file controllers.FileServer.serveUploadedFiles( file:String )
Also... Keep in mind that you should not be thinking of using play to serve your uploaded assets. Please use nginx or something similar.