Mongodb data schema validation errors - mongodb

In my flask app I am trying to insert data into a mongodb database and validating the data, but I keep getting validation errors and I can seem to be able to locate why this is the case. I am new to mongodb. Note the validation rules are applied when I create the users collection like this:
try:
collection = d.list_collections()
if collection is not None:
if os.environ.get("FLASK_ENV") == 'development':
d.users.drop()
d.emails.drop()
d.create_collection(name="users", validator=users.user, validationLevel='strict', validationAction='error')
d.create_collection(name="emails", validator=emails.email, validationLevel='strict', validationAction='error')
except OperationFailure as e:
app.logger.error(f'Validation setup error: {e}')
Here is my validation python file (users.py):
user = {
"bsonType": "object",
"additionalProperties": False,
"required": [
"username", "first_name", "last_name",
"password", "email", "pin",
"phone_number", "firebase_uid", "groups",
"created_at"
],
"properties": {
"username": {
"bsonType": "string",
"description": "must be a string and is required"
},
"first_name": {
"bsonType": "string",
"description": "must be a string and is required"
},
"last_name": {
"bsonType": "string",
"description": "must be a string and is required"
},
"password": {
"bsonType": "binData",
"description": "must be a hashed string and is required"
},
"email": {
"bsonType": "string",
"pattern": "^[a-z0-9]+[\\._]?[a-z0-9]+[#]\\w+[.]\\w{2,3}$",
"description": "must be a string and is required"
},
"pin": {
"bsonType": "int",
"description": "must be a number and is required"
},
"phone_number": {
"bsonType": "string",
"pattern": "^\\+234[^0](?:\\d{9})",
"description": "must be a string and is required"
},
"firebase_uid": {
"bsonType": "string",
"description": "User firebase registration uid"
},
"groups": {
"bsonType": "object",
"required": ["owned", "belong"],
"properties": {
"owned": {
"bsonType": "array",
"uniqueItems": True,
"items": {
"grp_id": {
"bsonType": "objectId",
"description": "Must be an objectId"
}
}
},
"belong": {
"bsonType": "array",
"uniqueItems": True,
"items": {
"grp_id": {
"bsonType": "objectId",
"description": "Must be an objectId"
},
}
}
}
},
"created_at": {
"bsonType": "date",
"description": "The date and time user was created"
}
}
}
And this is the data being inserted:
{
"username": "joshua223",
"first_name": "Joy",
"last_name": "Kudosen",
"password": "password",
"email": "joshua22#gmail.com",
"pin": 1245,
"phone_number": "+2348022192256",
"firebase_uid": "XttdyeAOGxewe3434renuRTjCX0KsHXxod92",
"groups": {
"owned": [
],
"belong": [
]
}
}
And this is the error:
[2022-05-19 01:30:28 +0100] [420098] [ERROR] Validation error: Document failed validation, full error: {'index': 0, 'code': 121, 'errInfo': {'failingDocumentId': ObjectId('62858fa4e0decf4b6664dc6e'), 'details': {'operatorName': '$and', 'clausesNotSatisfied': [{'index': 0, 'details': {'operatorName': '$eq', 'specifiedAs': {'bsonType': 'object'}, 'reason': 'field was missing'}}, ...
Please note I am inserting the "created_at" field using a python Datime.now(timezone.utc)" function and it isn't in the original data. This is how I am inserting:
try:
current_app.logger.info("register_user Function called ...")
# Check for duplicate user registration using
# email and phone_number
# print(f'email: {dict(dumps(json_string))}')
if is_valid_data(json_string["email"], mtype="email") \
and is_valid_data(json_string["phone_number"], mtype="phone_number") \
and is_valid_data(json_string["username"], mtype="username"):
if get_user_by_property("email", json_string["email"], confirm=True) \
or get_user_by_property("phone_number", json_string["phone_number"], confirm=True) \
or get_user_by_property("username", json_string["username"], confirm=True):
return False
else:
now = datetime.now(timezone.utc)
print(f"DateTime: {now}") # TODO: Remove
# print(f'Password: {user_info["password"]}')
user_info = {
"username": json_string["username"],
"first_name": json_string["first_name"],
"last_name": json_string["last_name"],
"password": hash_user_input(json_string["password"]),
"email": json_string["email"],
"pin": json_string["pin"],
"phone_number": json_string["phone_number"],
"firebase_uid": json_string["firebase_uid"],
"groups": json_string["groups"],
"created_at": now
}
try:
print("Insert_one about to be called ...") # TODO: Remove
u = db.users.insert_one(user_info).inserted_id
except Exception as i:
return current_app.logger.error(f'Validation error: {i}')

Related

Pymongo "and" operator not filtering query results [duplicate]

This question already has answers here:
Retrieve only the queried element in an object array in MongoDB collection
(18 answers)
Closed 25 days ago.
I have this monogdb schema:
groups = {
"bsonType": "object",
"required": [
"created_at", "group_name", "owner",
"members", "currency", "country",
"group_username"
],
"properties": {
"created_at": {
"bsonType": "date",
"description": "The date and time when the group was created"
},
"group_name": {
"bsonType": "string",
"description": "Name of group"
},
"group_username": {
"bsonType": "string",
"description": "groups username"
},
"country": {
"bsonType": "string",
"description": "User's country and is required"
},
"currency": {
"bsonType": "string",
"description": "User's currency type and is required"
},
"description": {
"bsonType": "string",
"description": "Brief description of what the group does"
},
"block": {
"bsonType": "bool",
"description": "Field to check if user disabled from group activity"
},
"owner": {
"bsonType": "object",
"required": ["user_id", "status"],
"properties": {
"user_id": {
"bsonType": "objectId",
"description": "User id of the owner of the group"
},
"status": {
"bsonType": "string",
"description": "Default value is 'owner'",
}
},
},
"group_invite_link": {
"bsonType": "string",
"description": "Link used to invite new members"
},
"members": {
"bsonType": "array",
"items": {
"bsonType": "object",
"required": ["user_id", "name", "status", "confirmed", "joined_at"],
"properties": {
"user_id": {
"bsonType": "objectId"
},
"name": {
"bsonType": "string",
"description": "Name of group member"
},
"status": {
"enum": ["admin", "member"],
"description": "can only be one of the enum values and is required"
},
"confirmed": {
"bsonType": "bool",
"description": "Confirms if user is fully authorized to operate in group"
},
"joined_at": {
"bsonType": "date",
"description": "Date user joined group"
}
}
}
},
"group_wallet_id": {
"bsonType": "objectId",
"description": "Id of wallet associated with group"
}
}
}
and I am querying it to return only documents that match the below query:
db.groups.find({
"$and": [
{
"members.user_id": ObjectId(user_id)
},
{
"members.confirmed": True
}
]
})
It returns this:
{
"sucess": true,
"message": "Users groups",
"data": [
{
"_id": "63987c4263d7eba1e79a1df1",
"created_at": "2022-12-13 13:21:03",
"group_name": "Demo 1",
"is_owner": true,
"balance": {
"$numberDecimal": "250.00"
},
"members": [
{
"user_id": "63987b7c63d7eba1e79a1dd7",
"name": "Joy Kudosen",
"status": "admin",
"confirmed": true,
"joined_at": "2022-12-13 13:21:03"
},
{
"user_id": "63987b7c63d7eba1e79a1dd8",
"name": "Joy Kudosen",
"status": "member",
"confirmed": true,
"joined_at": "2022-12-14 13:22:03"
},
{
"user_id": "63987b7c63d7eba1e79a1dd7",
"name": "Joy Kudosen",
"status": "admin",
"confirmed": true,
"joined_at": "2022-12-14 11:05:39"
}
]
},
{
"_id": "63987c5a63d7eba1e79a1df4",
"created_at": "2022-12-13 13:21:27",
"group_name": "Demo 2",
"is_owner": true,
"balance": {
"$numberDecimal": "100"
},
"members": [
{
"user_id": "63987b7c63d7eba1e79a1dd7",
"name": "Joy Kudosen",
"status": "admin",
"confirmed": true,
"joined_at": "2022-12-13 13:21:27"
}
]
},
{
"_id": "6399ae06ebaca8cd5fbcd639",
"created_at": "2022-12-14 11:05:39",
"group_name": "Demo 4",
"is_owner": true,
"balance": {
"$numberDecimal": "0"
},
"members": [
{
"user_id": "63987b7c63d7eba1e79a1dd7",
"name": "Joy Kudosen",
"status": "admin",
"confirmed": true,
"joined_at": "2022-12-14 11:05:39"
}
]
},
{
"_id": "63d1810699edcb7379f21c27",
"created_at": "2023-01-25 19:20:33",
"group_name": "Hello world",
"is_owner": false,
"balance": {
"$numberDecimal": "0"
},
"members": [
{
"user_id": "63d180c399edcb7379f21c23",
"name": "Joy Kudosen",
"status": "admin",
"confirmed": true,
"joined_at": "2023-01-25 19:20:33"
},
{
"user_id": "63987b7c63d7eba1e79a1dd7",
"name": "Joy Kudosen",
"status": "member",
"confirmed": false,
"joined_at": "2023-01-25 19:23:31"
}
]
}
]
}
I want to return only array objects that match the user's id and "confirmed" value of "true" in the members array like this:
{
"sucess": true,
"message": "Users groups",
"data": [
{
"_id": "63987c4263d7eba1e79a1df1",
"created_at": "2022-12-13 13:21:03",
"group_name": "Demo 1",
"is_owner": true,
"balance": {
"$numberDecimal": "250.00"
},
"members": [
{
"user_id": "63987b7c63d7eba1e79a1dd7",
"name": "Joy Kudosen",
"status": "admin",
"confirmed": true,
"joined_at": "2022-12-13 13:21:03"
},
{
"user_id": "63987b7c63d7eba1e79a1dd8",
"name": "Joy Kudosen",
"status": "member",
"confirmed": true,
"joined_at": "2022-12-14 13:22:03"
},
{
"user_id": "63987b7c63d7eba1e79a1dd7",
"name": "Joy Kudosen",
"status": "admin",
"confirmed": true,
"joined_at": "2022-12-14 11:05:39"
}
]
},
{
"_id": "63987c5a63d7eba1e79a1df4",
"created_at": "2022-12-13 13:21:27",
"group_name": "Demo 2",
"is_owner": true,
"balance": {
"$numberDecimal": "100"
},
"members": [
{
"user_id": "63987b7c63d7eba1e79a1dd7",
"name": "Joy Kudosen",
"status": "admin",
"confirmed": true,
"joined_at": "2022-12-13 13:21:27"
}
]
},
{
"_id": "6399ae06ebaca8cd5fbcd639",
"created_at": "2022-12-14 11:05:39",
"group_name": "Demo 4",
"is_owner": true,
"balance": {
"$numberDecimal": "0"
},
"members": [
{
"user_id": "63987b7c63d7eba1e79a1dd7",
"name": "Joy Kudosen",
"status": "admin",
"confirmed": true,
"joined_at": "2022-12-14 11:05:39"
}
]
}
]
}
I have used different forms of this query but I am unable to filter out the "hello world" group information. I have tried aggregation and the "or" operation and still get the same results.
"$elemMatch" is used to find documents that match multiple fields within an array element.
For example, like this:
db.groups.find({
"members": {
"$elemMatch": {
"user_id": "63987b7c63d7eba1e79a1dd7",
"confirmed": true
}
}
})
Try it on mongoplayground.net.

How is the value NULL?

I am getting query results that determine if user story hasn't been changed (changedate) in the last one day.
I'm following this article to build the logic app as the intention is similar
For some reason, despite the query returning a valid response (at least 1 user story result), the foreach expression is throwing this error:
ExpressionEvaluationFailed. The execution of template action 'For_each' failed: the result of the evaluation of 'foreach' expression '#body('Parse_JSON')?['body']?['value']' is of type 'Null'. The result must be a valid array.
How is it NULL when clearly there is a user story returned?
Get query results:
OUTPUTS:
[
{
"System.Id": 12345,
"System.WorkItemType": "User Story",
"System.State": "New",
"System.Title": "Experiment"
}
]
Parse JSON:
Inputs:
Content:
{
"value": [
{
"System.Id": 12345,
"System.WorkItemType": "User Story",
"System.State": "New",
"System.Title": "Experiment"
}
],
"#odata.nextLink": null
}
Schema
{
"type": "object",
"properties": {
"body": {
"type": "object",
"properties": {
"value": {
"type": "array",
"items": {
"type": "object",
"properties": {
"System.AssignedTo": {
"type": "string"
},
"System.Id": {
"type": "integer"
},
"System.State": {
"type": "string"
},
"System.Tags": {
"type": "string"
},
"System.Title": {
"type": "string"
},
"System.WorkItemType": {
"type": "string"
}
},
"required": [
"System.Id",
"System.WorkItemType",
"System.State",
"System.AssignedTo",
"System.Title"
]
}
},
"#odata.nextLink": {}
}
},
"headers": {
"type": "object",
"properties": {
"Cache-Control": {
"type": "string"
},
"Content-Length": {
"type": "string"
},
"Content-Type": {
"type": "string"
},
"Date": {
"type": "string"
},
"Expires": {
"type": "string"
},
"Pragma": {
"type": "string"
},
"Set-Cookie": {
"type": "string"
},
"Strict-Transport-Security": {
"type": "string"
},
"Timing-Allow-Origin": {
"type": "string"
},
"Transfer-Encoding": {
"type": "string"
},
"Vary": {
"type": "string"
},
"X-Content-Type-Options": {
"type": "string"
},
"X-Frame-Options": {
"type": "string"
},
"x-ms-apihub-cached-response": {
"type": "string"
},
"x-ms-apihub-obo": {
"type": "string"
},
"x-ms-request-id": {
"type": "string"
}
}
},
"statusCode": {
"type": "integer"
}
}
}
Outputs:
{
"value": [
{
"System.Id": 12345,
"System.WorkItemType": "User Story",
"System.State": "New",
"System.Title": "Experiment"
}
],
"#odata.nextLink": null
}
Using the Value from the Get Query Results directly works.

Kafka JDBC sink no handling null values

I am trying to insert data with the Kafka JDBC Sink connector, but it is returning me this exception.
org.apache.kafka.connect.errors.DataException: Invalid null value for required INT64 field
The records have the following schema:
[
{
"schema": {
"type": "struct",
"fields": [
{
"type": "int64",
"field": "ID"
},
{
"type": "int64",
"field": "TENANT_ID"
},
{
"type": "string",
"field": "ITEM"
},
{
"type": "int64",
"field": "TIPO"
},
{
"type": "int64",
"field": "BUSINESS_CONCEPT"
},
{
"type": "string",
"field": "ETIQUETA"
},
{
"type": "string",
"field": "VALOR"
},
{
"type": "string",
"field": "GG_T_TYPE"
},
{
"type": "string",
"field": "GG_T_TIMESTAMP"
},
{
"type": "string",
"field": "TD_T_TIMESTAMP"
},
{
"type": "string",
"field": "POS"
}
]
},
"payload": {
"ID": 298457,
"TENANT_ID": 83,
"ITEM": "0-0-0",
"TIPO": 4,
"BUSINESS_CONCEPT": null,
"ETIQUETA": "¿Cuándo ha ocurrido?",
"VALOR": "2019-05-31T10:33:00Z",
"GG_T_TYPE": "I",
"GG_T_TIMESTAMP": "2019-05-31 14:35:19.002221",
"TD_T_TIMESTAMP": "2019-06-05T10:46:55.0106",
"POS": "00000000530096832544"
}
}
]
As you can see, the value BUSINESS_CONCEPT can be null. It is the only null value, so I suppose the exception is due to that field. How could I make the sink insert the value as null?
You need to change the definition of
{
"type": "int64",
"field": "BUSINESS_CONCEPT"
}
to
{
"type": ["null", "int64"],
"field": "BUSINESS_CONCEPT"
}
in order to treat BUSINESS_CONCEPT as optional field.

Gupshup consume post to create Embedding form

im working with Gupshup and I want to add subview in my chat.
To make this I create this vars:
var url="https://api.gupshup.io/sm/api/facebook/smartmsg/form/create";
var header = {"apikey":"xxxxxxxxxxxxxxxxxxx","Content-Type": "application/x-www-form-urlencoded","Accept":"application/json"};
var param={"formJSON":{
"title": "This is a test.",
"autoClose": false,
"message": "Thank You",
"callback-url": "https://www.gupshup.io/developer/bot/Cotizador/public",
"fields": [{
"type": "fbid",
"name": "fbname",
"label": "fbName"
}, {
"type": "input",
"name": "name",
"label": "Name",
"validations": [{
"regex": "^[A-Z a-z]+$",
"msg": "Only alphabets are allowed in this field"
}, {
"regex": "^[A-Z a-z]{6,}$",
"msg": "Minimum 6 characters required"
}]
}, {
"type": "radio",
"name": "gender",
"label": "Gender",
"options": [
"Male",
"Female"
],
"validations": [{
"regex": "",
"msg": ""
}]
}, {
"type": "select",
"name": "account",
"label": "AccountType",
"options": [
"current",
"savings"
],
"validations": [{
"regex": "",
"msg": ""
}]
}, {
"type": "checkbox",
"name": "interest",
"label": "Interests",
"options": [
"Cooking",
"Reading"
],
"validations": [{
"regex": "",
"msg": ""
}]
}],
"users": [
"Testing"
]
}}
And call post with:
context.simplehttp.makePost(url,JSON.stringify(param),header,parser);
And my call back
function parser(context, event) {
context.console.log("Handler https")
var result= JSON.parse(event.getresp);
if(result=="success"){
context.sendResponse("We have successfully stored your data");
}else{
context.sendResponse("We dont shoot");
}
}
But when, make the request post, but don't show me response in chat or in callback. What im doing wrong?
Sohan from Gupshup here.
The result from the API that you are using is this:
[{
"embedlink": "https://api.gupshup.io/sm/api/facebook/smartmsg/embed/66438dde-ec76-4d6e-a0d0-8cfc0c730e57",
"expired": false,
"fb-button": {
"title": "This is a test.",
"type": "web_url",
"url": "https://api.gupshup.io/sm/api/facebook/smartmsg/embed/66438dde-ec76-4d6e-a0d0-8cfc0c730e57",
"webview_height_ratio": "tall"
},
"id": "66438dde-ec76-4d6e-a0d0-8cfc0c730e57",
"signed-for": {
"display": "Testing",
"subdisplay": "Testing"
},
"smid": "1009"
}]
Thus when you do:
var result= JSON.parse(event.getresp);
if(result=="success"){
context.sendResponse(result) will display the entire JSON that you see above. To display the 'expired' field you can use result.expired.
Check this document for more information.

Bluemix : API Management - error "Swagger jsonReference". This is not supported. How do I fix it?

I've tried to import swagger document in json format. I got the error
The field paths["/namespaces"].get.responses["401"] uses a Swagger
jsonReference. This is not supported. Remove this field, or in-line
the referenced JSON instead, and resubmit the request.
(I also attached the screenshot for just incase). The code snippet which might caused the error is like this:
"401": {
"$ref": "#/responses/UnauthorizedRequest"
},
"500": {
"$ref": "#/responses/ServerError"
}
What's wrong with this contents ? Appreciate if you could point me how to fix the problem.
Thanks !
Ref: screenshot
This error looks like a limitation/bug but as the error description suggests, you can in-line the definition to get around it. Here is an example of inlining references in a Swagger document.
The following Swagger document has a $ref
"responses": {
"200": {
"description": "Task retrieved",
"schema": {
"$ref": "#/definitions/Task"
}
},
"404": {
"description": "Task not found"
}
}
...
"definitions": {
"Task": {
"type": "object",
"required": [
"deadline",
"description",
"status"
],
"properties": {
"description": {
"type": "string",
"example": "Make an app for Demo"
},
"status": {
"type": "string",
"example": "Created"
},
"deadline": {
"type": "string",
"format": "date",
"example": "01/15/16"
}
}
}
After inlining the $ref definition, the Swagger document would look like this:
"responses": {
"200": {
"description": "Task retrieved",
"schema": {
"type": "object",
"required": [
"deadline",
"description",
"status"
],
"properties": {
"description": {
"type": "string",
"example": "Make an app for Demo"
},
"status": {
"type": "string",
"example": "Created"
},
"deadline": {
"type": "string",
"format": "date",
"example": "01/15/16"
}
}
}
},
"404": {
"description": "Task not found"
}
}