DMS writing data to kinesis with changed format - postgresql

I am new to dms, but I have a scenario where the record updated in rds to be like this:
{
"user_id": 224,
"firstname": "dede",
"lastname": "gam",
"address": "Aswan, Egypt",
"email": "email#exm.com"
},
the dms task has the target as kinesis, however, the when I retrive the data from kinesis I get it like this:
{
"data": {
"user_id": 224,
"firstname": "dede",
"lastname": "gam",
"address": "Aswan, Egypt",
"email": "email#exm.com"
},
"metadata": {
"timestamp": "2023-01-31T18:42:47.515698Z",
"record-type": "data",
"operation": "update",
"partition-key-type": "schema-table",
"schema-name": "public",
"table-name": "users_cdc_test",
"transaction-id": 542923
}
}
I want it to be as it was:
{
"user_id": 224,
"firstname": "dede",
"lastname": "gam",
"address": "Aswan, Egypt",
"email": "email#exm.com"
},
so, I followed the aws document: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Kinesis.html#CHAP_Target.Kinesis.ObjectMapping
and updated the mapping rules to be:
{
"rules": [
{
"rule-type": "selection",
"rule-id": "857119726",
"rule-name": "857119726",
"object-locator": {
"schema-name": "%public",
"table-name": "%sers_cdc_test"
},
"rule-action": "include",
"filters": []
},
{
"rule-type": "object-mapping",
"rule-id": "2",
"rule-name": "DefaultMapToKinesis",
"rule-action": "map-record-to-record",
"object-locator": {
"schema-name": "puplic",
"table-name": "users_cdc_test"
}
}
]
}
and I restarted the dms task, but got the same structure again not the structure I am expecting, and idea?

Related

Loopback 3: Multiple HasOne relation on one model

So, I opened an issue here, coz in my opinion it should work as I think... but might be wrong so looking for another way
So, pretty much I have two models, Wedding and Person. The Wedding one has these relations set:
"people": {
"type": "hasMany",
"model": "person",
"foreignKey": "",
"options": {
"nestRemoting": true
}
},
"partner1": {
"type": "hasOne",
"model": "person",
"foreignKey": ""
},
"partner2": {
"type": "hasOne",
"model": "person",
"foreignKey": ""
}
And one of my wedding documents looks like this (I am using mongoDB if you cannot tell):
{
"_id": "5de78c76f89d1a8ad4091ca5",
"date": "2019-12-04T10:37:42.000Z",
"userId": "5de78c76f89d1a8ad4091ca4",
"created": "2019-12-04T10:37:42.720Z",
"partner1Id": "5de78c77f89d1a8ad4091ca6",
"partner2Id": "5de78c77f89d1a8ad4091ca7"
}
So, when I set include filter and do:
{ "include": ["partner1", "partner2"]}
in my loopback API explorer on
http://localhost:3000/api/weddings/5de78c76f89d1a8ad4091ca5
I get:
{
"date": "2019-12-04T10:37:42.000Z",
"id": "5de78c76f89d1a8ad4091ca5",
"userId": "5de78c76f89d1a8ad4091ca4",
"created": "2019-12-04T10:37:42.720Z",
"partner1Id": "5de78c77f89d1a8ad4091ca6",
"partner2Id": "5de78c77f89d1a8ad4091ca7",
"partner1": {
"id": "5de78c77f89d1a8ad4091ca7",
"fullName": "Jessica Alba",
"spouse": "spouse2",
"contacts": [],
"verified": false,
"created": "2019-12-04T10:37:43.292Z",
"updated": "2019-12-04T10:37:43.292Z",
"userId": "5de78c76f89d1a8ad4091ca4",
"weddingId": "5de78c76f89d1a8ad4091ca5"
},
"partner2": {
"id": "5de78c77f89d1a8ad4091ca7",
"fullName": "Jessica Alba",
"spouse": "spouse2",
"contacts": [],
"verified": false,
"created": "2019-12-04T10:37:43.292Z",
"updated": "2019-12-04T10:37:43.292Z",
"userId": "5de78c76f89d1a8ad4091ca4",
"weddingId": "5de78c76f89d1a8ad4091ca5"
}
}
But, I am expecting this:
{
"date": "2019-12-04T10:37:42.000Z",
"id": "5de78c76f89d1a8ad4091ca5",
"userId": "5de78c76f89d1a8ad4091ca4",
"created": "2019-12-04T10:37:42.720Z",
"partner1Id": "5de78c77f89d1a8ad4091ca6",
"partner2Id": "5de78c77f89d1a8ad4091ca7",
"partner1": {
"id": "5de78c77f89d1a8ad4091ca6",
"fullName": "Michael Knight",
"spouse": "spouse1",
"contacts": [],
"verified": false,
"created": "2019-12-04T10:37:43.292Z",
"updated": "2019-12-04T10:37:43.292Z",
"userId": "5de78c76f89d1a8ad4091ca4",
"weddingId": "5de78c76f89d1a8ad4091ca5"
},
"partner2": {
"id": "5de78c77f89d1a8ad4091ca7",
"fullName": "Jessica Alba",
"spouse": "spouse2",
"contacts": [],
"verified": false,
"created": "2019-12-04T10:37:43.292Z",
"updated": "2019-12-04T10:37:43.292Z",
"userId": "5de78c76f89d1a8ad4091ca4",
"weddingId": "5de78c76f89d1a8ad4091ca5"
}
}
Any ideas as of why? why do I get the same two records for partner1 and partner2?
Instead of using "hasOne", use "belongsTo".

How to check $exists in one to many relationship mongodb sails js?

Is there any way to find records that collection is empty?
For example please find below array. I want only that records with index "companydata" is empty. and also how can i get data that does not have empty "companydata" data.
Thanks in advance.
[
{
"company_id": {
"company_name": "C2",
"slug": "c2",
"is_organized": 1,
"status": "1",
"id": "5adf158f547f7f0314ca8b56",
"companydata": []
},
"user_id": "5ab889aee74a151b50d04ec1",
"status": "0",
"id": "5ae014e7432e85298081be0b"
},
{
"company_id": {
"company_name": "My Compnay",
"slug": "my-compnay",
"is_organized": 1,
"status": "1",
"id": "5ad442d98a0e0c1358ca93df",
"companydata": [
{
"name": "Bhavesh Amin",
"company_id": "5ad442d98a0e0c1358ca93df",
"status": "0",
"id": "5ad442da8a0e0c1358ca93e0"
}
]
},
"user_id": "5ab889aee74a151b50d04ec1",
"status": "0",
"id": "5ae01388432e85298081bdf8"
},
{
"company_id": {
"company_name": "Organization Name",
"slug": "organization-name",
"is_organized": 1,
"status": "1",
"id": "5ad08f9b938d1131eceea624",
"companydata": [
{
"name": "Helen H. Langley",
"company_id": "5ad08f9b938d1131eceea624",
"status": "1",
"id": "5ad08f9b938d1131eceea625"
}
]
},
"user_id": "5ab889aee74a151b50d04ec1",
"status": "0",
"id": "5ad42a5f52851a2b1449db2d"
},
]

MongoDB EmbedsMany Query

So I'm using loopback to create a RESTful node.js API. I'm using mLab for my database. I'm trying to query data that is embedded inside of my Account model.
An Account model has a property called favorites which has an array of Songs.
Here is an example:
{
"_id": {
"$oid": "58512c1b7c05c491b2cff1f6"
},
"firstName": "Matt",
"lastName": "Thomas",
"username": "mattietea",
"password": "$2a$10$dcquPHK48H5Q7FcaTyzgf.vnrUOgKpBENS9fSa0hjoaASs0uyOWba",
"email": "mattcthomas#me.com",
"_favorites": [
{
"title": "Ellusive & Eljay - Like This Before...",
"artist": "Ellusive",
"audio": "https://api.soundcloud.com/tracks/236100069/stream?client_id=90d140308348273b897fab79f44a7c89",
"image": "https://i1.sndcdn.com/artworks-000145902859-5467c0-t500x500.jpg",
"download": "https://api.soundcloud.com/tracks/236100069/download",
"url": "https://soundcloud.com/iamellusive/like-this-before",
"created": {
"$date": "2016-12-13T19:00:50.136Z"
},
"genres": {
"hipHop": 100
},
"id": "5850456247ef7e110099748b"
},
{
"title": "Dazed and Confused",
"artist": "Sir Skiff",
"audio": "https://api.soundcloud.com/tracks/103322663/stream?client_id=90d140308348273b897fab79f44a7c89",
"image": "https://i1.sndcdn.com/artworks-000054154948-m9cwii-t500x500.jpg",
"download": null,
"url": "http://soundcloud.com/skiffmusic/dazed-and-confused",
"created": {
"$date": "2016-12-13T08:49:59.128Z"
},
"genres": {
"hipHop": 100
},
"id": "584fb637fbf7df995f4341d3"
},
{
"title": "Runnat",
"artist": "Rory Fresco",
"audio": "https://api.soundcloud.com/tracks/258497600/stream?client_id=90d140308348273b897fab79f44a7c89",
"image": "https://i1.sndcdn.com/artworks-000157594908-cu10q4-t500x500.jpg",
"download": null,
"url": "http://soundcloud.com/roryxxx/runnat-2",
"created": {
"$date": "2016-12-13T08:50:52.397Z"
},
"genres": {
"hipHop": 100
},
"id": "584fb66cfbf7df995f4341d4"
},
{
"title": "Breathe (Prod. by Yuri Beat$)",
"artist": "Chiddy Bang",
"audio": "https://api.soundcloud.com/tracks/106182458/stream?client_id=90d140308348273b897fab79f44a7c89",
"image": "https://i1.sndcdn.com/artworks-000055716908-f9qdau-t500x500.jpg",
"download": null,
"url": "http://soundcloud.com/chiddybang/breathe-prod-by-yuri-beat",
"created": {
"$date": "2016-12-13T08:32:36.447Z"
},
"genres": {
"hipHop": 100
},
"id": "584fb224fbf7df995f4341ce"
}
]
}
I have an endpoint set up which returns all the favorites
api/Accounts/{id}/favorites
But I can't seem to query them. When I query all the songs from api/Songs it works but not when I query favorites.
The query I'm using is:
{'order': 'created DESC'}
I've tried multiple other queries, and none work.
Thank you!

what is the date format from stash api?

In the below json response, what is the date format for createdDate and updatedDate? I am not sure how to work in reverse to find what format the api is using for date. I couldn't find this any where in the documentation.
{
"size": 1,
"limit": 25,
"isLastPage": true,
"values": [
{
"id": 101,
"version": 1,
"title": "Talking Nerdy",
"description": "It’s a kludge, but put the tuple from the database in the cache.",
"state": "OPEN",
"open": true,
"closed": false,
"createdDate": 1359075920,
"updatedDate": 1359085920,
"fromRef": {
"id": "refs/heads/feature-ABC-123",
"repository": {
"slug": "my-repo",
"name": null,
"project": {
"key": "PRJ"
}
}
},
"toRef": {
"id": "refs/heads/master",
"repository": {
"slug": "my-repo",
"name": null,
"project": {
"key": "PRJ"
}
}
},
"locked": false,
"author": {
"user": {
"name": "tom",
"emailAddress": "tom#example.com",
"id": 115026,
"displayName": "Tom",
"active": true,
"slug": "tom",
"type": "NORMAL"
},
"role": "AUTHOR",
"approved": true
},
"reviewers": [
{
"user": {
"name": "jcitizen",
"emailAddress": "jane#example.com",
"id": 101,
"displayName": "Jane Citizen",
"active": true,
"slug": "jcitizen",
"type": "NORMAL"
},
"role": "REVIEWER",
"approved": true
}
],
"participants": [
{
"user": {
"name": "dick",
"emailAddress": "dick#example.com",
"id": 3083181,
"displayName": "Dick",
"active": true,
"slug": "dick",
"type": "NORMAL"
},
"role": "PARTICIPANT",
"approved": false
},
{
"user": {
"name": "harry",
"emailAddress": "harry#example.com",
"id": 99049120,
"displayName": "Harry",
"active": true,
"slug": "harry",
"type": "NORMAL"
},
"role": "PARTICIPANT",
"approved": true
}
],
"link": {
"url": "http://link/to/pullrequest",
"rel": "self"
},
"links": {
"self": [
{
"href": "http://link/to/pullrequest"
}
]
}
}
],
"start": 0
}
Just making a note that in my case, it is a UNIX timestamp, but I have to remove three trailing zeroes. E.g. the data looks like this:
"createdDate":1555621993000
If interpreted as a UNIX timestamp, that would be 09/12/51265 # 4:16am (UTC).
By removing the three trailing zeroes I get 1555621993, which is the correct time 04/18/2019 # 9:13pm (UTC)
Your mileage may vary but that was a key discovery for me :)
It looks like a UNIX timestamp.
https://en.wikipedia.org/wiki/Unix_time

Elasticsearch nested query

I'm new to elasticsearch, managed to set it up and import recordset from my mongodb collection using the river plugin. For a start, I want to query against the "desc" field but just can't manage to get the query .. not sure if the problem is driven by the way index was defined.. can anyone help please?
Sample recordset in elastic search looks like this
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 107209,
"max_score": 1,
"hits": [
{
"_index": "shiv",
"_type": "shiv",
"_id": "iG1eIzN7RGO7hFfxTlnLuA",
"_score": 1,
"_source": {
"_id": {
"$oid": "50901d7f485bf7bd1c000021"
},
"brand": "",
"category": {
"$ref": "categories",
"$id": {
"$oid": "4fbd2221758cb11d14000174"
}
},
"comments": [],
"count_comment": 0,
"count_fav": 2,
"count_hotness": 1.46,
"count_rekick": 0,
"count_share": 0,
"country": {
"$ref": "countries",
"$id": {
"$oid": "4fec98f7758cb18c6e0002c9"
}
},
"currency": "pound",
"desc": "A men's automatic watch, this Seamaster Bond model features a Co-Axial escapement and date function. Its blue dial is teamed with a stainless steel case and bracelet for a look that's sporty and refined.",
"gender": "male",
"ident": "omega-seamaster-diver-bond-men-s-automatic-watch---ernest-jones-1351622015",
"img_url": "http://s7ondemand4.scene7.com/is/image/Signet/5735793?$detail$",
"lifestyles": [
{
"$ref": "lifestyles",
"$id": {
"$oid": "508ff6ca485bf73112000060"
}
}
],
"location": "United Kingdom",
"owner": {
"$ref": "accounts",
"$id": {
"$oid": "50742fd8485bf74b7a00213f"
}
},
"price": 2400,
"store": "ernestjones.co.uk",
"tags": [
"ernest-jones",
"bond"
],
"timestamp_creation": 1351622015,
"timestamp_exp": 1356825600,
"timestamp_update": 1351622015,
"title": "Omega Seamaster Diver Bond men's automatic watch - Ernest Jones",
"url": "http%3A%2F%2Fwww.ernestjones.co.uk%2Fwebstore%2Fd%2F5735793%2Fomega%20seamaster%20diver%20bond%20men%27s%20automatic%20watch%2F%3Futm_source%3Dgooglebase%26utm_medium%3Dfeedmanager%26cm_mmc%3DFroogle-_-CKB-_-nurses_fobs-_-watches%26cm_mmca1%3Domega%26cm_mmca2%3Dmale%26cm_mmca3%3Dadult"
}
}
]
}
}
The mapping of the index "shiv" looks like
{
"shiv": {
"properties": {
"$oid": {
"type": "string"
}
}
}
}
Thanks again
There are lots of ways to query, have you tried a match query?
Using curl or a rest client of your choice...
http://[host]:9200/[index_name]/[doc_type]/_search
{
"query" : {
"match" : {
"desc" : "some value you want to find in desc"
}
}
}