I am trying to create a data pipeline to redshift using AWS Datapipeline.
I tried to import a definition.
{
"objects":[
{"id" : "first",
"type" : "SqlActivity",
"database" : {"ref":"RedshiftDatabase"},
"clusterId" : "",
"username" : "",
"password" : "",
"databaseName" : "",
"script" : "unload ('select _id from device_id_match LIMIT 100') to 's3://my_bucket' credentials
'aws_access_key_id=;aws_secret_access_key=' delimiter as ',' ",
"schedule" : { "ref": "Hour" },
"startAt": "FIRST_ACTIVATION_DATE_TIME",
"period": "1 hours",
"occurrences": "3",
"queue" : "priority"}]
}
But got the following error-
I am unable to resolve the error.
Yeah, as #Junren said the 'name' field is missing. This seems to be a very old version of pipeline definition export. Normally database is defined as a standalone resource, rather than inside the SqlActivity block.
Why don't you just use the Architect (data pipeline web UI) to build a pipeline from scratch?
try to add "name" field to SqlActivity object. It should unblock you from this error.
Related
I've a complex collection in MongoDB. In this, I want to query departments data based on the subDeptName. I used query like below db.getCollection('users').find({"departments.subDeptName" : "Java"}), but its fetching all the array elements of the department. I only wants to query such department where "subDeptName" : "Java". How can we do that ?
{
"firstName" : "John",
"lastName" : "Kerr",
"email" : "john.kerr#gmail.com",
"countryName" : "USA",
"usRestrictionSw" : "N",
"status" : "Active",
"effDate" : ISODate("2012-08-24T01:46:33.000Z"),
"departments" : [
{
"subDeptCd" : "AB",
"languageCd" : "en",
"desc" : "Development Group",
"subDeptName" : "Java",
"status" : "Active"
},
{
"subDivisionAlpha2Cd" : "KG",
"subDivisionCd" : "B",
"languageCd" : "ru",
"desc" : "Testing Group",
"subDeptName" : "Python",
"status" : "Active"
},
..........
..........
..........
..........
}
}
db.getCollection('users').find({"departments.subDeptName" : "Java"})
Will match all users with a sub deperament java.
Easiest way to achieve the result you want is this:
db.getCollection('users').aggregate([
{
$unwind: "$departments"
},
{
$match: {
"departments.subDeptName" : "Java"
}
])
Now you can also add a $project phase to get only the specific fields you want.
Try this:-
db.getCollection('users').find({"departments.subDeptName" : "Java"})
i have checked this on command line work fine.
Please check your mongo version and mongo shell version
mongod --version (check mongo version)
mongo --version // check mongo shell version.
And also check error log for have unother issue.
I am exporting document using "Sudio 3T" and it generates below JSON.
{ "_id" : { "$oid" : "5aa8bf0077bbfe296c1727de" }, "_t" : "DiscreteProperty", "Name" : "AHRI_bCertified"}
Next I using migration tool import JSON to Azure CosmosDB. Result is, it uploads but not viewing the document because it doesn't _id field.
Note: I am not providing anything in "Id Field" of target screen(CosmosDB).
I followed your steps but did't reproduce your issue.Please refer to my details.
sample .json file:
[
{
"_id" : { "$oid" : "5aa8bf0077bbfe296c1727de" },
"_t" : "DiscreteProperty",
"Name" : "AHRI_bCertified"
}
]
target info:
import result:
I get the response from Json like this
{
"id":"1090",
"title" : "My User",
"description" : "First User",
"country" : "US",
"state" : "FL",
"city" : "Miam"
"auth" : "Scott"
}
I need to write an update which updates only the fields which are changed.
If the updated JSON looks like
{
"id":"1090",
"title" : "New User",
"description" : "First User",
"country" : "US",
"state" : "Texas",
"city" : "Dallas"
"auth" : "Scott"}
I can achieve using below
Blog blogDB=datastore.find(Blog.class, "blog_ID", blog.getBlog_ID()).get();
Query<Blog> query = datastore.createQuery(Blog.class).field("_id").equal(blogDB.getId()); //Find the object that is in database
UpdateOperations<Blog> ops = datastore.createUpdateOperations(Blog.class).set("title", blog.getcountry()).set("country", blog.getcity()).set("city", ....
datastore.update(query, ops);
I don't want to write an UpdateOperations as shown above. Is there any more efficient way?
There is no dirty state tracking in Morphia, no. You'll need to track those changes manually.
I have two collection as bellow products has reference of user. i search product by name & in return i want combine output of product and user using map reduce method
user collection
{
"_id" : ObjectId("52ac5dd1fb670c2007000000"),
"company" : {
"about" : "This is textile machinery dealer",
"contactAddress" : [{
"address" : "abcd",
"city" : "52ac4bc6fb670c1007000000",
"zipcode" : "39as46as80"
},{
"address" : "abcd",
"city" : "52ac4bc6fb670c1007000000",
"zipcode" : "39as46as80"
}],
"fax" : "58784868",
"mainProducts" : "ads,asd,asd",
"mobileNumber" : "9537236588",
"name" : "krishna steels",
}
"user" : ObjectId("52ac4eb7fb670c0c07000000")
}
product colletion
{
"_id" : ObjectId("52ac5722fb670cf806000002"),
"category" : "52a2a9cc48a508b80e00001d",
"deliveryTime" : "10 days after received the ",
"price" : {
"minPrice" : "2000",
"maxPrice" : "3000",
"perUnit" : "5288ac6f7c104203e0976851",
"currency" : "INR"
},
"productName" : "New Mobile Solar Charger with Carabiner",
"rejectReason" : "",
"status" : 1,
"user" : ObjectId("52ac4eb7fb670c0c07000000")
}
This cannot be done. Mongo support Map Reduce only on one collection. You could try to fetch and merge in a java collection. Couple of days back I solved a similar problem using java collection.
Click to see similar response about joins and multi collection not supported in mongo.
This can be done using two map reduces.
You run your first MR and then you reduce out the second MR onto the results of the first.
You shouldn't do this though. JOINs are not designed to be done through MR, in fact it sounds like you are trying to do this MR with inline output which in itself is a very bad idea.
MRs are not designed to run inline to the application.
You would be better off doing the JOIN else where.
i have two tables history and jobs
my history table contains
> db.history.find()
{ "id" : "21", "browser" : "FF","os" : "Windows" "datetime" : "2013-11-26 17:04:21", "_id" : ObjectId("5294873d6b441e2c16000002") }
db.jobs.find()
{ "_id" : ObjectId("5289c147db9ed2b022f95a36"), "id" : "21", "launch" : "ertret", "names" : "234", "script" : "art-pagination" }
From the above two tables i need to get browser, launch, script and os by using common id: 21
How it is possible.
You can do it by using following two queries. It is not possible to get it with single query.
> db.history.find({'id':21}, {'browser':1, 'os':1})
> db.jobs.find({'id':21}, {'launch':1,'script':1 })