I want rename to rename my dict key in mongodb.
normally it works like that db.update({'_id':id},{$rename:{'oldfieldname':newfieldname}})
My document structure looks like that
{
'data':'.....',
'field':{'1':{'data':....},'2':{'data'...}},
'more_data':'....',
}
if i want to set
a new field in field 1 i do db.update({'_id':id},{$set:{'field.0.1.name':'peter'}})
for field two it is 'field'.1.2.name'
i thought with the rename it should be similar but it isn't ... (like $rename:{'field'.0.1': 2}
Here's a flexible method for renaming keys in a database
Given a document structure like this...
{
"_id": ObjectId("4ee5e9079b14f74ef14ddd2f"),
"code": "130.4",
"description": "4'' Socket Plug",
"technicalData": {
"Drawing No": "50",
"length": "200mm",
"diameter: "20mm"
},
}
I want to loop through all documents and rename technicalData["Drawing No"] to technicalData["Drawing Number"]
Run the following javascript in the execute panel in (the excellent) RockMongo
function remap(x){
dNo = x.technicalData["Drawing No"];
db.products.update({"_id":x._id}, {
$set: {"technicalData.Drawing Number" : dNo},
$unset: {"technicalData.Drawing No":1}
});
}
db.products.find({"technicalData.Drawing No":{$ne:null}}).forEach(remap);
The code will also run in a mongo shell
Your question is unclear but it seems you'd like to rename a field name within an array.
The short answer is you can't. As stated in the docs, $rename doesn't expand arrays to find a matching name. It only works on top level fields.
What you can do to simulate rename is by copying the field and its data to the new name, and then deleting the original field. You might also need a way to account for potentially concurrent writes if you have a lot of writes to that object/field.
Related
I understand that index has a cost in firestore. Most of the time we simply store objects without really caring about index and even if we don’t want most of the fields to be indexed.
If I understand correctly, any field at any level are indexed. I.e. for the following document in pseudo json
{
"root_field1": "abc" (indexed)
"root_field2": "def" (indexed)
"root_field3": {
"Sub_field1: "ghi" (indexed)
"sub_field2: "jkl" (indexed)
"sub_field3: {
"Inner_field1: "mno" (indexed)
"Inner_field2: "pqr" (indexed)
}
}
Let’s assume that I have the following record
{
"name": "abc"
"birthdate": "2000-01-01"
"gender": "m"
}
Let’s assume that I just want the field "name" to be indexed. One solution (A), without having to specify every field is to define it this way (i.e. move the root fields to a sub level unindexed), and exclude unindexed from being indexed
{
"name": "abc"
"unindexed" {
"birthdate": "2000-01-01"
"gender": "m"
}
Ideally I would like to just specify a prefix such as _ to prevent each field to be indexed but there is no global solution for that.
{
"name": "abc"
"_birthdate": "2000-01-01"
"_gender": "m"
}
Is my solution (A) correct and is there a more elegant generic solution?
Thanks!
Accordinig to the documentation
https://cloud.google.com/firestore/docs/query-data/indexing
Add a single-field index exemption
Single-field index exemptions allow you to override automatic index
settings for specific fields in a collection. You can add a
single-field exemptions from the console:
Go to the Single Field Indexes section.
Click Add Exemption.
Enter a Collection ID and Field path.
Select new indexing settings for this field. Enable or disable
automatically updated ascending, descending, and array-contains
single-field indexes for this field.
Click Save Exemption.
I'm trying to use the Model.deleteMany() function from mongoose. I'm trying to do something like this:
MyModel.deleteMany({"_id": {$in: ['objectid 1', 'objectid 2'
But this only deletes the first element of the matches from DB (in this case, if 'objectid 1' exists in the DB, it deletes that, but if it isn't nothing happens (both n and deletedCount is 0 in the returned data from the function). I tried using something other than the _id as a query, and this worked. If I had three elements with the same 'name' field, I could delete these.
I tried _id both with and without quotation marks. I also tried converting the object id strings to actual object ids before passing them to deleteMany, but this had no difference either. I have also of course tried to google this, but everything I've found are examples of usage, where it looks like I'm doing the exact same thing as the various blog posts.
I haven't added much code here because I don't really see what else I could be adding. I'm already printing out the input to the $in object, and this is correct. The strangest thing, I think, is that the first element of the list is deleted. Is it treated as a deleteOne request for some reason? are there any config options I need?
As per request, I've added the query and the documents I'd hope to delete:
//Request
MemberModel.deleteMany({"_id": {$in: [
5ee4f6308631dc413c7f04b4,
5ee4f6308631dc413c7f04b5,
5ee4f6308631dc413c7f04b6
]}};
//Expected to be deleted
[
{
"_id": "5ee4f62f8631dc413c7f04b5",
"firstName": "Name",
"lastName": "Nameson",
"__v": 0
},
{
"_id": "5ee4f62f8631dc413c7f04b6",
"firstName": "Other",
"lastName": "Person",
"__v": 0
}
]
If you have any ideas for what I could try, that would be much appreciated.
I have documents that I want to index into Elasticsearch with an existing unique "id" field.
I get an array of documents from a REST api endpoint ( eg.: http://some.url/api/products) in no particular order and if a document with the _id already exists in Elasticsearch it should update and reindex the document.
I want to create a new document if no document with the _id in Elasticsearch exists and then update a document, if it matches with an existing document in Elasticsearch.
This could be done with:
PUT products/product/un1qu3-1d-b718-105973677e95
{
"id": "un1qu3-1d-b718-105973677e95",
"state": "packaged"
}
The basic idea is to use the provided "id" field to create or update a document. Extraction of _id from document fields seems deprecated (link). But the indexing/ reindexing of documents with the "id" field can be done manually very easy with the kibana dev tools, with postman or a cURL request.
I want to achieve this (re-)indexing of documents that I receive over this api endpoint programmatically.
Is it possible to achieve this with logstash or a simple cronjob? Does Elasticsearch provide any functionality for this? Or do I need to write some custom backend to achieve this?
I thought of either:
1) index the document into Elasticsearch with the "id" field of my document or
2) find an Elasticsearch query that first searches for the document with the specific "id" field and then updates the document.
I was unable to find a solution for either way and have no clue how a good approach would look like.
Can anyone point me into the right direction on how to achieve this, suggest a better approach or provide a solution?
Any help much appreciated!
Update
I solved the problem with the help of the accepted answer. I used Logstash, the Http_poller input plugin, this article: https://www.elastic.co/blog/new-way-to-ingest-part-1 and this elastic.co question: https://discuss.elastic.co/t/upsert-with-logstash/59116
My output of logstash looks like this at the moment:
output {
elasticsearch {
index => "products"
document_type => "product"
pipeline => "rename_id"
document_id => "%{id}"
doc_as_upsert => true
action => "update"
}
Update 2
just for the sake of completeness I added the "rename_id" pipeline
{
"rename_id": {
"description": "_description",
"processors": [
{
"set": {
"field": "_id",
"value": "{{id}}"
}
}
]
}
}
It works this way!
Thanks alot!
Peter,
If I understand correctly, you want to ingest your documents into elastic search and will have some updates in future for these documents ?
If that's the case,
- Use your documents primary key as id for elastic documents.
- You can ingest entire document with updated values, elastic will replace the previous document with new one. given the primary key is same. Old document with same id will be deleted.
We use this approach for our search data.
you can use ingest pipelines to extract the id from the body and the _create endpoint to only create a document if it does not exist. Minor note: If you could specify the id on the client side indexing would be faster, as adding a pipeline adds a certain overhead.
PUT _ingest/pipeline/my_pipeline
{
"description": "_description",
"processors": [
{
"set": {
"field": "_id",
"value": "{{id}}"
}
}
]
}
PUT twitter/tweet/1?op_type=create&pipeline=my_pipeline
{
"foo" : "bar",
"id" : "123"
}
GET twitter/tweet/123
# this call will fail
PUT twitter/tweet/1?op_type=create&pipeline=my_pipeline
{
"foo" : "bar",
"id" : "123"
}
You can use script to UPSERT (update or insert) your document
PUT /products/product/un1qu3-1d-b718-105973677e95/_update
{
"script": {
"inline": "ctx._source.state = \"packaged\"",
"lang": "painless"
},
"upsert": {
"id": "un1qu3-1d-b718-105973677e95",
"state": "packaged"
}
}
Above query find the document with _id = "un1qu3-1d-b718-105973677e95"
if it is able to find any document then it will update state to "packaged" otherwise create a new document with field "id" and "state" (you can insert as many fields as you want).
In mongodb i saved document like this.
"Angela_Merkel": {
"birthPlace": "Hamburg,West_Germany",
"thumbnail": "http://commons.wikimedia.org/wiki/Special:FilePath/Angela-Merkel-2014.jpg?width=300",
"almaMater": "Leipzig_University",
"birthDate": "1954-07-17",
"spouse": "Joachim_Sauer"
}
There are many person's information like this way. Now if I want to get all the information of "Angela_Merkel" or only a particular like "birthdDate" of "Angela_Merkel" then what will be the query?
Like chridam says would be more practical that you refactor your documents like this:
{"name": "Angela_Merkel",
"birthPlace": "Hamburg,West_Germany",
"thumbnail": "http://commons.wikimedia.org/wiki/Special:FilePath/Angela-Merkel-2014.jpg?width=300",
"almaMater": "Leipzig_University",
"birthDate": "1954-07-17",
"spouse": "Joachim_Sauer"
}
Inside the "people" collection (Its a convention name the collections as plurals, beeing a set of documents)
Then you could do:
db.people.find({"name":"Angela Merkel"},{"_id":0," "birthdDate":1 })
To get:
{"birthDate": "1954-07-17"}
Anyway you can get with your document what you want this way:
"birthDate":
db.person.find({"Angela_Merkel.birthDate": "1954-07-17"})
or all the info:
db.person.find({"Angela_Merkel": { $exists: true}})
But doesn't have much sense do it this way.
PD: Edited to include last part.
A mongo db has documents that look like:
{
"_id": : ObjectId("55cb43e8c78b04f43f2eb503"),
<some fields>
"topics": {
"test/23/result": 149823788,
"test/27/result": 147862733,
"input/misc/test": 14672882
}
}
I need to find all documents that have a topics field that contains a particular key. i.e. find all documents that have a topics.key = "test/27/result"
I've tried a number of things but none work yet, neither attempt below work,
they return no records event though some should match:
db.collName.find({"topics.test/27/result": {$exists:true}});
db.collName.find({"topics.test\/27\/result": {$exists:true}});
How can I make the query work?
The slash characters are inserted by another process. They are mqtt topic names.
I found the solution to my problem:
I was building the query wrong in my code. In the example below, evtData.source contains the key name to search for, i.e. 'test/27/result'
The query methodology that works for me is:
var query = {};
query['topics.' + evtData.source] = {$exists: true};
db.collName.find(query)