I am trying create an equivalent of create/update trigger used in traditional RDBMs. create_ts is being created fine, however update_ts part is not working for me.
"updates": {
"add_ts": "function(doc, req)
{ if(!doc){
var result=JSON.parse(req.body);
result.created_ts=new Date();
return [result, 'Created']
}
doc.update_ts=new Date();
return [doc,'Updated'];
}"
},
The document creates all right:
curl -X POST $COUCHDB_URL/mobile_gateway/_design/devicetokens/_update/add_ts -d ' {"_id":"aaaa", "boris":"Ioffe"} '
{
"_id": "aaaa",
"_rev": "7-70069ed48a5fa2a571b5ad83067010b9",
"boris": "Ioffe",
"created_ts": "2018-12-24T20:24:58.064Z"
}
curl -X PUT $COUCHDB_URL/mobile_gateway/_design/devicetokens/_update/add_ts -d ' {"_id":"aaaa", "boris":"Loffe"} '
{"error":"conflict","reason":"Document update conflict."}
I feel I am missing something fundamental in my understanding couchdb document updates.
Moving comment to answer based on OP request.
When the request to an update handler includes a document ID in the URL, the server will provide the function with the most recent version of that document. Based on this sentence at the beginning of the second paragraph, it seems that you have left off the id in the url in the PUT and have only provided the body.
Your request should look something like this
curl -X PUT $COUCHDB_URL/mobile_gateway/_design/devicetokens/_update/add_ts/aaaa -d ' {"_id":"aaaa", "boris":"Loffe"}
Related
New to REST API, and struggling a little.
I learned recently from here and here that I can receive a JSON describing an issue by calling a REST API of the form <JIRA_BASE_URL>/rest/api/2/issue/{issueIdOrKey}, e.g.:
curl -s -X GET -u super_user:super_password https://jira.server.com/rest/api/2/issue/TEST-12
Is there a way I can query many issues at once if I have a list of issue-ids, e.g. ["TEST-12", "TEST-13", "TEST-14"]?
I'm specifically interested in getting the summary field of each issue in my list of issue-ids. I.e. I'm trying to create a map of [issue-id:summary]. I currently do this by invoking the above curl command in a loop for each issue-id in my list. But I observe that this takes a long time, and I wonder if performance might be improved if there's a way to do a "bulk get" -- if such a feature exists.
Give the JQL Search API endpoint a try:
https://jira-url/rest/api/latest/search?fields=summary&jql=key%20in%20(TEST-12,%20TEST-13)
The fields parameter limits the fields returned, and the jql parameter lists out an array of the issue keys you'd like to retrieve.
The response looks like this:
{
...
"startAt": 0,
"maxResults": 50,
"total": 2,
"issues": [
{
...
"key": "TEST-12",
"fields": {
"summary": "TEST-12 Summary"
}
},
{
...
"key": "TEST-13",
"fields": {
"summary": "TEST-13 Summary"
}
}
]
}
I have a collection of documents with generated identifiers. The question is: is it possible to get a list of identifiers without querying all documents data? Is it better to store these keys in a separate collection?
The answer depends on which API you're trying to use.
For mobile/web SDKs there is no way to do what you're asking for since these clients do not support projections of any kind.
For server SDKs you can do an empty projection, i.e.
db.collection('foo').select()
In this case the server will send you the documents that match, but will omit all fields from the query result.
For the REST API you can do the equivalent with a runQuery that includes a field mask of '__name__', like so:
curl -vsH 'Content-Type: application/json' \
--data '{
"parent": "projects/my-project/databases/(default)",
"structuredQuery":{
"from": [{"collectionId": "my-collection"}],
"select": {
"fields": [{"fieldPath":"__name__"}]
}
}
}' \
'https://firestore.googleapis.com/v1beta1/projects/my-project/databases/(default)/documents:runQuery'
Substitute my-project and my-collection as appropriate. Note that the "collectionId" in the "from" is only the right most name component. If you want keys in a subcollection the REST API wants the parent document name in the "parent" field.
On node.js runtime you can get the list of document IDs like this
const documentReferences = await admin.firestore()
.collection('someCollection')
.listDocuments()
const documentIds = documentReferences.map(it => it.id)
I had a need to fetch only ids without pulling fields of all documents in Python.
I used google-cloud-firestore 2.6.0 package.
select and field_paths keywords were the very important part of this query. They allowed me to process without downloading all documents.
from google.cloud import firestore_v1
db = firestore_v1.Client()
items = db.collection("your_collection_name").select(field_paths=[]).get()
ids = [item.id for item in items]
print(ids)
Here is some JavaScript and a little React code that seems to be working for me with the V1 REST API's runQuery using client SDK bearer tokens in the browser (Chrome). This is patterned off of Gil Gilbert's answer. However, note that parent does not appear in the body by the structured query, and unlike some other answers on Stack Overflow, there is no API key necessary.
const [token, setToken] = useState("");
useEffect(() => {
if (!token) firebase.auth().currentUser.getIdToken(true).then(setToken);
}, [token]);
const getCollectionAsync = useCallback(async collection => {
try {
if (!token) return [];
const parent = `projects/${projectId}/databases/(default)/documents`;
const url = `https://firestore.googleapis.com/v1/${parent}:runQuery`;
const Authorization = `Bearer ${token}`;
const headers = {Authorization, "Content-Type": "application/json"};
const body = {structuredQuery: {from: [{collectionId: collection}],
select: {fields: [{"fieldPath": "__name__"}]}}};
const response = await fetch(url,
{method: "POST", headers, body: JSON.stringify(body)});
const json = await response?.json?.();
return json;
} catch (error) {
console.error(error);
return [];
}
}, [cache, token]);
I'm getting different response when querying a service from a specific field, depending on if I'm using CURL or running app.service().find() on the server. I'm using MongoDB, here's an example of my results.
// Here's a list of all data in the service
curl 'http://192.168.99.100:3030/players/'
[
{"_id":"5a04bd4eee3648000fbad08f","userId":"59f8e18c14b066000ff63cbb","gameId":"5a04bd4eee3648000fbad08e","isHost":true,"handle":"dude"},
{"_id":"5a0b41bbacd285000ffb310c","userId":"59f8e18c14b066000ff63cbb","gameId":"5a0b41bbacd285000ffb310b","isHost":true,"handle":"dude"},
{"_id":"5a0b440aacd285000ffb310d","gameId":"5a0b41bbacd285000ffb310b"},
{"_id":"5a0b44f7acd285000ffb310e","gameId":"5a0b41bbacd285000ffb310b"},
{"_id":"5a0b498dc31cea000fef17dd","gameId":"5a0b41bbacd285000ffb310b","userId":"5a0b4117acd285000ffb310a","handle":"dog"},
{"_id":"5a0b4a76c31cea000fef17de","gameId":"5a0b41bbacd285000ffb310b","userId":"5a0b4117acd285000ffb310a","handle":"dog"}
]}
// If I query on the gameId 5a0b41bbacd285000ffb310b from curl, it only returns
// 4 of 5 rows skipping the first player
curl 'http://192.168.99.100:3030/players/?gameId=5a0b41bbacd285000ffb310b'
[
{"_id":"5a0b440aacd285000ffb310d","gameId":"5a0b41bbacd285000ffb310b"},
{"_id":"5a0b44f7acd285000ffb310e","gameId":"5a0b41bbacd285000ffb310b"},
{"_id":"5a0b498dc31cea000fef17dd","gameId":"5a0b41bbacd285000ffb310b","userId":"5a0b4117acd285000ffb310a","handle":"dog"},
{"_id":"5a0b4a76c31cea000fef17de","gameId":"5a0b41bbacd285000ffb310b","userId":"5a0b4117acd285000ffb310a","handle":"dog"}
]}
// When I run this query server side passing gameID = 5a0b41bbacd285000ffb310b
// I only get the first created player in the list.
var gameID = hook.result._id;
app.service('players').find({
query:{
gameId: gameID,
}
}).then(players => {
console.log(players);
});
[
{"_id":"5a0b41bbacd285000ffb310c","userId":"59f8e18c14b066000ff63cbb","gameId":"5a0b41bbacd285000ffb310b","isHost":true,"handle":"dude"}
]
My filters are blank, and the hooks don't remove data they just get the "handle" from the userId.
It seems like the query might not work because there may be a difference in datatype. As in the fieldname gameId or the data stored.
The first ID in Players 5a0b41bbacd285000ffb310c was given the gameId from the server side after the game object was created using a hook. Like this:
// This hook is run after a new game is created
return function joinNewGame (hook) {
var userid = hook.params.user._id;
var gameid = hook.result._id;
return hook.app.service('players').create({
userId: userid,
gameId: gameid,
isHost: true,
}).then(player => {
return hook;
});
};
The other players were written in a post request where gameId was passed as a string.
curl 'http://192.168.99.100:3030/players/' -H 'Content-Type: application/json' --data-binary '{ "gameId":"5a0b41bbacd285000ffb310b"}'
Thanks for your help!
After some additional testing I find that it's a typing error. The typeof gameId for the server side created entry was "Object" while the client side was "String".
So my current fix is to cast the id's to strings before storing / querying from the server. Is this the best way to do Many to One relationships in the ORM? The Docs doesn't go into detail.
Update: Here's a different fix, I created a new hook that converts string to a MongoDB Object ID when storing the data, pass the fields that need to be updated in an array.
var ObjectID = require('mongodb').ObjectID;
module.exports = function (fields, options = {}) { // eslint-disable-line no-unused-vars
return function stringToObjectId (hook) {
// For each passed field
for(var i = 0; fields.length > i; i++){
// Test if there's queries on the field, then convert
if(hook.params.query && typeof hook.params.query[fields[i]] != "Object"){
hook.params.query[fields[i]] = ObjectID(hook.params.query[fields[i]]);
}
// Test if there's data on the field, then convert
if(hook.data && typeof hook.data[fields[i]] != "Object"){
hook.data[fields[i]] = ObjectID(hook.data[fields[i]]);
}
}
return Promise.resolve(hook);
};
};
I am currently building an API which uses the JSON patch specification to do partial updates to MongoDB using the Mongoose ORM.
I am using the node module mongoose-json-patch to apply patches to my documents like so:
var patchUpdate = function(req, res){
var patches = req.body;
var id = req.params.id;
User.findById(id, function(err, user){
if(err){ res.send(err);}
user.patch(patches, function(err){
if(err){ res.send(err);}
user.save(function(err){
if(err) {res.send(err);}
else {res.send("Update(s) successful" + user);}
});
});
});
};
My main issues occur when I am trying to remove or replace array elements with the JSON patch syntax:
var patches = [{"op":"replace", "path": "/interests/0", "value":"Working"}]
var user = {
name: "Chad",
interests: ["Walking", "Eating", "Driving"]
}
This should replace the first item in the array ("Walking") with the new value ("Working"), however I can't figure out how to validate what is actually being replaced. If another request removed /interests/0 prior to the patch being applied, "Eating" would be replaced by "Working" instead of "Walking", which would no longer exist in the array.
I would like to be sure that if the client thinks he is editing "Walking", then he will either successfully edit it, or at least get an error.
After running into the same issue like this myself i'll share my solution. The spec (described here) describes six operations, one of which is test. The source describes the test operation as
Tests that the specified value is set in the document. If the test fails, then the patch as a whole should not apply.
To ensure that you're changing the values that you're expecting you should validate the state of the data. You do this by preceeding your replace or remove operation with a test operation, where the value is equal to the expected data state. If the test fails, the following operations will not be executed.
With the test operation your patch data will look like this:
var patches = [
{"op":"test", "path": "/interests/0", "value": currentValue}, //where currentValue is the expected value
{"op":"replace", "path": "/interests/0", "value":"Working"}
]
I try to create/update nodes via the REST API with Cypher's MERGE-statement. Each node has attributes of ca. 1kb (sum of all sizes). I create/update 1 node per request. (I know there are other ways to create lots of nodes in a batch, but this is not the question here.)
I use Neo4j community 2.1.6 on a Windows Server 2008 R2 Enterprise (24 CPUs, 64GB) and the database directory resides on a SAN drive. I get a rate of 4 - 6 nodes per second. Or in other words, a single create or update takes around 200ms. This seems rather slow for me.
The query looks like this:
MERGE (a:TYP1 { name: {name}, version: {version} })
SET
a.ATTR1={param1},
a.ATTR2={param2},
a.ATTR3={param3},
a.ATTR4={param4},
a.ATTR5={param5}
return id(a)
There is an index on name, version and two of the attributes.
Why does it take so long? And what can I try to improve the situation?
I could imagine that one problem is that every request must create a new connection? Is there a way to keep the http connection open for multiple requests?
For a query I'm pretty sure you can only use one index per query per label, so depending on your data they index usage might not be efficient.
As far as a persistent connection, that is possible, though I think it would depend on the library you're using to connect to the REST API. In the ruby neo4j gem we use the Faraday gem which has a NetHttpPersistent adapter.
The index is only used when you use ONE attribute with MERGE
If you need to merge on both, create a compound property, index it (or better use a constraint) and merge on that compound property
Use ON CREATE SET otherwise you (over-)write the attributes everytime, even if you didn't actually create the node.
Adapted Statement
MERGE (a:TYP1 { name_version: {name_version} })
ON CREATE SET
a.version = {version}
a.name = {name}
a.ATTR1={param1},
a.ATTR2={param2},
a.ATTR3={param3},
a.ATTR4={param4},
a.ATTR5={param5}
return id(a)
This is an example of how you can execute a batch of cypher queries from nodejs in one communication with the Neo4j.
To run it,
get nodejs installed (if you don't have it already)
get a token from https://developers.facebook.com/tools/explorer giving you access to user_groups
run it as > node {yourFileName}.js {yourToken}
prerequisites:
var request=require("request") ;
var graph = require('fbgraph');
graph.setAccessToken(process.argv[2]);
function now() {
instant = new Date();
return instant.getHours()
+':'+ instant.getMinutes()
+':'+ instant.getSeconds()
+'.'+ instant.getMilliseconds();
}
Get facebook data:
graph.get('me?fields=groups,friends', function(err,res) {
if (err) {
console.log(err);
throw now() +' Could not get groups from faceBook';
}
Create cypher statements
var batchCypher = [];
res.groups.data.forEach(function(group) {
var singleCypher = {
"statement" : "CREATE (n:group{group}) RETURN n, id(n)",
"parameters" : { "group" : group }
}
batchCypher.push(singleCypher);
Run them one by one
var fromNow = now();
request.post({
uri:"http://localhost:7474/db/data/transaction/commit",
json:{statements:singleCypher}
}, function(err,res) {
if (err) {
console.log('Could not commit '+ group.name);
throw err;
}
console.log('Used '+ fromNow +' - '+ now() +' to commit '+ group.name);
res.body.results.forEach(function(cypherRes) {
console.log(cypherRes.data[0].row);
});
})
});
Run them in batch
var fromNow = now();
request.post({
uri:"http://localhost:7474/db/data/transaction/commit",
json:{statements:batchCypher}
}, function(err,res) {
if (err) {
console.log('Could not commit the batch');
throw err;
}
console.log('Used '+ fromNow +' - '+ now() +' to commit the batch');
})
});
The log shows that a transaction for 5 groups is significantly slower than a transactions for 1 group but significantly faster than 5 transactions for 1 group each.
Used 20:38:16.19 - 20:38:16.77 to commit Voiture occasion Belgique
Used 20:38:16.29 - 20:38:16.82 to commit Marches & Randonnées
Used 20:38:16.31 - 20:38:16.86 to commit Vlazarus
Used 20:38:16.34 - 20:38:16.87 to commit Wijk voor de fiets
Used 20:38:16.33 - 20:38:16.91 to commit Niet de bestemming maar de route maakt de tocht goed.
Used 20:38:16.35 - 20:38:16.150 to commit the batch
I just read your comment, Andreas, do it is not applicable for you, but you might use it to find out if the time is spent in the communication or in the updates