I've been playing around with Neo4j auto indexing and I'm quite confused...
What I'm using:
neo4j-community-1.8.M05
ruby1.9.3p194
The following code is also at https://github.com/LouisSayers/Neo4jChallenges
My intention is to create nodes which have unique 'uuid' values (generated by me). To ensure that they are unique, my thinking was to check against an index, and if the uuid value already exists, then I'd generate a new uuid value.
What I see is that for one of my uuids ('blahblahuuid') this works, but for the other it doesn't... and I don't know why (check in the Neo4j Monitoring tool console).
If you try the code below on a clean install of neo4j on ruby1.9.3 (no changes to properties etc) then hopefully you'll see the behaviour that I'm seeing!
Here's my code:
Gemfile:
source "http://rubygems.org"
source "http://gems.rubyforge.org"
source "http://gemcutter.org"
#gem install bundler -v 1.2.0.pre --pre
ruby '1.9.3'
gem 'bundler', '1.2.0.pre'
gem 'neography', :git => 'https://github.com/maxdemarzi/neography'
test.rb:
require "rubygems"
require "bundler"
require 'digest/sha1'
require 'securerandom'
Bundler.setup(:default)
require 'neography'
$neo = Neography::Rest.new("http://localhost:7474")
$neo.create_node_index("node_auto_index")
$neo.set_node_auto_index_status(true)
$neo.execute_script("g.clear()")
def generate_uuid(deviceUUID)
uuid = Digest::SHA1.base64digest deviceUUID
existing_node_with_uuid = $neo.get_node_auto_index('uuid', uuid)
until existing_node_with_uuid.nil?
uuid = Digest::SHA1.base64digest (deviceUUID.to_s + SecureRandom.base64(8))
existing_node_with_uuid = $neo.get_node_auto_index('uuid', uuid)
end
uuid
end
def generate_node(deviceUUID)
uuid = generate_uuid(deviceUUID)
secret = SecureRandom.base64(256)
currentTime = DateTime.now
$neo.create_node("uuid" => uuid, "deviceUUID" => deviceUUID,
"secret" => secret, "currentTime" => currentTime)
end
generate_node('blahblahuuid')
generate_node('blahblahuuid')
generate_node('4edb096b479588f6')
generate_node('4edb096b479588f6')
To run the code:
ruby test.rb
Listing results in the neo4j console:
start all = node(*)
return all;
# UPDATE
I figured out what was going wrong - firstly I didn't index uuid properly, I needed to add:
$neo.add_node_auto_index_property("uuid")
and as Max pointed out, I had a '/' in my uuid.
Also as pointed out by Russell Duhon (https://groups.google.com/forum/?fromgroups#!topic/neo4j/KyW0s5p5-fM)
I should have just used a uuid library - which is what I'm doing now (using the uuid gem - https://github.com/assaf/uuid/) and then I avoid having to check whether the same uuid exists - as the uuid should be unique.
I am also encoding all of my parameters that I pass through neography - as there is a chance that some of them may contain slashes, and '?&=' symbols which could interfere with the rest url!
Thank you Max and Russell for your help!
It is doing:
existing_node_with_uuid = $neo.get_node_auto_index('uuid', '7sZfZnCgz4sL/TkE4tPqb5+GgF0=')
Which is sending a get request to:
/db/data/index/node/node_auto_index/uuid/7sZfZnCgz4sL/TkE4tPqb5+GgF0=
I think that "/" is making a mess of things.
Related
I have a Django model called Comment (database is Postgres) which I populate with responses from an external API.
class Poster(models.Model):
username = models.TextField()
class Comment(models.Model):
created_dt = models.DateTimeField()
body = models.TextField()
poster = models.ForeignKey('Poster', on_delete=models.PROTECT, related_name = 'comments'
client_version = models.CharField(max_length=30)
Each comment object in the API response contains a client_version which is the version of the android client used by the poster to submit said comment.
I wish to write a query which tells me which client_version each poster submitted their most recent comment with, to identify which posters should be instructed to upgrade their client.
In this example scenario, let's assume that once a poster has upgraded their android client to a new version they can't go back to an earlier version; so the client_version from their latest comment is indicative of the version that poster is currently using.
To do this for a single poster is easy:
def get_latest_version(poster):
return Comment.objects.filter(
poster__username=poster
).order_by('-created_dt'
).values('poster__username', 'client_version').first()
>>> get_latest_version('kevin')
{'poster__username': 'kevin', 'client_version': 'v4.20.0'}
Obviously running this code for every individual poster is inefficient.
How can I use the Django ORM to perform a single postgres query, which returns something like:
[{'poster__username': 'kevin', 'client_version': 'v4.20.0'},
{'poster__username': 'perry', 'client_version': 'v4.21.0'},
{'poster__username': 'paul', 'client_version': 'v4.20.0'},
]
I have solved this with the method in this answer...
Comment.objects.all(
).order_by('poster__username', '-created_dt'
).distinct('poster__username'
).values('poster__username', 'client_version', 'created_dt')]
This also gives me the date of the latest comment.
I have been using nsupdate for a long time in various scripts dealing with dynamic DNS zone updates without any issue. I have always used TSIG to authenticate the requests against the DNS server, where the keys have been generated by ddns-confgen. That means that I didn't use key pairs like those generated by dnssec-keygen; rather, the keys file format is like the following:
key "ddns-key.my.domain.example" {
algorithm hmac-sha512;
secret "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefijklmnopqrstuvwxyzabcdefghij==";
};
The respective zone configuration then contains:
update-policy {
grant ddns-key.my.domain.example name host.my.domain.example ANY;
};
Now I have a more complicated task and try to solve it by a Perl script. I have studied the documentation of Perl::DNS::Update, Perl::DNS and Perl::DNS:RR:TSIG and have studied a lot of examples which should make the usage of those modules clear.
However, each example I saw, when coming to signing a request via TSIG, used key files in the format dnssec-keygen produces, and not key files in the format I have. And indeed, something like
$o_Resolver = new Net::DNS::Resolver(nameservers => ['127.0.0.1']);
$o_Update = new Net::DNS::Update('my.domain.example', 'IN');
$o_Update -> push(update => rr_del('host A'));
$o_Update -> push(update => rr_add('host 1800 IN A 192.0.2.1'));
$o_Update -> sign_tsig('/etc/bind/ddns-key.my.domain.example.key');
$o_Reply = ($o_Resolver -> send($o_Update));
does not work, producing the following message:
TSIG: unable to sign packet at /path/to/script.pl line 240.
unknown type "ddns-key.my.domain.example" at /usr/local/share/perl/5.20.2/Net/DNS/RR.pm line 669.
file /etc/bind/ddns-key.my.domain.example.key line 1
at /usr/local/share/perl/5.20.2/Net/DNS/RR/TSIG.pm line 403.
TSIG: unable to sign packet at /path/to/script.pl line 240.
I suppose I now have two options: Either use keys in the format dnssec-keygen produces, which seem to be directly usable with Net::DNS and its friends, or construct the TSIG key manually as shown in the docs:
my $key_name = 'tsig-key';
my $key = 'awwLOtRfpGE+rRKF2+DEiw==';
my $tsig = new Net::DNS::RR("$key_name TSIG $key");
$tsig->fudge(60);
my $update = new Net::DNS::Update('example.com');
$update->push( update => rr_add('foo.example.com A 10.1.2.3') );
$update->push( additional => $tsig );
[Of course, I wouldn't hard-code the key in my Perl script, but read it from the key file instead.]
Switching to another key file format would mean changing the DNS server configuration, which is not an elegant solution. "Manually" reading the key files and then "manually" constructing the keys is not very satisfying either, hence the question:
Did I understand correctly that it is not possible to use key files in the ddns-confgen format directly with Net::DNS and its sub-modules to TSIG-sign DNS update requests?
and thanks for looking!
I have an instance of YouTrack with several custom fields, some of which are String-type. I'm implementing a module to create a new issue via the YouTrack REST API's PUT request, and then updating its fields with user-submitted values by applying commands. This works great---most of the time.
I know that I can apply multiple commands to an issue at the same time by concatenating them into the query string, like so:
Type Bug Priority Critical add Fix versions 5.1 tag regression
will result in
Type: Bug
Priority: Critical
Fix versions: 5.1
in their respective fields (as well as adding the regression tag). But, if I try to do the same thing with multiple String-type custom fields, then:
Foo something Example Something else Bar P0001
results in
Foo: something Example Something else Bar P0001
Example:
Bar:
The command only applies to the first field, and the rest of the query string is treated like its String value. I can apply the command individually for each field, but is there an easier way to combine these requests?
Thanks again!
This is an expected result because all string after foo is considered a value of this field, and spaces are also valid symbols for string custom fields.
If you try to apply this command via command window in the UI, you will actually see the same result.
Such a good question.
I encountered the same issue and have spent an unhealthy amount of time in frustration.
Using the command window from the YouTrack UI I noticed it leaves trailing quotations and I was unable to find anything in the documentation which discussed finalizing or identifying the end of a string value. I was also unable to find any mention of setting string field values in the command reference, grammer documentation or examples.
For my solution I am using Python with the requests and urllib modules. - Though I expect you could turn the solution to any language.
The rest API will accept explicit strings in the POST
import requests
import urllib
from collections import OrderedDict
URL = 'http://youtrack.your.address:8000/rest/issue/{issue}/execute?'.format(issue='TEST-1234')
params = OrderedDict({
'State': 'New',
'Priority': 'Critical',
'String Field': '"Message to submit"',
'Other Details': '"Fold the toilet paper to a point when you are finished."'
})
str_cmd = ' '.join(' '.join([k, v]) for k, v in params.items())
command_url = URL + urllib.urlencode({'command':str_cmd})
result = requests.post(command_url)
# The command result:
# http://youtrack.your.address:8000/rest/issue/TEST-1234/execute?command=Priority+Critical+State+New+String+Field+%22Message+to+submit%22+Other+Details+%22Fold+the+toilet+paper+to+a+point+when+you+are+finished.%22
I'm sad to see this one go unanswered for so long. - Hope this helps!
edit:
After continuing my work, I have concluded that sending all the field
updates as a single POST is marginally better for the YouTrack
server, but requires more effort than it's worth to:
1) know all fields in the Issues which are string values
2) pre-process all the string values into string literals
3) If you were to send all your field updates as a single request and just one of them was missing, failed to set, or was an unexpected value, then the entire request will fail and you potentially lose all the other information.
I wish the YouTrack documentation had some mention or discussion of
these considerations.
I'm making an app similar to instagram using Rails/Mongoid. I want a unique ID that I can use in a url like http://instagr.am/p/DJmU8/
What's the easiest way to do that? Can I derive such an ID from the default BSON ObjectID Mongo creates?
You may try to use first 4 bytes of ObjectID (they will represent timestamp).
But, to be 100% safe, it's better to produce really unique short id, by implementing a counter. You can use separate collection to maintain current value of your counter.
More details on mongo's ObjectID structure can be found here: http://www.mongodb.org/display/DOCS/Object+IDs
As an alternative you can convert convert hex string id representation to a representation based on 36 symbols (26 latin letters + 10 digits). It will obviously be shorter.
It seems, that there is a ruby library, that can do such conversions http://rubyworks.github.com/radix/
Why not use dylang/shortid?
Install using npm npmjs.com/package/shortid:
npm i shortid
Then require:
const shortid = require('shortid');
In mongoose schema:
new Schema {
_id: {
type: String,
default: shortid.generate
}
}
or just insert directly:
users.insert({
_id: shortid.generate()
name: ...
email: ...
});
You could try Mongoid::Token
https://github.com/thetron/mongoid_token
From the docs:
This library is a quick and simple way to generate unique, random
tokens for your mongoid documents, in the cases where you can't, or
don't want to use slugs, or the default MongoDB IDs.
Mongoid::Token can help turn this:
http://myawesomewebapp.com/video/4dcfbb3c6a4f1d4c4a000012/edit
Into something more like this:
http://myawesomewebapp.com/video/83xQ3r/edit
#aav was mention that you can use first 4 bytes, but this value are in seconds and you can get even 10.000 or more insert per seconds. Other thing objectID is Uniq and you need check "when" you get error from duplicate value "Write Concerns"?
new Date().getTime() - is in milliseconds => 1557702577900 why not use last 4 bytes ? Timestamp in base62 is rqiJgPq
This code look interesting:
https://github.com/treygriffith/short-mongo-id/blob/master/lib/objectIdToShortId.js
Check also ObjectID timestamp parser:
https://steveridout.github.io/mongo-object-time/
Or you can execute ObjectId().toString() and base of this string create new by hashids [nodejs,php, andmanymore]
Maybe best options it to use 4-5 bytes from js timestamp and INC from
bson then hash this value by hids
var id = ObjectID('61e33b8467a45920f80eba52').toString();
console.log("id:", id);
console.log("timestamp:", parseInt(id.substring(0, 8),16).toString());
console.log("machineID:", parseInt(id.substring(8, 14),16) );
console.log("processID:", parseInt(id.substring(14, 18),16) );
console.log("counter:", parseInt(id.slice(-6),16) );
var ObjTimestamp = parseInt(ObjectID2.substring(0, 8),16).toString().slice(-5);
var Counter = parseInt(ObjectID2.slice(-6),16);
//https://github.com/base62/base62.js/
console.log('Final:',base62.encode(parseInt(Counter) + parseInt(ObjTimestamp) ));
Output:
vI7o
15V9L
5t4l
You can get collision if: more process are running, then consider add PID to unique and when you run multiple instance on different computer
Try gem https://github.com/jffjs/mongoid_auto_inc
The Hashids library is meant for generating IDs like this. Check it out here ☞ https://github.com/peterhellberg/hashids.rb
This might sound like a trivial question, but it is rather important for consumer facing apps
What is the easiest way and most scalable way to map the scary mongo id onto a id that is friendly?
xx.com/posts/4d371056183b5e09b20001f9
TO
xx.com/posts/a
M
You can create a composite key in mongoid to replace the default id using the key macro:
class Person
include Mongoid::Document
field :first_name
field :last_name
key :first_name, :last_name
end
person = Person.new(:first_name => "Syd", :last_name => "Vicious")
person.id # returns "syd-vicious"
If you don't like this way to do it, check this gem: https://github.com/hakanensari/mongoid-slug
Define a friendly unique field (like a slug) on your collection, index it, on your model, define to_param to return it:
def to_param
slug
end
Then in your finders, find by slug rather than ID:
#post = Post.where(:slug => params[:id].to_s).first
This will let you treat slugs as your effective PK for the purposes of resource interaction, and they're a lot prettier.
Unfortunately, the key macro has been removed from mongo. For custom ids,
users must now override the _id field.
class Band
include Mongoid::Document
field :_id, type: String, default: ->{ name }
end
Here's a great gem that I've been using to successfully answer this problem: Mongoid-Slug
https://github.com/digitalplaywright/mongoid-slug.
It provides a nice interface for adding this feature across multiple models. If you'd rather roll your own, at least check out their implementation for some ideas. If you're going this route, look into the Stringex gem, https://github.com/rsl/stringex, and acts_as_url library within. That will help you get the nice dash-between-url slugs.