I am using Spark to make a JanusGraph from a data stream, but am having issues indexing and creating properties. I want to create an index by a vertex property called "register_id". I am not sure I'm doing it the right way.
So, here's my code:
var gr1 = JanusGraphFactory.open("/Downloads/janusgraph-cassandra.properties")
gr1.close()
// This is done to clear the graph made in every run.
JanusGraphFactory.drop(gr1)
gr1 = JanusGraphFactory.open("/Downloads/janusgraph-cassandra.properties")
var reg_id_prop = gr1.makePropertyKey("register_id").dataType(classOf[String]).make()
var mgmt = gr1.openManagement()
gr1.tx().rollback()
mgmt.buildIndex("byRegId", classOf[Vertex]).addKey(reg_id_prop).buildCompositeIndex()
When I run the above, I get an error saying:
"Vertex with id 5164 was removed".
Also, how do I check if I have vertices with a certain property in the graph or not in Scala. I know in gremlin, g.V().has('name', 'property_value') works, but can't figure out how to do this in Scala. I tried Gremlin-Scala but can't seem to find it.
Any help will be appreciated.
You should be using mgmt object to build the schema, not the graph object. You also need to make sure to mgmt.commit() the schema updates.
gr1 = JanusGraphFactory.open("/Downloads/janusgraph-cassandra.properties")
var mgmt = gr1.openManagement()
var reg_id_prop = mgmt.makePropertyKey("register_id").dataType(classOf[String]).make()
mgmt.buildIndex("byRegId", classOf[Vertex]).addKey(reg_id_prop).buildCompositeIndex()
mgmt.commit()
Refer to the indexing docs from JanusGraph.
For your second question on checking for the existence of a vertex using the composite index, you need to finish your traversal with a terminal step. For example, in Java, this would return a boolean value:
g.V().has('name', 'property_value').hasNext()
Refer to iterating the traversal docs from JanusGraph.
Reading over the gremlin-scala README, it looks like it has a few options for terminal steps that you could use like head, headOption, toList, or toSet.
g.V().has('name', 'property_value').headOption
You should also check out the gremlin-scala-examples and the gremlin-scala traversal specification.
Related
Please could somebody confirm the following..
I am using Mirth Connect 3.5.08232.
My Source Connector is a Database Reader.
Say, I am using a query that returns multiple rows, and return the result (via JavaScript), as documentation suggests, so that Mirth would treat each row as a separate message. I also use a couple of mappers as source transformers, and save the mapped fields in my channel map (which ends up to contain only those fields that I define in transformers)
In the destination, and specifically, in destination response transformer (or destination body, if it is a JavaScript writer), how do I access the source fields?
the only way I found by trial and error is
var rawMsg = connectorMessage.getRawData();
var xmlMsg = new XML(rawMsg);
logger.info(xmlMsg.some_field); // ignore the root element of rawMsg
Is this the right way to do this? I thought that maybe the fields that were nicely automatically detected would be put in some kind of a map, like sourceMap - but that doesn't seem to be the case, right?
Thank you
If you are using Mapper steps in your transformer to extract the data and put it into a variable map (like the channel map), then you can use any of the following methods to retrieve it from a subsequent JavaScript context (including a JavaScript Writer, and your response transformer):
var value = channelMap.get('key');
var value = $c('key');
var value = $('key');
Look at the Variable Maps section of the User Guide for more information.
So to recap, say you're selecting a column "mycolumn" with a Database Reader. The XML sent to the channel will be something like this:
<result>
<mycolumn>value</mycolumn>
</result>
Then you can choose to extract pieces of that message into specific variables for later use. The transformer allows you to easily drag-and-drop pieces of the sample inbound message.
Finally in your JavaScript Writer (or in any subsequent filter, transformer, or response transformer), just drag the value into the field you want:
And the corresponding JavaScript code will automatically be inserted:
One last note, if you are selecting a lot of variables and don't want to make Mapper steps for each one individually, you can use a JavaScript Step to iterate through the message and extract each column into a separate map variable:
for each (child in msg.children()) {
channelMap.put(child.localName(), child.toString());
}
Or, you can just reference the columns directly from within the JavaScript Writer:
var msg = new XML(connectorMessage.getEncodedData());
var column1 = msg.column1.toString();
var column2 = msg.column2.toString();
...
Background: I'm trying to implement a time-series versioned DB using this approach, using gremlin (tinkerpop v3).
I want to get the latest state node (in red) for a given identity node (in blue) (linked by a 'state' edge which contains a timestamp range), but I want to return a single aggregated object which contains the id (cid) from the identity node and all the properties from the state node, but I don't want to have to list them explicitly.
(8640000000000000 is my way of indicating no 'to' date - i.e. the edge is current - slightly different from the image shown).
I've got this far:
:> g.V().hasLabel('product').
as('cid').
outE('state').
has('to', 8640000000000000).
inV().
as('name').
as('price').
select('cid', 'name','price').
by('cid').
by('name').
by('price')
=>{cid=1, name="Cheese", price=2.50}
=>{cid=2, name="Ham", price=5.00}
but as you can see I have to list out the properties of the 'state' node - in the example above the name and price properties of a product. But this will apply to any domain object so I don't want to have to list the properties all the time. I could run a query before this to get the properties but I don't think I should need to run 2 queries, and have the overhead of 2 round trips. I've looked at 'aggregate', 'union', 'fold' etc but nothing seems to do this.
Any ideas?
===================
Edit:
Based on Daniel's answer (which doesn't quite do what I want ATM) I'm going to use his example graph. In the 'modernGraph' people-create->software. If I run:
> g.V().hasLabel('person').valueMap()
==>[name:[marko], age:[29]]
==>[name:[vadas], age:[27]]
==>[name:[josh], age:[32]]
==>[name:[peter], age:[35]]
then the results are a list of entities's with the properties. What I want is, on the assumption that a person can only create one piece of software ever (although hopefully we will see how this could be opened up later for lists of software created), to include the created software 'language' property into the returned entity to get:
> <run some query here>
==>[name:[marko], age:[29], lang:[java]]
==>[name:[vadas], age:[27], lang:[java]]
==>[name:[josh], age:[32], lang:[java]]
==>[name:[peter], age:[35], lang:[java]]
At the moment the best suggestion so far comes up with the following:
> g.V().hasLabel('person').union(identity(), out("created")).valueMap().unfold().group().by {it.getKey()}.by {it.getValue()}
==>[name:[marko, lop, lop, lop, vadas, josh, ripple, peter], lang:[java, java, java, java], age:[29, 27, 32, 35]]
I hope that's clearer. If not please let me know.
Since you didn't provide I sample graph, I'll use TinkerPop's toy graph to show how it's done.
Assume you want to merge marko and lop:
gremlin> g = TinkerFactory.createModern().traversal()
==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]
gremlin> g.V(1).valueMap()
==>[name:[marko],age:[29]]
gremlin> g.V(1).out("created").valueMap()
==>[name:[lop],lang:[java]]
Note, that there are two name properties and in theory you won't be able to predict which name makes it into your merged result; however that doesn't seem to be an issue in your graph.
Get the properties for both vertices:
gremlin> g.V(1).union(identity(), out("created")).valueMap()
==>[name:[marko],age:[29]]
==>[name:[lop],lang:[java]]
Merge them:
gremlin> g.V(1).union(identity(), out("created")).valueMap().
unfold().group().by(select(keys)).by(select(values))
==>[name:[lop],lang:[java],age:[29]]
UPDATE
Thank you for the added sample output. That makes it a lot easier to come up with a solution (although I think your output contains errors; vadas didn't create anything).
gremlin> g.V().hasLabel("person").
filter(outE("created")).map(
union(valueMap(),
outE("created").limit(1).inV().valueMap("lang")).
unfold().group().by {it.getKey()}.by {it.getValue()})
==>[name:[marko], lang:[java], age:[29]]
==>[name:[josh], lang:[java], age:[32]]
==>[name:[peter], lang:[java], age:[35]]
Merging edge and vertex properties using gremlin java DSL:
g.V().has('User', 'id', userDbId).outE(Edges.TWEETS)
.union(__.identity().valueMap(), __.inV().valueMap())
.unfold().group().by(__.select(Column.keys)).by(__.select(Column.values))
.map(v -> converter.toTweet((Map) v.get())).toList();
Thanks for the answer by Daniel Kuppitz and youhans it has given me a basic idea on the solution of the issue. But later I found out that the solution is not working for multiple rows. It is required to have local step for handling multiple rows. The modified gremlin query will look like:
g.V()
.local(
__.union(__.valueMap(), __.outE().inV().valueMap())
.unfold().group().by(__.select(Column.keys)).by(__.select(Column.values))
)
This will limit the scope of union and group by to a single row.
If you can work with custom DSL ,create custom DSL with java like this one.
public default GraphTraversal<S, LinkedHashMap> unpackMaps(){
GraphTraversal<S, LinkedHashMap> it = map(x -> {
LinkedHashMap mapSource = (LinkedHashMap) x.get();
LinkedHashMap mapDest = new LinkedHashMap();
mapSource.keySet().stream().forEach(key->{
Object obj = mapSource.get(key);
if (obj instanceof LinkedHashMap) {
LinkedHashMap childMap = (LinkedHashMap) obj;
childMap.keySet().iterator().forEachRemaining( key_child ->
mapDest.put(key_child,childMap.get(key_child)
));
} else
mapDest.put(key,obj);
});
return mapDest;
});
return it;
}
and use it freely like
g.V().as("s")
.valueMap().as("value_map_0")
.select("s").outE("INFO1").inV().valueMap().as("value_map_1")
.select("s").outE("INFO2").inV().valueMap().as("value_map_2")
.select("s").outE("INFO3").inV().valueMap().as("value_map_3")
.select("s").local(__.outE("INFO1").count()).as("value_1")
.select("s").outE("INFO1").inV().value("name").as("value_2")
.project("val_map1","val_map2","val_map3","val1","val2")
.by(__.select("value_map_1"))
.by(__.select("value_map_2"))
.by(__.select("value_1"))
.by(__.select("value_2"))
.unpackMaps()
results to rows with
map1_val1, map1_val2,.... ,map2_va1, map2_val2....,value1, value2
This can handle mix of values and valueMaps in a natural gremlin way.
The goal
I have a simple enough task to accomplish: Set the weight of a specific edge property. Take this scenario as an example:
What I would like to do is update the value of weight.
Additional Requirements
If the edge does not exist, it should be created.
There may only exist at most one edge of the same type between the two nodes (i.e., there can't be multiple "votes_for" edges of type "eat" between Joey and Pizza.
The task should be solved using the Java API of Titan (which includes Gremlin as part of TinkerPop 3).
What I know
I have the following information:
The Vertex labeled "user"
The edge label votes_for
The value of the edge property type (in this case, "eat")
The value of the property name of the vertex labeled "meal" (in this case "pizza"), and hence also its Vertex.
What I thought of
I figured I would need to do something like the following:
Start at the Joey vertex
Find all outgoing edges (which should be at most 1) labeled votes_for having type "eat" and an outgoing vertex labeled "meal" having name "pizza".
Update the weight value of the edge.
This is what I've messed around with in code:
//vertex is Joey in this case
g.V(vertex.id())
.outE("votes_for")
.has("type", "eat")
//... how do I filter by .outV so that I can check for "pizza"?
.property(Cardinality.single, "weight", 0.99);
//... what do I do when the edge doesn't exist?
As commented in code there are still issues. Would explicitly specifying a Titan schema help? Are there any helper/utility methods I don't know of? Would it make more sense to have several vote_for labels instead of one label + type property, like vote_for_eat?
Thanks for any help!
You are on the right track. Check out the vertex steps documentation.
Label the edge, then traverse from the edge to the vertex to check, then jump back to the edge to update the property.
g.V(vertex.id()).
outE("votes_for").has("type", "eat").as("e").
inV().has("name", "pizza").
select("e").property("weight", 0.99d).
iterate()
Full Gremlin console session:
gremlin> Titan.version()
==>1.0.0
gremlin> Gremlin.version()
==>3.0.1-incubating
gremlin> graph = TitanFactory.open('inmemory'); g = graph.traversal()
==>graphtraversalsource[standardtitangraph[inmemory:[127.0.0.1]], standard]
gremlin> vertex = graph.addVertex(T.label, 'user', 'given_name', 'Joey', 'family_name', 'Tribbiani')
==>v[4200]
gremlin> pizza = graph.addVertex(T.label, 'meal', 'name', 'pizza')
==>v[4104]
gremlin> votes = vertex.addEdge('votes_for', pizza, 'type', 'eat', 'weight', 0.8d)
==>e[1zh-38o-4r9-360][4200-votes_for->4104]
gremlin> g.E(votes).valueMap(true)
==>[label:votes_for, weight:0.8, id:2rx-38o-4r9-360, type:eat]
gremlin> g.V(vertex.id()).outE('votes_for').has('type','eat').as('e').inV().has('name','pizza').select('e').property('weight', 0.99d).iterate(); g.E(votes).valueMap(true)
==>[label:votes_for, weight:0.99, id:2rx-38o-4r9-360, type:eat]
Would explicitly specifying a Titan schema help?
If you wanted to start from the Joey node without having a reference to the vertex or its id, this would be a good use case for a Titan composite index. The traversal would start with:
g.V().has("given_name", "Joey")
Are there any helper/utility methods I don't know of?
In addition to the TinkerPop reference documentation, there are several tutorials that you can read through:
Getting Started
The Gremlin Console
Recipes
Would it make more sense to have several vote_for labels instead of one label + type property, like vote_for_eat?
Depends on what your graph model or query patterns are, but more granular labels like vote_for_eat can work out fine. You can pass multiple edge labels on the traversal step:
g.V(vertex.id()).outE('vote_for_eat', 'vote_for_play', 'vote_for_sleep')
Update
There may only exist at most one edge of the same type between the two nodes
You can use the Titan schema to help with this, specifically define an edge label with multiplicity ONE2ONE. An exception will be thrown if you create more than one votes_for_eat between Joey and pizza.
Jason already answered nearly all of your questions. The only aspect missing is:
If the edge does not exist, it should be created.
So I'll try to answer this point with a slightly different query. This query adds a new edge if it doesn't exist already and then updates / adds the weight property:
g.V(vertex.id()).outE('votes_for').has('type', 'eat')
.where(__.inV().hasLabel('meal').has('name','pizza')) // filter for the edge to update
.tryNext() // select the edge if it exists
.orElseGet({g.V(vertex.id()).next()
.addEdge('votes_for', g.V(pizzaId).next(), 'type', 'eat')}) // otherwise, add the edge
.property('weight', 0.99) // finally, update / add the 'weight' property
I have a traversal as follows:
g.V().hasLabel("demoUser")
.as("demoUser","socialProfile","followCount","requestCount")
.select("demoUser","socialProfile","followCount","postCount")
.by(__.valueMap())
.by(__.out("socialProfileOf").valueMap())
.by(__.in("followRequest").hasId(currentUserId).count())
.by(__.outE("postAuthorOf").count())
I'm trying to select a user vertex, their linked social profile vertex, and some other counts. The issue is that all users may not have a socialProfile edge. When this is the case the traversal fails with the following error:
The provided start does not map to a value: v[8280]->[TitanVertexStep(OUT,[socialProfileOf],vertex), PropertyMapStep(value)]
I did find this thread from the gremlin team. I tried wrapping the logic inside of .by() with a coalesce(), and also appending a .fold() to the end of the statement with no luck.
How do I make that selection optional? I want to select a socialProfile if one exists, but always select the demoUser.
coalesce is the right choice. Let's assume that persons in the modern graph have either one or no project associated with them:
gremlin> g.V().hasLabel("person").as("user","project").
select("user","project").by("name").by(coalesce(out("created").values("name"),
constant("N/A")))
==>{user=marko, project=lop}
==>{user=vadas, project=N/A}
==>{user=josh, project=ripple}
==>{user=peter, project=lop}
Another way would be to completely exclude it from the result:
g.V().hasLabel("person").as("user","project").choose(out("created"),
select("user","project").by("name").by(out("created").values("name")),
select("user").by("name"))
But obviously this will only look good if each branch returns a map / selects more than 1 thing, otherwise you're going to have mixed result types.
I new to pandas and trying to learn how to work with it. Im having a problem when trying to use an example I saw in one of wes videos and notebooks on my data. I have a csv file that looks like this:
filePath,vp,score
E:\Audio\7168965711_5601_4.wav,Cust_9709495726,-2
E:\Audio\7168965711_5601_4.wav,Cust_9708568031,-80
E:\Audio\7168965711_5601_4.wav,Cust_9702445777,-2
E:\Audio\7168965711_5601_4.wav,Cust_7023544759,-35
E:\Audio\7168965711_5601_4.wav,Cust_9702229339,-77
E:\Audio\7168965711_5601_4.wav,Cust_9513243289,25
E:\Audio\7168965711_5601_4.wav,Cust_2102513187,18
E:\Audio\7168965711_5601_4.wav,Cust_6625625104,-56
E:\Audio\7168965711_5601_4.wav,Cust_6073165338,-40
E:\Audio\7168965711_5601_4.wav,Cust_5105831247,-30
E:\Audio\7168965711_5601_4.wav,Cust_9513082770,-55
E:\Audio\7168965711_5601_4.wav,Cust_5753907026,-79
E:\Audio\7168965711_5601_4.wav,Cust_7403410322,11
E:\Audio\7168965711_5601_4.wav,Cust_4062144116,-70
I loading it to a data frame and the group it by "filePath" and "vp", the code is:
res = df.groupby(['filePath','vp']).size()
res.index
and the output is:
[E:\Audio\7168965711_5601_4.wav Cust_2102513187,
Cust_4062144116, Cust_5105831247,
Cust_5753907026, Cust_6073165338,
Cust_6625625104, Cust_7023544759,
Cust_7403410322, Cust_9513082770,
Cust_9513243289, Cust_9702229339,
Cust_9702445777, Cust_9708568031,
Cust_9709495726]
Now Im trying to approach the index like a dict, as i saw in examples, but when im doing
res['Cust_4062144116']
I get an error:
KeyError: 'Cust_4062144116'
I do succeed to get a result when im putting the filepath, but as i understand and saw in previouse examples i should be able to use the vp keys as well, isnt is so?
Sorry if its a trivial one, i just cant understand why it is working in one example but not in the other.
Rutger you are not correct. It is possible to "partial" index a multiIndex series. I simply did it the wrong way.
The index first level is the file name (e.g. E:\Audio\7168965711_5601_4.wav above) and the second level is vp. Meaning, for each file name i have multiple vps.
Now, this is correct:
res['E:\Audio\7168965711_5601_4.wav]
and will return:
Cust_2102513187 2
Cust_4062144116 8
....
but trying to index by the inner index (the Cust_ indexes) will fail.
You groupby two columns and therefore get a MultiIndex in return. This means you also have to slice using those to columns, not with a single index value.
Your .size() on the groupby object converts it into a Series. If you force it in a DataFrame you can use the .xs method to slice a single level:
res = pd.DataFrame(df.groupby(['filePath','vp']).size())
res.xs('Cust_4062144116', level=1)
That works. If you want to keep it as a series, boolean indexing can help, something like:
res[res.index.get_level_values(1) == 'Cust_4062144116']
The last option is a bit less readable, but sometimes also more flexibile, you could test for multiple values at once for example:
res[res.index.get_level_values(1).isin(['Cust_4062144116', 'Cust_6073165338'])]