I am able to successfully build vertices in IBM Graph, but when I execute query, output doesn't show the edges between vertices. I am running following code:
def v1 = graph.addVertex("name", "Darshit", label, "Inventorymanager","zip","95112");
def v2 = graph.addVertex("name", "Vikas", label, "Vendor","zip","95112");
def v3 = graph.addVertex("name", "Sidharth", label, "Vendor","zip","95112");
def v4 = graph.addVertex("name", "iPhone7", label, "Item","price",750,"quantity",10);
def v5 = graph.addVertex("name", "chair", label, "Item","price",20,"quantity",15);
def v6 = graph.addVertex("name", "laptop", label, "Item","price",800,"quantity",3);
v1.addEdge("ordered_to", v2);
v1.addEdge("ordered_to", v3);
v2.addEdge("has", v4);
v3.addEdge("has", v5);
v2.addEdge("has", v6);
v3.addEdge("has", v6);
def g = graph.traversal();
g.V().has("name", "Darshit").out("ordered_to").has("name", "Vikas").path();
Change your query to
g.V().has("name", "Darshit").outE("ordered_to").inV().has("name", "Vikas").path();
That'll get you the edges as well
Related
I have created a Graph using node and edges data in Spark application. Now, I want to come up with the adjacency list for the created graph. How can I achieve this ?
I have written following code to read csv files for node and edge data and creating the Graph.
val grapha = sc.textFile("graph.csv")
val getgdata = grapha.map(line=>line.split(","))
val node1 = getgdata.map(line=>(line(3).toLong,(line(0)))).distinct
val node2 = getgdata.map(line=>(line(4).toLong,(line(1)))).distinct
// This is node list of a graph.
val nodes = node1.union(node2).distinct
//This is edge list.
val routes = getgdata.map(line=>
(Edge(line(3).toLong,line(4).toLong,line(2)))).distinct
// now create graph using Graph library
val graphx = Graph(nodes,routes)
Now I need to see adjacency list for each node from this graph. How can I do it using scala?
Looking at your code, I am assuming that your graph.csv looks like following,
node_1, node_2, node_1_relation_node_2, 1, 2
node_1, node_3, node_1_relation_node_3, 1, 3
node_2, node_3, node_2_relation_node_3, 2, 3
Now, you can read this into an RDD as follows,
val graphData = sc.textFile("graph.csv").map(line => line.split(","))
Now, you to create your graph you need two things,
RDD of Vertices,
val verticesRdd = graphData.flatMap(line => List(
(line(3), line(0)),
(line(4), line(1))
)).distinct
RDD of Edges,
val edgesRdd = graphData.map(line => Edge(line(3), line(4), line(2))).distinct
Now, you can create your graph as,
val graph = Graph(verticesRdd, edgesRdd)
But if you just need adjacency list, you can obtain it just from the graphData as following,
val adjacencyRdd = graphData
.flatMap(line => {
val v1 = line(3).toLong
val v2 = line(4).toLong
List(
(v1, v2),
(v2, v1)
)
)
.aggregateByKey(Set.empty[Long])(
{ case (adjacencySet, vertexId) => adjacencySet + vertexId }
{ case (adjacencySet1, adjacencySet2) => adjacencySet1 ++ adjacencySet2 }
)
.map({ case (vertexId, adjacencySet) => (vertextId, adjacencySet.toList) })
Can you add a color gradient to GeoTiffs?
I am trying to do what is explained in Kernel Density
We already know that with a line like this:
kde.renderPng(kdeColorMap).write(“kde-buoy-waveHeight.png”)
we can write out a PNG with color…
But I can’t seem to figure out how to add that color to a GeoTiff…
I've tried this:
val iColorMap: IndexedColorMap = new IndexedColorMap(delauColorMap.colors)
val geoTiffOptions = GeoTiffOptions(colorMap = iColorMap)
val delauWebMer: Raster[Tile] = delau.reproject(extent, LatLng, WebMercator)
val extentWebMer: Extent = extent.reproject(LatLng, WebMercator)
val headTags: Map[String, String] = Map()
val bandTags: List[Map[String, String]] = List()
val tags: Tags = Tags(headTags, bandTags)
val tiff = SinglebandGeoTiff(delauWebMer, extentWebMer, WebMercator, tags, geoTiffOptions)
tiff.write("BuoyDelau3857.tif")
but get the following exception:
IncompatibleGeoTiffOptionsException: 'Palette' color space only supported for 8 or 16 bit integral cell types.
This works:
val tiff = GeoTiff(delauWebMer, extentWebMer, WebMercator)
tiff.write("BuoyDelau3857.tif")
but doesn't give us a color map, the output is in grey-scale.
Convert the Tile's CellType
val converted = delau.interpretAs(CellType.fromName("int16"))
I want to select k-means model in terms of 'k' parameter based on the lowest k-means score.
I can find find optimal value of 'k' parameter by hand, writing something like
def clusteringScore0(data: DataFrame, k: Int): Double = {
val assembler = new VectorAssembler().
setInputCols(data.columns.filter(_ != "label")).
setOutputCol("featureVector")
val kmeans = new KMeans().
setSeed(Random.nextLong()).
setK(k).
setPredictionCol("cluster").
setFeaturesCol("featureVector")
val pipeline = new Pipeline().setStages(Array(assembler, kmeans))
val kmeansModel = pipeline.fit(data).stages.last.asInstanceOf[KMeansModel]
kmeansModel.computeCost(assembler.transform(data)) / data.count() }
(20 to 100 by 20).map(k => (k, clusteringScore0(numericOnly, k))).
foreach(println)
Should I use CrossValitor API?
Something like this:
val paramGrid = new ParamGridBuilder().addGrid(kmeansModel.k, 20 to 100 by 20).build()
val cv = new CrossValidator().setEstimator(pipeline).setEvaluator(new KMeansEvaluator()).setEstimatorParamMaps(paramGrid).setNumFolds(3)
There are Evaluators for regression and classification, but no Evaluator for clustering.
So I should implement Evaluator interface. I am stuck with evaluate method.
class KMeansEvaluator extends Evaluator {
override def copy(extra: ParamMap): Evaluator = defaultCopy(extra)
override def evaluate(data: Dataset[_]): Double = ??? // should I somehow adapt code from KMeansModel.computeCost()?
override val uid = Identifiable.randomUID("cost_evaluator")
}
Hi ClusteringEvaluator is available from Spark 2.3.0. You can use to find optimal k values by including ClusteringEvaluator object into your for-loop. You can also find more detail for silhouette analysis in Scikit-learn page. In short, the score should be between [-1,1], the larger score is the better. I have modified a for loop below for your codes.
import org.apache.spark.ml.evaluation.ClusteringEvaluator
val evaluator = new ClusteringEvaluator()
.setFeaturesCol("featureVector")
.setPredictionCol("cluster")
.setMetricName("silhouette")
for(k <- 20 to 100 by 20){
clusteringScore0(numericOnly,k)
val transformedDF = kmeansModel.transform(numericOnly)
val score = evaluator.evaluate(transformedDF)
println(k,score,kmeansModel.computeCost(transformedDF))
}
I have a list of vertices and edges created this way:
val v1 = new Vertex(1L, "foo")
val v2 = new Vertex(2L, "bar")
val e1 = new Edge(v1, v2, 0.5)`
and want to create a Flink graph using the Graph.fromDataSet method (or any other for this matter). How can I transform those edges and vertices in something that is readable for Flink?
Thank you!!
Given a list of vertices val vertices: Seq[Vertex[Long, String]] = ... and edges val edges: Seq[Edge[Long, String]] = ... you can create a Graph using the Graph.fromCollection method:
val env = ExecutionEnvironment.getExecutionEnvironment
val vertices = Seq(new Vertex[Long, String](1L, "foo"), new Vertex[Long, String](2L, "bar"))
val edges = Seq(new Edge[Long, String](1L, 2L, "foobar"))
val graph = Graph.fromCollection(vertices, edges, env)
It is noteworthy that you have to import the Scala version of org.apache.flink.graph.scala.Graph.
Alternatively, you can also first create an edgeDataset: DataSet[Edge[Long, String]] and a vertexDataSet: DataSet[Vertex[Long, String]] using the ExecutionEnvironment. A Graph can then be created calling the Graph.fromDataSet method:
val vertexDataset = env.fromCollection(vertices)
val edgeDataset = env.fromCollection(edges)
val graph = Graph.fromDataSet(vertexDataset, edgeDataset, env)
Code:
package vu.co.kaiyin.sfreechart.plots
import java.awt.{Shape, Stroke, RenderingHints}
import javax.swing.JFrame
import org.jfree.chart.plot.{PlotOrientation, XYPlot}
import org.jfree.chart.{ChartFactory => cf}
import org.jfree.chart.renderer.GrayPaintScale
import org.jfree.chart.renderer.xy.XYBlockRenderer
import org.jfree.chart.title.PaintScaleLegend
import org.jfree.chart._
import org.jfree.chart.axis.{AxisLocation, NumberAxis}
import org.jfree.data.Range
import org.jfree.data.general.DatasetUtilities
import org.jfree.data.statistics.HistogramDataset
import org.jfree.data.xy.{IntervalXYDataset, XYZDataset}
import org.jfree.ui.{RectangleEdge, RectangleInsets}
import vu.co.kaiyin.sfreechart.{ColorPaintScale, ExtendedFastScatterPlot}
import vu.co.kaiyin.sfreechart.implicits._
import scala.util.Random.nextGaussian
/**
* Created by kaiyin on 2/10/16.
*/
object Plots {
def histogram(
dataset: IntervalXYDataset,
title: String = "Histogram",
xAxisLabel: String = "Intervals",
yAxisLabel: String = "Count",
orientation: PlotOrientation = PlotOrientation.VERTICAL,
legend: Boolean = true,
tooltips: Boolean = true,
urls: Boolean = true,
alpha: Float = 0.5F,
pannable: Boolean = false
): JFreeChart = {
val hist = cf.createHistogram(
title, xAxisLabel, yAxisLabel, dataset, orientation, legend, tooltips, urls
)
val xyPlot = hist.getPlot.asInstanceOf[XYPlot]
if (pannable) {
xyPlot.setDomainPannable(true)
xyPlot.setRangePannable(true)
}
xyPlot.setForegroundAlpha(alpha)
hist
}
def controuPlot(dataset: XYZDataset, title: String = "Contour plot", scaleTitle: String = "Scale"): JFreeChart = {
val xAxis = new NumberAxis("x")
val yAxis = new NumberAxis("y")
val blockRenderer = new XYBlockRenderer
val zBounds: Range = DatasetUtilities.findZBounds(dataset)
println(zBounds.getLowerBound, zBounds.getUpperBound)
val paintScale = new ColorPaintScale(zBounds.getLowerBound, zBounds.getUpperBound)
blockRenderer.setPaintScale(paintScale)
val xyPlot = new XYPlot(dataset, xAxis, yAxis, blockRenderer)
xyPlot.setAxisOffset(new RectangleInsets(1D, 1D, 1D, 1D))
xyPlot.setDomainPannable(true)
xyPlot.setRangePannable(true)
val chart = new JFreeChart(title, xyPlot)
chart.removeLegend()
val scaleAxis = new NumberAxis(scaleTitle)
val paintScaleLegend = new PaintScaleLegend(paintScale, scaleAxis)
paintScaleLegend.setAxisLocation(AxisLocation.BOTTOM_OR_LEFT)
paintScaleLegend.setPosition(RectangleEdge.BOTTOM)
chart.addSubtitle(paintScaleLegend)
chart
}
def fastScatter(data: Array[Array[Float]], title: String = "Scatter plot", pointSize: Int = 5, pointAlpha: Float = 0.3F): JFreeChart = {
val xAxis = new NumberAxis("x")
val yAxis = new NumberAxis("y")
xAxis.setAutoRangeIncludesZero(false)
yAxis.setAutoRangeIncludesZero(false)
val fsPlot = new ExtendedFastScatterPlot(data, xAxis, yAxis, pointSize, pointAlpha)
fsPlot.setDomainPannable(true)
fsPlot.setRangePannable(true)
val chart = new JFreeChart(title, fsPlot)
chart.getRenderingHints.put(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON)
chart
}
def main(args: Array[String]) {
// fastScatter(Array(Array(1.0F, 2.0F, 3.0F), Array(1.0F, 2.0F, 3.0F))).vis()
val x = (1 to 10000).map(_.toFloat).toArray
val y = x.map(i => i * nextGaussian().toFloat * 3F).toArray
fastScatter(Array(x, y)).vis()
val x1 = (-13.0 to 13.0 by 0.2).toArray
val y1 = (-13.0 to 13.0 by 0.2).toArray
val xyzData = (for {
i <- x1
j <- y1
if i > j
} yield Array(i, j, math.sin(i) + math.cos(j))).transpose
controuPlot(xyzData.toXYZDataset()).vis()
histogram((1 to 10000).map(_ => nextGaussian()).toArray.toHistogramDataset()).vis()
}
}
Full project can be found here: https://github.com/kindlychung/sfreechart
Running the above code will give you this:
If you look carefully, you will find a narrow band of pixels along the diagonal edge that doesn't quite fit (this is a contour plot of sin(x) + cos(y)), as if there was a tear and shift. But if I comment out the if i < j line, then the plot looks normal:
What went wrong and how can this be solved?
Update
Actually, if you look carefully at the right edge of the second figure above, there is also a strip that shouldn't be there.
I managed to fake a contour plot by a scatter plot:
val x = (-12.0 to 12.0 by 0.1).toArray
val y = (-12.0 to 12.0 by 0.1).toArray
val xyzData = (for {
i <- x
j <- y
} yield {
val s = math.sin(i)
val c = math.cos(j)
Array(i, j, s + c)
}).transpose
fastScatter(xyzData.toFloats, grid = (false, false), pointSize = 4, pointAlpha = 1F).vis()
Implementation of fastScatter can be found here: https://github.com/kindlychung/sfreechart (disclosure: I am the author.)