I've got a dataflow with a csv file as source. The column NewPositive is a string and it contains numbers formatted in European style with a dot as thousand seperator e.g 1.019 meaning 1019
If I use the function toInteger to convert my NewPositive column to an int via toInteger(NewPositive,'#.###','de'), I only get the thousand cipher e.g 1 for 1.019 and not the rest. Why? For testing I tried creating a constant column: toInteger('1.019','#.###','de') and it gives 1019 as expected. So why does the function not work for my column? The column is trimmed and if I compare the first value with equality function: equals('1.019',NewPositive) returns true.
Please note: I know it's very easy to create a workaround by toInteger(replace(NewPositive,'.','')), but I want to learn how to use the toInteger function with the locale and format parameters.
Here is sample data:
Dato;NewPositive
2021-08-20;1.234
2021-08-21;1.789
I was able to repro this and probably looks to be a bug to me . I have reported this to the ADF team , will let you know once I hear back from them . You already have a work around please go ahead that to unblock yourself .
I have an rrd file in which traffic_in and out stats of interfaces are stored.
What i want is that i want Max and min values in certain time period.
I'm trying this command but it is giving me error ERROR: invalid rpn expression in: v,MAX
rrdtool graph -s 1537466100 -e 1537552237 DEF:v=lhr-spndc-7609_traffic_in_612.rrd:traffic_in:MAX CDEF:vm=v,MAX PRINT:vm:%lf
Can you please help to enter correct command & achieving desired Functionality?
You should be using VDEF for the definition of vm, not CDEF.
A CDEF is for transforming one or more data series created by either a DEF or CDEF into another series, ready for graphing or summarising.
A VDEF is for transforming a single data series into a single value via a consolodation function, such as to get the maximum value of a series over the entire graph. This is different from the function specified in a DEF, which only specifies how to consolodate a higher-granularity series into a lower-granularity series.
I am using WEKA for classification. I am using two function, "setClassIndex" and "setAttributeIndices". My dataset have two attributes, class and one more attribute. Following are some instances in my database:
#relation sms_test
#attribute spamclass {spam,ham}
#attribute mes String
#data
ham,'Go until jurong point'
ham,'Ok lar...'
spam,'Free entry in 2 a wkly'
Following is part of my code.
trainData.setClassIndex(0);
filter = new StringToWordVector();
filter.setAttributeIndices("2");
This code is running fine. But when I set, train.setClassIndex ("1") or filter.setAttributeIndices("1") , my code stops running. Do setClassIndex function take argument starting from 0 and setAttributeIndices takes argument starting from 1? How do we identify which WEKA function starts counting from 0 or 1?
Do setClassIndex function take argument starting from 0
Yes, index starts with 0.
and setAttributeIndices takes argument starting from 1?
Yes, indices start from 1
Source: http://weka.sourceforge.net/doc.stable
I have written my own Odoo module where I added a property weight to the product.template model.
The implementation in the Python code is
weight = fields.Float('Weight', digits=(12,4))
I also changed the view so that I can set this value in the form. So I created a new product in the Odoo GUI and set the weight to 7.85. After storing the value, 7.8500 is shown which seems to be reasonable as the definition declares 4 float digits. The value stored in the PostgreSQL is a numeric and the value is 7.8500. So this all seems to be correct.
When I now want to get the product with the Odoo API which is based on XML-RPC I do not get 7.8500 but 7.8500000000000005
<member>
<name>weight</name>
<value><double>7.8500000000000005</double></value>
</member>
So my question is, why is this and how can I prevent this?
EDIT:
This behavior occurs whenever I have 2 decimal places. So when I take 7.8 instead of 7.85 the return value is 7.8.
I am using Weka in Scala (although the syntax is virtually identical to Java). I am trying to evaluate my data with a SimpleKMeans clusterer, but the clusterer won't accept string data. I don't want to cluster on the string data; I just want to use it to label the points.
Here is the data I am using:
#relation Locations
#attribute ID string
#attribute Latitude numeric
#attribute Longitude numeric
#data
'Carnegie Mellon University', 40.443064, -79.944163
'Stanford University', 37.427539, -122.170169
'Massachusetts Institute of Technology', 42.358866, -71.093823
'University of California Berkeley', 37.872166, -122.259444
'University of Washington', 47.65601, -122.30934
'University of Illinois Urbana Champaign', 40.091022, -88.229992
'University of Southern California', 34.019372, -118.28611
'University of California San Diego', 32.881494, -117.243079
As you can see, it's essentially a collection of points on an x and y coordinate plane. The value of any patterns is negligible; this is simply an exercise in working with Weka.
Here is the code that is giving me trouble:
val instance = new Instances(new StringReader(wekaHeader + wekaData))
val simpleKMeans = new SimpleKMeans()
simpleKMeans.buildClusterer(instance)
val eval = new ClusterEvaluation()
eval.setClusterer(simpleKMeans)
eval.evaluateClusterer(new Instances(instance))
Logger.info(eval.clusterResultsToString)
I get the following error on simpleKMeans.buildClusterer(instance):
[UnsupportedAttributeTypeException: weka.clusterers.SimpleKMeans: Cannot handle string attributes!]
How do I get Weka to retain IDs while doing clustering?
Here are a couple of other steps I have taken to troubleshoot this:
I used the Weka Explorer and loaded this data as a CSV:
ID, Latitude, Longitude
'Carnegie Mellon University', 40.443064, -79.944163
'Stanford University', 37.427539, -122.170169
'Massachusetts Institute of Technology', 42.358866, -71.093823
'University of California Berkeley', 37.872166, -122.259444
'University of Washington', 47.65601, -122.30934
'University of Illinois Urbana Champaign', 40.091022, -88.229992
'University of Southern California', 34.019372, -118.28611
'University of California San Diego', 32.881494, -117.243079
This does what I want it to do in the Weka Explorer. Weka clusters the points and retains the ID column to identify each point. I would do this in my code, but I'm trying to do this without generating additional files. As you can see from the Weka Java API, Instances interprets a java.io.Reader only as an ARFF.
I have also tried the following code:
val instance = new Instances(new StringReader(wekaHeader + wekaData))
instance.deleteAttributeAt(0)
val simpleKMeans = new SimpleKMeans()
simpleKMeans.buildClusterer(instance)
val eval = new ClusterEvaluation()
eval.setClusterer(simpleKMeans)
eval.evaluateClusterer(new Instances(instance))
Logger.info(eval.clusterResultsToString)
This works in my code, and displays results. That proves that Weka is working in general, but since I am deleting the ID attribute, I can't really map the clustered points back on the original values.
I am answering my own question, and in doing so, there are two issues that I would like to address:
Why CSV works with string values
How to get cluster information from the cluster evaluation
As Sentry points out in the comments, the ID does in fact get converted to a nominal attribute when loaded from a CSV.
If the data must be in an ARFF format (like in my example where the Instances object is created from a StringReader), then the StringToNominal filter can be applied:
val instances = new Instances(new StringReader(wekaHeader + wekaData))
val filter = new StringToNominal()
filter.setAttributeRange("first")
filter.setInputFormat(instances)
val filteredInstance = Filter.useFilter(instances, filter)
val simpleKMeans = new SimpleKMeans()
simpleKMeans.buildClusterer(instance)
...
This allows for "string" values to be used in clustering, although it's really just treated as a nominal value. It doesn't impact the clustering (if the ID is unique), but it doesn't contribute to the evaluation as I had hoped, which brings me to the next issue.
I was hoping to be able to get a nice map of cluster and data, like cluster: Int -> Array[(ID, latitude, longitude)] or ID -> cluster: Int. However, the cluster results are not that convenient. In my experience these past few days, there are two approaches that can be used to find the cluster of each point of data.
To get the cluster assignments, simpleKMeans.getAssignments returns an array of integers that is the cluster assignments for each data element. The array of integers is in the same order as the original data items and has to be manually related back to the original data items. This can be easily accomplished in Scala by using the zip method on the original list of data items and then using other methods like groupBy or map to get the collection in your favorite format. Keep in mind that this method alone does not use the ID attribute at all, and the ID attribute could be omitted from the data points entirely.
However, you can also get the cluster centers with simpleKMeans.getClusterCentroids or eval.clusterResultsToString(). I have not used this very much, but it does seem to me that the ID attribute can be recovered from the cluster centers here. As far as I can tell, this is the only situation in which the ID data can be utilized or recovered from the cluster evaluation.
I got the same error while having String value in one of the line in a CSV file with couple of million rows. Here is how I figured out which line has string value.
Exception "Cannot handle string attributes!" doesn't give any clue about the line number. Hence:
I imported CSV file into Weka Explorer GUI and created a *.arff file.
Then manually changed type from string to numeric in the *.arrf file at the beginning as show below.
After that I tried to build the cluster using the *.arff file.
I got the exact line number as part of exception
I removed the line from *.arff file and loaded again. It worked without any issue.
Converted string --> numeric in *.arff file
#attribute total numeric
#attribute avgDailyMB numeric
#attribute mccMncCount numeric
#attribute operatorCount numeric
#attribute authSuccessRate numeric
#attribute totalMonthlyRequets numeric
#attribute tokenCount numeric
#attribute osVersionCount numeric
#attribute totalAuthUserIds numeric
#attribute makeCount numeric
#attribute modelCount numeric
#attribute maxDailyRequests numeric
#attribute avgDailyRequests numeric
Error reported the exact line number
java.io.IOException: number expected, read Token[value.total], line 1750464
at weka.core.converters.ArffLoader$ArffReader.errorMessage(ArffLoader.java:354)
at weka.core.converters.ArffLoader$ArffReader.getInstanceFull(ArffLoader.java:728)
at weka.core.converters.ArffLoader$ArffReader.getInstance(ArffLoader.java:545)
at weka.core.converters.ArffLoader$ArffReader.readInstance(ArffLoader.java:514)
at weka.core.converters.ArffLoader$ArffReader.readInstance(ArffLoader.java:500)
at weka.core.Instances.<init>(Instances.java:138)
at com.lokendra.dissertation.ModelingUtils.kMeans(ModelingUtils.java:50)
at com.lokendra.dissertation.ModelingUtils.main(ModelingUtils.java:28)