Weka : how to use cross validation in code - classification

I was trying to use cross validation in following code:
Program:
TextDirectoryToArff d = new TextDirectoryToArff();
try {
Instances dataset = d.createDataset("C:\\mytest");
dataset.setClassIndex(dataset.numAttributes() - 1 );
double precision = 0, recall=0,fmeasure=0,error=0;
int size1 = dataset1.numInstances() / 10;
int begin = 0;
int end = size1 - 1 ;
for (int i=1 ; i<=10;i++)
{
System.out.println("iteration :" + 1);
Instances training = new Instances(dataset);
Instances testing = new Instances(dataset, begin , (end - begin));
for (int j=0;j < (end - begin); j++)
training.delete(begin);
Classifier tree = new NaiveBayes();
Instances filteredInstaces = training;
StringToNominal nominal ;
for(int a=0;a<training.numAttributes()-1;a++)
{
if(training.attribute(a).isString())
{
nominal = new StringToNominal();
nominal.setInputFormat(filteredInstaces);
training = Filter.useFilter(training, nominal);
}
}
tree.buildClassifier(training);
Evaluation eval = new Evaluation(testing);
eval.evaluateModel(tree, testing);
System.out.println("Precision:" + eval.precision(1));
System.out.println("Recall:" + eval.recall(1));
System.out.println("Fmeasure:" + eval.fMeasure(1));
System.out.println("Error:" + eval.errorRate());
I've some code for cross validation but not able to integrate with above code. Please suggest how can I integrate following code in above code to find cross validation?
Code:
Evaluation eval = new Evaluation(dataset);
eval.evaluateModel(cls, dataset2);
eval.crossValidateModel(cls,dataset1,10, dataset2.getRandomNumberGenerator(1));
System.out.println(eval.toSummaryString("\nResults\n======\n", false));

I think you are confused a bit about weka Evaluation.crossValidateModel method. It already calculate the 10 different train and test fold, and train 10 models on trains and evaluate the models on test, so not necessary to calculate it as you tried in your code.
so in your code:
TextDirectoryToArff d = new TextDirectoryToArff();
try {
Instances dataset = d.createDataset("C:\\mytest");
dataset.setClassIndex(dataset.numAttributes() - 1 );
// you already have a dataset
Classifier naiveBayes = new NaiveBayes();
//you need a classifier
Evaluation eval = new Evaluation(dataset);
//only call the crossValidateModel with your classifier, on your dataset, with 10 fold, and random
eval.crossValidateModel(naiveBayes, dataset, 10, new Random(1));
//print the results of 10 fold
System.out.println(classifier);
System.out.println(eval.toSummaryString());
System.out.println(eval.toMatrixString());
System.out.println(eval.toClassDetailsString());
you can find more at https://weka.wikispaces.com/Generating+cross-validation+folds+(Java+approach)

Related

How to generate a model for my code using boolector?

I'm experimenting a bit with boolector so I'm trying to create model for simple code. Suppose that I have the following pseudo code:
int a = 5;
int b = 4;
int c = 3;
For this simple set of instructions I can create the model and all works fine. The problem is when I have other instructions after that like
b = 10;
c = 20;
Obviously it fails to generate the model because b cannot be equal to 4 and 10 within the same module. One of the maintainer suggested me to use boolector_push and boolector_pop in order to create new Contexts when needed.
The code for boolector_push is :
void
boolector_push (Btor *btor, uint32_t level)
{
BTOR_ABORT_ARG_NULL (btor);
BTOR_TRAPI ("%u", level);
BTOR_ABORT (!btor_opt_get (btor, BTOR_OPT_INCREMENTAL),
"incremental usage has not been enabled");
if (level == 0) return;
uint32_t i;
for (i = 0; i < level; i++)
{
BTOR_PUSH_STACK (btor->assertions_trail,
BTOR_COUNT_STACK (btor->assertions));
}
btor->num_push_pop++;
}
Instead for boolector_pop is
void
boolector_pop (Btor *btor, uint32_t level)
{
BTOR_ABORT_ARG_NULL (btor);
BTOR_TRAPI ("%u", level);
BTOR_ABORT (!btor_opt_get (btor, BTOR_OPT_INCREMENTAL),
"incremental usage has not been enabled");
BTOR_ABORT (level > BTOR_COUNT_STACK (btor->assertions_trail),
"can not pop more levels (%u) than created via push (%u).",
level,
BTOR_COUNT_STACK (btor->assertions_trail));
if (level == 0) return;
uint32_t i, pos;
BtorNode *cur;
for (i = 0, pos = 0; i < level; i++)
pos = BTOR_POP_STACK (btor->assertions_trail);
while (BTOR_COUNT_STACK (btor->assertions) > pos)
{
cur = BTOR_POP_STACK (btor->assertions);
btor_hashint_table_remove (btor->assertions_cache, btor_node_get_id (cur));
btor_node_release (btor, cur);
}
btor->num_push_pop++;
}
In my opinion, those 2 functions maintains track of the assertions generated using boolector_assert so how is it possible to obtain the final and correct model using boolector_push and boolector_pop considering that the constraints are going to be the same?
What am I missing?
Thanks
As you suspected, solver's push and pop methods aren't what you're looking for here. Instead, you have to turn the program you are modeling into what's known as SSA (Static Single Assignment) form. Here's the wikipedia article on it, which is quite informative: https://en.wikipedia.org/wiki/Static_single_assignment_form
The basic idea is that you "treat" your mutable variables as time-varying values, and give them unique names as you make multiple assignments to them. So, the following:
a = 5
b = a + 2
c = b + 3
c = c + 1
b = c + 6
becomes:
a0 = 5
b0 = a0 + 2
c0 = b0 + 3
c1 = c0 + 1
b1 = c1 + 6
etc. Note that conditionals are tricky to deal with, and generally require what's known as phi-nodes. (i.e., merging the values of branches.) Most compilers do this sort of conversion automatically for you, as it enables many optimizations down the road. You can either do it by hand, or use an algorithm to do it for you, depending on your particular problem.
Here's another question on stack-overflow, that's essentially asking for something similar: Z3 Conditional Statement
Hope this helps!

How can I measure Precision and Recall on Logistic Regression with PySpark?

I am using a Logistic Regression model on PySpark through databricks but i am not able to get my precision and recall. Everything works fine and I am able to get my ROC but there is not attribute or lib for Precision and Recall
lrModel = LogisticRegression()
predictions = bestModel.transform(testData)
# Instantiate metrics object
results = predictions.select(['probability', 'label'])
results_collect = results.collect()
results_list = [(float(i[0][0]), 1.0-float(i[1])) for i in results_collect]
scoreAndLabels = sc.parallelize(results_list)
metrics = MulticlassMetrics(scoreAndLabels)
# Overall statistics
precision = metrics.precision()
recall = metrics.recall()
f1Score = metrics.fMeasure()
print("Summary Stats")
print("Precision = %s" % precision)
print("Recall = %s" % recall)
print("F1 Score = %s" % f1Score)
>>>Summary Stats
>>>Precision = 0.0
>>>Recall = 0.0
>>>F1 Score = 0.0
I was able to create my own function to do so. It returns everything and more. I am using the "MulticlassMetrics()" from mllib package. Since its a multiclass it calculates metrics for each label so, you have to specify which label you want to retrieve.
### Model Evaluator User Defined Functions
def udfModelEvaluator(dfPredictions, labelColumn='label'):
colSelect = dfPredictions.select(
[F.col('prediction').cast(DoubleType())
,F.col(labelColumn).cast(DoubleType()).alias('label')])
metrics = MulticlassMetrics(colSelect.rdd)
mAccuracy = metrics.accuracy
mPrecision = metrics.precision(1)
mRecall = metrics.recall(1)
mF1 = metrics.fMeasure(1.0, 1.0)
mMatrix = metrics.confusionMatrix().toArray().astype(int)
mTP = metrics.confusionMatrix().toArray()[1][1]
mTN = metrics.confusionMatrix().toArray()[0][0]
mFP = metrics.confusionMatrix().toArray()[0][1]
mFN = metrics.confusionMatrix().toArray()[1][0]
mResults = [mAccuracy, mPrecision, mRecall, mF1, mMatrix, mTP, mTN, mFP, mFN, "Return [[0]=Accuracy, [1]=Precision, [2]=Recall, [3]=F1, [4]=ConfusionMatrix, [5]=TP, [6]=TN, [7]=FP, [8]=FN]"]
return mResults
To call the function:
metricsList = udfModelEvaluator(predictionsData, "label")
metricsList

Optimal dispatch of power generators

I want to write a matlab code that can optimally dispatch power generators to satisfy electric demand by considering profit too. I used below code that is in matlab website. https://ww2.mathworks.cn/help/optim/examples/optimal-dispatch-of-power-generators.html?requestedDomain=en
But in this code they are not considering the demand.
My target is , there are 20 power plants and I want to schedule this plants to satisfy demand. And the running power level can be any value in between the max and minimum power value of each power plant. High cost power plants should dispatch too.
name = 'generationcost.xlsx';
poolPrice = xlsread(name,'AD3:AD50'); % Revenue in dollars per MWh in interval
demand = xlsread(name,'AC3:AC52');
month = numel(demand);
e = xlsread(name,'W2:W4')'; %Cost for a unit of fuel
fuelPrice = repmat(e,1,1,2);
nPeriods = length(poolPrice);
nGens = 3;
genhigh = xlsread(name,'F2:F4');
genlow = xlsread(name,'Q2:Q4');
gen = [genlow(:),genhigh(:),]; %MW
fuel = [50,330;45,325;55,750]; % Fuel consumption for generator
startCost = [10000;10000;10000]; %Cost in dollars to start a generator after it has been off
y = optimvar('y',nPeriods,nGens,{'Low','High'},'Type','integer','LowerBound',0,'UpperBound',1);
z = optimvar('z',nPeriods,nGens,'Type','integer','LowerBound',0,'UpperBound',1);
powercons = y(:,:,'Low') + y(:,:,'High') <= 1;
yFuel = zeros(nPeriods,nGens,2);
yFuel(:,1,1) = fuel(1,1);
yFuel(:,1,2) = fuel(1,2);
yFuel(:,2,1) = fuel(2,1);
yFuel(:,2,2) = fuel(2,2);
yFuel(:,3,1) = fuel(3,1);
yFuel(:,3,2) = fuel(3,2);
fuelUsed = sum(reshape((sum(y.*yFuel)),[3 2])');
%fuelcons = fuelUsed <= totalFuel;
w = optimexpr(nPeriods,nGens);
idx = 1:(nPeriods-1);
w(idx,:) = y(idx+1,:,'Low') - y(idx,:,'Low') + y(idx+1,:,'High') -
y(idx,:,'High');
w(nPeriods,:) = y(1,:,'Low') - y(nPeriods,:,'Low') + y(1,:,'High') -
y(nPeriods,:,'High');
switchcons = w - z <= 0;
generatorlevel = zeros(size(yFuel));
generatorlevel(:,1,1) = gen(1,1);
generatorlevel(:,1,2) = gen(1,2);
generatorlevel(:,2,1) = gen(2,1);
generatorlevel(:,2,2) = gen(2,2);
generatorlevel(:,3,1) = gen(3,1);
generatorlevel(:,3,2) = gen(3,2);
revenue = optimexpr(size(y));
for ii = 1:nPeriods
revenue(ii,:,:) = poolPrice(ii)*y(ii,:,:).*generatorlevel(ii,:,:);
end
val = optimexpr(size(y));
demands = optimexpr(nPeriods,1);
id = 1:nPeriods;
val(id,:,:)= y(id,:,:).*generatorlevel(id,:,:);
demands(id,:)= sum(sum(val,3),2);
demandcons = demands-demand ;
fuelCost = sum(sum(fuelPrice.*sum(y.*yFuel)));
startingCost = z*startCost;
profit = sum(sum(sum(revenue))) - fuelCost - sum(startingCost);
dispatch = optimproblem('ObjectiveSense','maximize');
dispatch.Objective = profit;
dispatch.Constraints.powercons = powercons;
dispatch.Constraints.demandcons = demands - demand <= 0 ;
options = optimoptions('intlinprog','Display','final');
[dispatchsol,fval,exitflag,output] = solve(dispatch,'options',options);
subplot(5,1,1)
bar(dispatchsol.y(:,1,1)*gen(1,1)+dispatchsol.y(:,1,2)*gen(1,2),.5,'g')
xlim([.5,48.5])
ylabel('MW')
title('Generator 1 Optimal Schedule','FontWeight','bold')
subplot(5,1,2)
bar(dispatchsol.y(:,2,1)*gen(2,1)+dispatchsol.y(:,2,2)*gen(2,2),.5,'c')
title('Generator 2 Optimal Schedule','FontWeight','bold')
xlim([.5,48.5])
ylabel('MW')
subplot(5,1,3)
bar(dispatchsol.y(:,3,1)*gen(3,1)+dispatchsol.y(:,3,2)*gen(3,2),.5,'c')
title('Generator 3 Optimal Schedule','FontWeight','bold')
xlim([.5,48.5])
ylabel('MW')
subplot(5,1,4)
bar(demand,.5)
xlim([.5,48.5])
title('Daily Demand','FontWeight','bold')
xlabel('Period')
ylabel('MW')
starttimes = find(round(dispatchsol.z) == 1);
[theperiod,thegenerator] = ind2sub(size(dispatchsol.z),starttimes)
This code is not satisfying Demand. The output demand value is more lesser than the actual demand value. And also here only use 'High' or 'Low' power levels only. In between power values not using.

matlab oop "use data of hole class" for calculation

First, sorry for the bad title - I'm new to OO programming - basically I'd like to understand how Matlab works with oop classes.
Before I ask my question, here is a basic example of what I want to do:
I have two houses and some data about them and I got the idea, to work with oop classes. Here is my .m file.
classdef building
properties
hohe = 0;
lange = 0;
breite = 0;
xabstandsolar = 0;
yabstandsolar = 0;
end
methods
function obj = building(hohe, lange, breite, xabstandsolar, yabstandsolar)
obj.hohe = hohe;
obj.lange = lange;
obj.breite = breite;
obj.xabstandsolar = xabstandsolar;
obj.yabstandsolar = yabstandsolar;
end
function hohenwinkel(h)
h = h
d = sqrt(obj.xabstandsolar^2 + yabstandsolar^2);
gamma_v = atand((obj.hohe - h)/(d));
end
end
end
I filled it with some data - for example
>>H1 = building(10,8,6,14,8)
>>H2 = building(18,8,6,14,0)
And now I want to calculate "gamma_v" (as an 1x2 array) for each building. Any ideas, how I can archive this?
Edit:
I want to create gamma_v (as an array) automatically for all objects in the class "building". There will be a lot more "houses" in the final script.
Your hohenwinkel method needs to accept two input arguments. The first argument for non-static methods is always the object itself (unlike C++, you'll have to explicitly list it as an input argument) and the second input will be your h variable. You'll also want to actually return the gamma_v value using an output argument for your method.
function gamma_v = hohenwinkel(obj, h)
d = sqrt(obj.xabstandsolar^2 + obj.yabstandsolar^2);
gamma_v = atand((obj.hohe - h)/(d));
end
Then you can invoke this method on each building to get the value
gamma_v1 = hohenwinkel(H1);
gamma_v2 = hohenwinkel(H2);
If you want to have an array of buildings, you can create that array
houses = [building(10,8,6,14,8), building(18,8,6,14,0)];
gamma_v = hohenwinkel(houses);
and then construct your hohenwinkel function to operate on each one and return the result
function gamma_v = hohenwinkel(obj, h)
if numel(obj) > 1
% Compute hohenwinkel for each one
gamma_v = arrayfun(#(x)hohenwinkel(x, h), obj);
return
end
d = sqrt(obj.xabstandsolar^2 + obj.yabstandsolar^2);
gamma_v = atand((obj.hohe - h)/(d));
end
there is some tricky solution (and its not recommended)(#Suever solution is better)
you should create a handle class
classdef gvclass<handle
properties
gvarr=[];
end
methods
function setgvarr(obj,value)
obj.gvarr=[obj.gvarr,value];
end
end
end
then use this class in your building class
classdef building<handle
properties
gv
hohe = 0;
lange = 0;
breite = 0;
xabstandsolar = 0;
yabstandsolar = 0;
end
methods
function obj = building(hohe, lange, breite, xabstandsolar, yabstandsolar,handle_of_your_gv_class,h)
obj.hohe = hohe;
obj.lange = lange;
obj.breite = breite;
obj.xabstandsolar = xabstandsolar;
obj.yabstandsolar = yabstandsolar;
obj.gv=handle_of_your_gv_class;
obj.hohenwinkel(h);
end
function hohenwinkel(obj,h)
d = sqrt(obj.xabstandsolar^2 + yabstandsolar^2);
gamma_v = atand((obj.hohe - h)/(d));
obj.gv.setgvarr(gamma_v);
end
end
end
finally before creating any building you should create an object of gv class and pass it to the building constructor,
Egv=gvclass();
H1 = building(10,8,6,14,8,Egv,2)
H2 = building(18,8,6,14,0,Egv,3)
and to access gv array:
Egv.gvarr
you should do more effort on this issue to debug possible errors.

How/When to update bias in RPROP neural network?

I am implementing this neural network for some classification problem. I initially tried back propagation but it takes longer to converge. So I though of using RPROP. In my test setup RPROP works fine for AND gate simulation but never converges for OR and XOR gate simulation.
How and when should I update bias for RPROP?
Here my weight update logic:
for(int l_index = 1; l_index < _total_layers; l_index++){
Layer* curr_layer = get_layer_at(l_index);
//iterate through each neuron
for (unsigned int n_index = 0; n_index < curr_layer->get_number_of_neurons(); n_index++) {
Neuron* jth_neuron = curr_layer->get_neuron_at(n_index);
double change = jth_neuron->get_change();
double curr_gradient = jth_neuron->get_gradient();
double last_gradient = jth_neuron->get_last_gradient();
int grad_sign = sign(curr_gradient * last_gradient);
//iterate through each weight of the neuron
for(int w_index = 0; w_index < jth_neuron->get_number_of_weights(); w_index++){
double current_weight = jth_neuron->give_weight_at(w_index);
double last_update_value = jth_neuron->give_update_value_at(w_index);
double new_update_value = last_update_value;
if(grad_sign > 0){
new_update_value = min(last_update_value*1.2, 50.0);
change = sign(curr_gradient) * new_update_value;
}else if(grad_sign < 0){
new_update_value = max(last_update_value*0.5, 1e-6);
change = -change;
curr_gradient = 0.0;
}else if(grad_sign == 0){
change = sign(curr_gradient) * new_update_value;
}
//Update neuron values
jth_neuron->set_change(change);
jth_neuron->update_weight_at((current_weight + change), w_index);
jth_neuron->set_last_gradient(curr_gradient);
jth_neuron->update_update_value_at(new_update_value, w_index);
double current_bias = jth_neuron->get_bias();
jth_neuron->set_bias(current_bias + _learning_rate * jth_neuron->get_delta());
}
}
}
In principal you don't treat the bias differently than before when you did backpropagation. It's learning_rate * delta which you seem to be doing.
One source of error may be that the sign of the weight change depends on how you calculate your error. There's different conventions and (t_i-y_i) instead of (y_i - t_i) should result in returning (new_update_value * sgn(grad)) instead of -(new_update_value * sign(grad)) so try switching the sign. I'm also unsure about how you specifically implemented everything since a lot is not shown here. But here's a snippet of mine in a Java implementation that might be of help:
// gradient didn't change sign:
if(weight.previousErrorGradient * errorGradient > 0)
weight.lastUpdateValue = Math.min(weight.lastUpdateValue * step_pos, update_max);
// changed sign:
else if(weight.previousErrorGradient * errorGradient < 0)
{
weight.lastUpdateValue = Math.max(weight.lastUpdateValue * step_neg, update_min);
}
else
weight.lastUpdateValue = weight.lastUpdateValue; // no change
// Depending on language, you should check for NaN here.
// multiply this with -1 depending on your error signal's sign:
return ( weight.lastUpdateValue * Math.signum(errorGradient) );
Also, keep in mind that 50.0, 1e-6 and especially 0.5, 1.2 are empirically gathered values so they might need to be adjusted. You should definitely print out the gradients and weight changes to see if there's something weird going on (e.g. exploding gradients->NaN although you're only testing AND/XOR). Your last_gradient value should also be initialized to 0 at the first timestep.