How to write alternative constraint with interval variable in or-tools? - or-tools

How to write alternative constraint with interval variable in or-tools? I thought something like this would work but the method AddAlternative doesn´t exist. Another question is how to know if interval variable is active
from ortools.sat.python import cp_model
# Crear el modelo
model = cp_model.CpModel()
# Crear la variable de intervalo
interval = model.NewIntervalVar(start, end, duration, 'interval')
# Crear los intervalos alternativos
alt_interval1 = model.NewIntervalVar(start1, end1, duration1, 'alt_interval1')
alt_interval2 = model.NewIntervalVar(start2, end2, duration2, 'alt_interval2')
# Agregar los intervalos alternativos a la variable de intervalo
model.AddAlternative(interval, [alt_interval1, alt_interval2])
# Resolver el modelo
solver = cp_model.CpSolver()
status = solver.Solve(model)
My code is:
# VARIABLES#
x = [
model.NewIntervalVar(
start=model.NewIntVar(es, lf, f"start_{row.id_pozo}"),
size=model.NewIntVar(es, lf, f"size_{row.id_pozo}"),
end=model.NewIntVar(es, lf, f"end_{row.id_pozo}"),
name="pozo_intv_{}".format(row.id_pozo),
)
for row in pozos.itertuples()
]
y = [
model.NewOptionalIntervalVar(
start=model.NewIntVar(row.dia_inicio, row.dia_fin, f"start_{idx}"),
size=row.tiempo_total,
end=model.NewIntVar(row.dia_inicio, row.dia_fin, f"end_{idx}"),
is_present=True,
name="pte_intv_{}".format(idx),
)
for idx, row in pozo_time_equipment.iterrows()
]

This is a flexible job shop problem. In the OR tools source there is a working example. The basics are, firstly creating the main interval for each task:
# Create main interval for the task.
suffix_name = '_j%i_t%i' % (job_id, task_id)
start = model.NewIntVar(0, horizon, 'start' + suffix_name)
duration = model.NewIntVar(min_duration, max_duration,
'duration' + suffix_name)
end = model.NewIntVar(0, horizon, 'end' + suffix_name)
interval = model.NewIntervalVar(start, duration, end,
'interval' + suffix_name)
Then for each alternative, create an additional interval:
alt_suffix = '_j%i_t%i_a%i' % (job_id, task_id, alt_id)
l_presence = model.NewBoolVar('presence' + alt_suffix)
l_start = model.NewIntVar(0, horizon, 'start' + alt_suffix)
l_duration = task[alt_id][0]
l_end = model.NewIntVar(0, horizon, 'end' + alt_suffix)
l_interval = model.NewOptionalIntervalVar(
l_start, l_duration, l_end, l_presence,
'interval' + alt_suffix)
l_presences.append(l_presence)
The alternative is linked to the main interval, only if the alternative is selected:
# Link the primary/global variables with the local ones.
model.Add(start == l_start).OnlyEnforceIf(l_presence)
model.Add(duration == l_duration).OnlyEnforceIf(l_presence)
model.Add(end == l_end).OnlyEnforceIf(l_presence)
Then finally, add a constraint to ensure that only one of the alternatives is selected:
# Select exactly one presence variable.
model.AddExactlyOne(l_presences)

Related

Ortools VRP error when trying to merge penalties and different start-end depots

I'm trying to solve a VRP problem allowing dropping nodes through penalties and multiple depots.
Code works fine with penalties and vehicles starting and ending at the same depots:
data['num_vehicles'] = 2
data['start'] = [0, 4]
data['end'] = [0, 4]
Code works fine with vehicles starting and ending at different depots, commenting the for loop to AddDisjunction for penalties (without allowing dropping nodes):
data['num_vehicles'] = 2
data['start'] = [0, 4]
data['end'] = [0, 9]
.........................
#for node in range(1, len(data['penalties'])):
# routing.AddDisjunction([manager.NodeToIndex(node)], data['penalties'][node])
But with vehicles starting and ending at different depots and trying to add penalties to allow dropping nodes, python crashes with the error (debugging I can see that fails at adding the penalty of the different end depot):
F00-1 -1:-1:-1.000244 24944 routing.cc:1622] Check failed: kUnassigned != indices[i] (-1 vs. -1)
I cannot find any reference about this error. I looked at routing.cc source code around line 1622, but I cannot see any relation with the error. I need help to fix the problem.
Here is the souce code:
from ortools.constraint_solver import routing_enums_pb2
from ortools.constraint_solver import pywrapcp
def create_data_model():
"""Stores the data for the problem."""
data = {}
data['distance_matrix'] = [
[0, 2253, 2479, 2792, 4707, 6128, 1567, 5643, 1234, 3345, 1827],
[1731, 0, 2193, 2507, 3624, 5040, 3467, 2921, 5791, 1546, 2345],
[1867, 2112, 0, 676, 4406, 5824, 988, 4567, 2134, 4453, 2123],
[2339, 2585, 893, 0, 4879, 6302, 1543, 1298, 6890, 1456, 5623],
[4464, 3766, 5935, 4957, 0, 1749, 987, 3212, 3451, 5212, 3321],
[6568, 5862, 8023, 7055, 2148, 0, 4567, 2124, 4321, 3212, 1234],
[731, 2193, 2507, 7624, 4040, 4467, 0, 2621, 3791, 1567, 1345],
[1731, 3193, 1507, 3624, 6040, 2467, 4921, 0, 5791, 6723, 1345],
[2731, 3193, 2507, 6204, 5040, 1467, 2210, 6791, 0, 2567, 6421],
[3345, 1543, 4421, 1531, 5213, 3215, 1512, 6213, 2431, 0, 5673],
[1832, 2421, 2144, 5232, 3214, 1234, 1432, 1231, 6321, 5461, 0],
]
data['node_time'] = [0, 7200, 3600, 5400, 0, 5400, 7200, 3600, 7200, 0, 300]
data['num_vehicles'] = 2
data['start'] = [0, 4]
data['end'] = [0, 9]
# Penalizaciones por no visitar nodos (drop nodes) en caso de que no tenga solución:
# MAXINT 0x7FFFFFFFFFFFFFF: Hace obligatoria la visita al nodo, no se puede eliminar de la solución
# 1000000: Se puede no visitar el nodo con penalización. La penalización debe ser grande, mayor que 10x veces el mayor tiempo de visita, para evitar que salga mas rentable dejar caer el nodo que pagar la penalización
# 0: En los nodos "depósito" de vehículos, no son visitas intermedias sino inicio y fin de la ruta.
data['penalties'] = [0, 1000000, 1000000, 1000000, 0, 0x7FFFFFFFFFFFFFF, 0x7FFFFFFFFFFFFFF, 1000000, 1000000, 0, 1000000]
return data
def print_solution(data, manager, routing, solution):
"""Prints solution on console."""
print(f'Objective: {solution.ObjectiveValue()}')
# Display dropped nodes.
dropped_nodes = 'Nodos sin visitar:'
for node in range(routing.Size()):
if routing.IsStart(node) or routing.IsEnd(node):
continue
if solution.Value(routing.NextVar(node)) == node:
dropped_nodes += ' {}'.format(manager.IndexToNode(node))
print(dropped_nodes + '\n')
max_route_distance = 0
for vehicle_id in range(data['num_vehicles']):
index = routing.Start(vehicle_id)
plan_output = 'Ruta para vehículo {}:\n'.format(vehicle_id)
route_distance = 0
while not routing.IsEnd(index):
plan_output += ' {} -> '.format(manager.IndexToNode(index))
previous_index = index
index = solution.Value(routing.NextVar(index))
route_distance += routing.GetArcCostForVehicle(
previous_index, index, vehicle_id)
plan_output += '{}\n'.format(manager.IndexToNode(index))
plan_output += 'Tiempo de la ruta: {}sg\n'.format(route_distance)
print(plan_output)
max_route_distance = max(route_distance, max_route_distance)
print('Maximo tiempo de las rutas: {}sg'.format(max_route_distance))
def main():
"""Entry point of the program."""
# Instantiate the data problem.
data = create_data_model()
# Create the routing index manager.
manager = pywrapcp.RoutingIndexManager(len(data['distance_matrix']),
data['num_vehicles'], data['start'], data['end'])
# Create Routing Model.
routing = pywrapcp.RoutingModel(manager)
# Create and register a transit callback.
def distance_callback(from_index, to_index):
"""Returns the distance between the two nodes."""
# Convert from routing variable Index to distance matrix NodeIndex.
from_node = manager.IndexToNode(from_index)
to_node = manager.IndexToNode(to_index)
tiempo_desplazamiento = data['distance_matrix'][from_node][to_node]
tiempo_ejecucion = data['node_time'][to_node]
return tiempo_desplazamiento + tiempo_ejecucion
transit_callback_index = routing.RegisterTransitCallback(distance_callback)
# Define cost of each arc.
routing.SetArcCostEvaluatorOfAllVehicles(transit_callback_index)
# Add Distance constraint.
routing.AddDimension(
transit_callback_index,
0, # no slack
27000, # vehicle maximum travel distance (7.5 hours, in seconds)
True, # start cumul to zero
'Time')
distance_dimension = routing.GetDimensionOrDie('Time')
distance_dimension.SetGlobalSpanCostCoefficient(100)
# Allow to drop nodes.
for node in range(1, len(data['penalties'])):
routing.AddDisjunction([manager.NodeToIndex(node)], data['penalties'][node])
# Setting first solution heuristic.
search_parameters = pywrapcp.DefaultRoutingSearchParameters()
search_parameters.first_solution_strategy = (routing_enums_pb2.FirstSolutionStrategy.PATH_CHEAPEST_ARC)
search_parameters.local_search_metaheuristic = (routing_enums_pb2.LocalSearchMetaheuristic.GUIDED_LOCAL_SEARCH)
search_parameters.time_limit.seconds = 30
# Solve the problem.
solution = routing.SolveWithParameters(search_parameters)
# Print solution on console.
if solution:
print_solution(data, manager, routing, solution)
else:
print('No solution found!')
if __name__ == '__main__':
main()
I've posted the question in Ortools Google Group, with additional research:
[https://groups.google.com/g/or-tools-discuss/c/s3PfgLVZpj0][1]
Code seems to be working excluding start and end nodes from the disjunction as explained on that post, but I asked for more info to understand how it works.
With custom start and ends, you should use Routing.Start(vehicle_index) and Routing.End(vehicle_index) to get the index of these nodes.
In routing.h there is a comment about AddDisjunction
/// Adds a disjunction constraint on the indices: exactly 'max_cardinality' of
/// the indices are active. Start and end indices of any vehicle cannot be
/// part of a disjunction.
In the for loop to add nodes to the disjunction, excluding start and end nodes seems to be working:
# Allow to drop nodes.
for node in range(0, len(data['distance_matrix'])):
if(node in data['start'] or node in data['end']):
continue
else:
routing.AddDisjunction([manager.NodeToIndex(node)], data['penalties'][node])

OR Tools Constraint that requires equility of variables [ortools]

I want create a linear solver model using OR-Tools. I have two numVars and one of my constraints is the equility of these two variables. But I could not find a way to set a constraint that takes a variable. I can do this with gurobi library. Below are the codes for both libraries. Is there a way to pass variables to both sides of a constraint such as xa==xb?
Thanks in advance.
Gurobi Code:
from gurobipy import Model, GRB
m = Model('rafinery')
xa = m.addVar(vtype=GRB.CONTINUOUS)
xb = m.addVar(vtype=GRB.CONTINUOUS)
# production constraint
m.addConstr(xa*400 + xb*300 >= 25000)
m.addConstr(xa*300 + xb*400 >= 27000)
m.addConstr(xa*200 + xb*500 >= 30000)
m.addConstr(xa>=0)
m.addConstr(xb>=0)
m.addConstr(xa==xb)
m.setObjective(xa * 20000 + xb * 25000, GRB.MINIMIZE)
m.update()
m.setObjective(xa + xb, GRB.MINIMIZE)
m.update()
m.optimize()
for v in m.getVars():
print(v.varName, v.x)
print("Cost:", m.objVal)
OR-Tools Code:
from ortools.linear_solver import pywraplp
solver = pywraplp.Solver('SolveSimpleSystem', pywraplp.Solver.GLOP_LINEAR_PROGRAMMING)
# Create Variables
xa = solver.NumVar(0, solver.infinity(),'xa')
xb = solver.NumVar(0, solver.infinity(),'xb')
# Create Constraints
# xa*400 + xb*300 >= 25000
high = solver.Constraint(25000, solver.infinity())
high.SetCoefficient(xa, 400)
high.SetCoefficient(xb, 300)
# xa*300 + xb*400 >= 27000
middle = solver.Constraint(27000, solver.infinity())
middle.SetCoefficient(xa,300)
middle.SetCoefficient(xb,400)
# xa*200 + xb*500 >= 30000
high = solver.Constraint(30000, solver.infinity())
high.SetCoefficient(xa, 200)
high.SetCoefficient(xb, 500)
# another constraint that factories work for same days
# ******************** get error here ****************
noidle = solver.Constraint(xa,xa)
noidle.SetCoefficient(xb,1)
# obj = minimize production cost which is
# xa * 20000 + xb * 25000
obj = solver.Objective()
obj.SetCoefficient(xa, 20000)
obj.SetCoefficient(xb, 25000)
obj.SetMinimization()
solver.Solve()
print("Rafinery Solution:")
print("Rafinery A work days:", xa.solution_value())
print("Rafinery B work days:", xb.solution_value())
print("Cost : ", solver.Objective().Value())
from ortools.linear_solver import pywraplp
solver = pywraplp.Solver('SolveSimpleSystem', pywraplp.Solver.GLOP_LINEAR_PROGRAMMING)
# Create Variables
xa = solver.NumVar(0, solver.infinity(),'xa')
xb = solver.NumVar(0, solver.infinity(),'xb')
# production constraint
solver.Add(xa*400 + xb*300 >= 25000)
solver.Add(xa*300 + xb*400 >= 27000)
solver.Add(xa*200 + xb*500 >= 30000)
solver.Add(xa>=0)
solver.Add(xb>=0)
solver.Add(xa==xb)
solver.Minimize(xa * 20000 + xb * 25000)
solver.Solve()
print("Rafinery Solution:")
print("Rafinery A work days:", xa.solution_value())
print("Rafinery B work days:", xb.solution_value())
print("Cost : ", solver.Objective().Value())

Genetic Algorithm and neural network failing to learn

I am trying to make a Flappy Bird AI where the agent tries to learn to pass through the pipes via genetic algorithms and neural network.
My implementation is that I am using a neural network with two inputs (horizontal distance from pipe and vertical distance from pipe opening), one hidden layer of 5 neurons, and one output layer.
The genetic algorithm evolves the agent by constantly changing one of the weights of the neural network per generation. (based on this GA implementation)
However I noticed that the flappy bird agent is failing to learn, to the point where it never even attempts to flap once (the entire time, it keeps falling at the beginning of every generation) all the way until the 485th generation (where an "maximum recursion depth exceeded" error occurs)
Genetic Algorithm + Neural Network functions:
def flap(playery, playerFlapAcc):
playerVelY = playerFlapAcc
playerFlapped = True
if sound:
SOUNDS['wing'].play()
return playerVelY, playerFlapped
def mutate(n):
global in_to_hidden
global hidden_to_hidden2
global hidden_to_out
layer_select = random.uniform(0,2)
print 'Changed:'
if layer_select == 0:
selector = int(random.uniform(0,5))
print in_to_hidden.params[selector]
in_to_hidden.params[selector] = in_to_hidden.params[selector] + random.uniform(-5,5)
print in_to_hidden.params[selector]
elif layer_select == 1:
selector = int(random.uniform(0,5))
print hidden_to_hidden.params[selector]
hidden_to_hidden2.params[selector] = hidden_to_hidden2.params[selector] + random.uniform(-5,5)
print hidden_to_hidden.params[selector]
else:
selector = int(random.uniform(0,3))
print hidden_to_out.params[selector]
hidden_to_out.params[selector] = hidden_to_out.params[selector] + random.uniform(-5,5)
print hidden_to_out.params[selector]
return n
def predict_action(rangex, error, playery, playerFlapAcc, playerVelY, playerFlapped, i):
global jumped
if i % 10 == 0:
pred = n.activate([rangex, error]).argmax()
if pred == 1:
jumped = True
playerVelY, playerFlapped = flap(playery, playerFlapAcc)
return playerVelY, playerFlapped
def initalize_nn():
global in_to_hidden
global hidden_to_hidden2
global hidden_to_out
# Old code (regression)
n = FeedForwardNetwork()
# n = buildNetwork( 2, 3, data.outdim, outclass=SoftmaxLayer )
inLayer = LinearLayer(2)
hiddenLayer = SigmoidLayer(5)
hiddenLayer2 = SigmoidLayer(5)
outLayer = LinearLayer(1)
n.addInputModule(inLayer)
n.addModule(hiddenLayer)
n.addModule(hiddenLayer2)
n.addOutputModule(outLayer)
in_to_hidden = FullConnection(inLayer, hiddenLayer)
hidden_to_hidden2 = FullConnection(hiddenLayer, hiddenLayer2)
hidden_to_out = FullConnection(hiddenLayer2, outLayer)
n.addConnection(in_to_hidden)
n.addConnection(hidden_to_hidden2)
n.addConnection(hidden_to_out)
n.sortModules()
return n
def fitness_fun(score, x_distance, error):
# Fitness function was designed so that the largest distance is
# the most fit. Before going through the first pipe, total distance traveled is the fitness.
# Once agent passed through the first pipe and earned a point,
# the amount of points it gained is the main determinant of the fitness score
if error != 0:
fitval = abs((100*score) + (x_distance/(2*abs(error))))
else:
fitval = abs(100*score) + x_distance*2
return fitval
Sample implementation in the game:
def mainGame(movementInfo):
global fitness
global old_fitness
global num_nn_parameters
global score
global disx
global first_time
global n
global old_n
global in_to_hidden
global hidden_to_hidden2
global hidden_to_out
global generation
global jumped
print 'generation: ', generation
generation = generation + 1
if first_time:
### Initalizing the neural network
n = initalize_nn()
ds = ClassificationDataSet(2, nb_classes=2)
z = 0
for val in in_to_hidden.params:
in_to_hidden.params[z] = random.uniform(-2,2)
z = z + 1
num_nn_parameters = z
old_nn = n
else:
# create new nn (but with old_nn saved)
n = mutate(old_n)
disx = 0
score = 0
first_time = False
# Print weights
print_all_weights()
####
'''
NOTES:
playerx = player's x position (57)
playery = player's height
upper_gap
lower_gap
center_cord
'''
pipeHeight = IMAGES['pipe'][0].get_height()
upper_gap = newPipe1[0]['y'] + pipeHeight
lower_gap = upper_gap + PIPEGAPSIZE
center_cord = upper_gap + ((lower_gap - upper_gap)/2)
########### The main loop ###########
#playerx = 140
while True:
i = i + 1
disx = disx + 1
# Error is determined by comparing the agent's y distance from the pipe opening
error = playery - center_cord
for event in pygame.event.get():
if event.type == QUIT or (event.type == KEYDOWN and event.key == K_ESCAPE):
pygame.quit()
sys.exit()
if event.type == KEYDOWN and (event.key == K_SPACE or event.key == K_UP):
if playery > -2 * IMAGES['player'][0].get_height():
playerVelY = playerFlapAcc
playerFlapped = True
if sound:
SOUNDS['wing'].play()
# check for crash here
crashTest = checkCrash({'x': playerx, 'y': playery, 'index': playerIndex},
upperPipes, lowerPipes)
if crashTest[0]:
fitness = fitness_fun(score, disx, error)
print '------------------- Game Over ---------------------'
print 'fitness: [', fitness, ']'
print 'old fit: [', old_fitness, ']'
print ''
print ''
print 'error: ', error
#print 'score: ', score
print 'range_x', rangex
print 'player_x: ', disx
print '----------------------------------------------------'
print ''
print ''
print ''
print ''
print ''
# If it turns out the old nn was better
if old_fitness > fitness:
# prevents the old but good nn from being overwritten
n = old_n
fitness = old_fitness
else:
print 'Better fitness discovered'
# store the good nn as the old_nn
old_n = n
old_fitness = fitness
return {
'y': playery,
'groundCrash': crashTest[1],
'basex': basex,
'upperPipes': upperPipes,
'lowerPipes': lowerPipes,
'score': score,
'playerVelY': playerVelY,
}
rangex = upperPipes[0]['x'] - 92
# Make prediction
playerVelY, playerFlapped = predict_action(rangex, error, playery, playerFlapAcc, playerVelY, playerFlapped, i)
Does anyone know the cause of this and how I can fix this?

MATLAB Execution Time Increasing

Here is my code. The intent is I have a Wireshark capture saved to a particularly formatted text file. The MATLAB code is supposed to go through the Packets, dissect them for different protocols, and then make tables based on those protocols. I currently have this programmed for ETHERNET/IP/UDP/MODBUS. In this case, it creates a column in MBTable each time it encounters a new register value, and each time it comes across a change to that register value, it updates the value in that line of the table. The first column of MBTable is time, the registers start with the second column.
MBTable is preallocated to over 100,000 Rows (nol is very large), 10 columns before this code is executed. The actual data from a file I'm pulling into the table gets to about 10,000 rows and 4 columns and the code execution is so slow I have to stop it. The tic/toc value is calculated every 1000 rows and continues to increase exponentially with every iteration. It is a large loop, but I can't see where anything is growing in such a way that it would cause it to run slower with each iteration.
All variables get initialized up top (left out to lessen amount of code.
The variables eth, eth.ip, eth.ip.udp, and eth.ip.udp.modbus are all of type struct as is eth.header and eth.ip.header. WSID is a file ID from a .txt file opened earlier.
MBTable = zeros(nol,10);
tval = tic;
while not(feof(WSID))
packline = packline + 1;
fl = fl + 1;
%Get the next line from the file
MBLine = fgetl(WSID);
%Make sure line is not blank or short
if length(MBLine) >= 3
%Split the line into 1. Line no, 2. Data, 3. ASCII
%MBAll = strsplit(MBLine,' ');
%First line of new packet, if headers included
if strcmp(MBLine(1:3),'No.')
newpack = true;
newtime = false;
newdata = false;
stoppack = false;
packline = 1;
end
%If packet has headers, 2nd line contains timestamp
if newpack
Ordered = false;
if packline == 2;
newtime = true;
%MBstrs = strsplit(MBAll{2},' ');
packno = int32(str2double(MBLine(1:8)));
t = str2double(MBLine(9:20));
et = t - lastt;
if lastt > 0 && et > 0
L = L + 1;
MBTable(L,1) = t;
end
%newpack = false;
end
if packline > 3
dataline = int16(str2double(MBLine(1:4)));
packdata = strcat(packdata,MBLine(7:53));
end
end
else
%if t >= st
if packline > 3
stoppack = true;
newpack = false;
end
if stoppack
invalid = false;
%eth = struct;
eth.pack = packdata(~isspace(packdata));
eth.length = length(eth.pack);
%Dissect the packet data
eth.stbyte = 1;
eth.ebyte = eth.length;
eth.header.stbyte = 1;
eth.header.ebyte = 28;
%Ethernet Packet Data
eth.header.pack = eth.pack(eth.stbyte:eth.stbyte+27);
eth.header.dest = eth.header.pack(eth.header.stbyte:eth.header.stbyte + 11);
eth.header.src = eth.header.pack(eth.header.stbyte + 12:eth.header.stbyte + 23);
eth.typecode = eth.header.pack(eth.header.stbyte + 24:eth.header.ebyte);
if strcmp(eth.typecode,'0800')
eth.type = 'IP';
%eth.ip = struct;
%IP Packet Data
eth.ip.stbyte = eth.header.ebyte + 1;
eth.ip.ver = eth.pack(eth.ip.stbyte);
%IP Header length
eth.ip.header.length = 4*int8(str2double(eth.pack(eth.ip.stbyte+1)));
eth.ip.header.ebyte = eth.ip.stbyte + eth.ip.header.length - 1;
%Differentiated Services Field
eth.ip.DSF = eth.pack(eth.ip.stbyte + 2:eth.ip.stbyte + 3);
%Total IP Packet Length
eth.ip.length = hex2dec(eth.pack(eth.ip.stbyte+4:eth.ip.stbyte+7));
eth.ip.ebyte = eth.ip.stbyte + max(eth.ip.length,46) - 1;
eth.ip.pack = eth.pack(eth.ip.stbyte:eth.ip.ebyte);
eth.ip.ID = eth.pack(eth.ip.stbyte+8:eth.ip.stbyte+11);
eth.ip.flags = eth.pack(eth.ip.stbyte+12:eth.ip.stbyte+13);
eth.ip.fragoff = eth.pack(eth.ip.stbyte+14:eth.ip.stbyte+15);
%Time to Live
eth.ip.ttl = hex2dec(eth.pack(eth.ip.stbyte+16:eth.ip.stbyte+17));
eth.ip.typecode = eth.pack(eth.ip.stbyte+18:eth.ip.stbyte+19);
eth.ip.checksum = eth.pack(eth.ip.stbyte+20:eth.ip.stbyte+23);
%eth.ip.src = eth.pack(eth.ip.stbyte+24:eth.ip.stbyte+31);
eth.ip.src = ...
[num2str(hex2dec(eth.pack(eth.ip.stbyte+24:eth.ip.stbyte+25))),'.', ...
num2str(hex2dec(eth.pack(eth.ip.stbyte+26:eth.ip.stbyte+27))),'.', ...
num2str(hex2dec(eth.pack(eth.ip.stbyte+28:eth.ip.stbyte+29))),'.', ...
num2str(hex2dec(eth.pack(eth.ip.stbyte+30:eth.ip.stbyte+31)))];
eth.ip.dest = ...
[num2str(hex2dec(eth.pack(eth.ip.stbyte+32:eth.ip.stbyte+33))),'.', ...
num2str(hex2dec(eth.pack(eth.ip.stbyte+34:eth.ip.stbyte+35))),'.', ...
num2str(hex2dec(eth.pack(eth.ip.stbyte+36:eth.ip.stbyte+37))),'.', ...
num2str(hex2dec(eth.pack(eth.ip.stbyte+38:eth.ip.stbyte+39)))];
if strcmp(eth.ip.typecode,'11')
eth.ip.type = 'UDP';
eth.ip.udp.stbyte = eth.ip.stbyte + 40;
eth.ip.udp.src = hex2dec(eth.pack(eth.ip.udp.stbyte:eth.ip.udp.stbyte + 3));
eth.ip.udp.dest = hex2dec(eth.pack(eth.ip.udp.stbyte+4:eth.ip.udp.stbyte+7));
eth.ip.udp.length = hex2dec(eth.pack(eth.ip.udp.stbyte+8:eth.ip.udp.stbyte+11));
eth.ip.udp.checksum = eth.pack(eth.ip.udp.stbyte+12:eth.ip.udp.stbyte+15);
eth.ip.udp.protoID = eth.pack(eth.ip.udp.stbyte+20:eth.ip.udp.stbyte+23);
if strcmp(eth.ip.udp.protoID,'0000')
eth.ip.udp.proto = 'MODBUS';
%eth.ip.udp.modbus = struct;
eth.ip.udp.modbus.stbyte = eth.ip.udp.stbyte+16;
eth.ip.udp.modbus.transID = eth.pack(eth.ip.udp.modbus.stbyte:eth.ip.udp.modbus.stbyte+3);
eth.ip.udp.modbus.protoID = eth.ip.udp.protoID;
eth.ip.udp.modbus.length = int16(str2double(eth.pack(eth.ip.udp.modbus.stbyte + 8:eth.ip.udp.modbus.stbyte + 11)));
eth.ip.udp.modbus.UID = eth.pack(eth.ip.udp.modbus.stbyte + 12:eth.ip.udp.modbus.stbyte + 13);
eth.ip.udp.modbus.func = hex2dec(eth.pack(eth.ip.udp.modbus.stbyte + 14:eth.ip.udp.modbus.stbyte+15));
eth.ip.udp.modbus.register = eth.pack(eth.ip.udp.modbus.stbyte + 16: eth.ip.udp.modbus.stbyte+19);
%Number of words to a register, or the number of registers
eth.ip.udp.modbus.words = hex2dec(eth.pack(eth.ip.udp.modbus.stbyte+20:eth.ip.udp.modbus.stbyte+23));
eth.ip.udp.modbus.bytes = hex2dec(eth.pack(eth.ip.udp.modbus.stbyte+24:eth.ip.udp.modbus.stbyte+25));
eth.ip.udp.modbus.data = eth.pack(eth.ip.udp.modbus.stbyte + 26:eth.ip.udp.modbus.stbyte + 26 + 2*eth.ip.udp.modbus.bytes - 1);
%If func 16 or 23, loop through data/registers and add to table
if eth.ip.udp.modbus.func == 16 || eth.ip.udp.modbus.func == 23
stp = eth.ip.udp.modbus.bytes*2/eth.ip.udp.modbus.words;
for n = 1:stp:eth.ip.udp.modbus.bytes*2;
%Check for existence of register as a key?
if ~isKey(MBMap,eth.ip.udp.modbus.register)
MBCol = MBCol + 1;
MBMap(eth.ip.udp.modbus.register) = MBCol;
end
MBTable(L,MBCol) = hex2dec(eth.ip.udp.modbus.data(n:n+stp-1));
eth.ip.udp.modbus.register = dec2hex(hex2dec(eth.ip.udp.modbus.register)+1);
end
lastt = t;
end
%If func 4, make sure it is the response, then put
%data into table for register column
elseif false
%need code to handle serial to UDP conversion box
else
invalid = true;
end
else
invalid = true;
end
else
invalid = true;
end
if ~invalid
end
end
%end
end
%Display Progress
if int64(fl/1000)*1000 == fl
for x = 1:length(mess);
fprintf('\b');
end
%fprintf('Lines parsed: %i',fl);
mess = sprintf('Lines parsed: %i / %i',fl,nol);
fprintf('%s',mess);
%Check execution time - getting slower:
%%{
ext = toc(tval);
mess = sprintf('\nExecution Time: %f\n',ext);
fprintf('%s',mess);
%%}
end
end
ext = toc - exst;
Update: I updated my code above to remove the overloaded operators (disp and lt were replaced with mess and lastt)
Was asked to use the profiler, so I limited to 2000 lines in the table (added && L >=2000 to the while loop) to limit the execution time, and here are the top results from the profiler:
SGAS_Wireshark_Parser_v0p7_fulleth 1 57.110 s 9.714 s
Strcat 9187 29.271 s 13.598 s
Blanks 9187 15.673 s 15.673 s
Uigetfile 1 12.226 s 0.009 s
uitools\private\uigetputfile_helper 1 12.212 s 0.031 s
FileChooser.FileChooser>FileChooser.show 1 12.085 s 0.006s
...er>FileChooser.showPeerAndBlockMATLAB 1 12.056 s 0.001s
...nChooser>FileOpenChooser.doShowDialog 1 12.049 s 12.049 s
hex2dec 44924 2.944 s 2.702 s
num2str 16336 1.139 s 0.550 s
str2double 17356 1.025 s 1.025 s
int2str 16336 0.589 s 0.589 s
fgetl 17356 0.488 s 0.488 s
dec2hex 6126 0.304 s 0.304 s
fliplr 44924 0.242 s 0.242 s
It appears to be strcat calls that are doing it. I only explicitly call strcat on one line. Are some of the other string manipulations I'm doing calling strcat indirectly?
Each loop should be calling strcat the same number of times though, so I still don't understand why it takes longer and longer the more it runs...
also, hex2dec is called a lot, but is not really affecting the time.
But anyway, are there any other methods I can use the combine the strings?
Here is the issue:
The string (an char array in MATLAB) packdata was being resized and reallocated over and over again. That's what was slowing down this code. I did the following steps:
I eliminated the redundant variable packdata and now only use eth.pack.
I preallocated eth.pack and a couple "helper variables" of known lengths by running blanks ONCE for each before the loop ever starts
eth.pack = blanks(604);
thisline = blanks(47);
smline = blanks(32);
(Note: 604 is the maximum possible size of packdata based on headers + MODBUS protocol)
Then I created a pointer variable to point to the location of the last char written to packdata.
pptr = 1;
...
dataline = int16(str2double(MBLine(1:4)));
thisline = MBLine(7:53); %Always 47 characters
smline = [thisline(~isspace(thisline)),blanks(32-sum(~isspace(thisline)))]; %Always 32 Characters
eth.pack(pptr:pptr+31) = smline;
pptr = pptr + 32;
The above was inside the 'if packline > 3' block in place of the 'packdata =' statement, then at the end of the 'if stoppack' block was the reset statement:
pptr = 1; %Reset Pointer
FYI, not surprisingly this brought out other flaws in my code which I've mostly fixed but still need to finish. Not a big issue now as this loop executes lightning fast with these changes. Thanks to Yvon for helping point me in the right direction.
I kept thinking my huge table, MBTable was the issue... but it had nothing to do with it.

programming a range of data : Matlab

I have this program, something related with statistic.
maximo = max(muestra);
minimo = min(muestra);
rango = maximo - minimo;
num_intervalo = round(1 + 3.322*log(length(muestra)));
amplitud = rango/num_intervalo;
rango_intervalo = [];
for i=1 : num_intervalo + 1
if i == 1
rango_intervalo(i: num_intervalo + 1) = minimo;
else
rango_intervalo(i: num_intervalo + 1) = rango_tabulado(i) + amplitud;
end
if i == num_intervalo + 1
rango_intervalo(i: num_intervalo + 1) = maximo;
end
end
rango_intervalo = rango_intervalo';
the intention is to create nine (or k intervals) intervals, where each interval has it ranges:
[1.580 - 2.587]
[2.587 - 3.594]
.
.
[9.636 - 10.650]
With code that I've programmed, it is resulting in 10 data not nine as per the intention.
Any idea, the improve this code?
Thanks.
How about:
intervals = linspace(minimo, maximo, num_intervalo + 1 );
intervals = [ intervals(1:end-1); intervals(2:end) ]';