Keras history order problem when saving to GDrive - callback

My aim is to save Keras logs (accuracies, losses, etc) to Google Drive after every epoch
I am using the following code:
from google.colab import drive
drive.mount('/content/drive')
class HistoryCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs={}):
with open("drive/My Drive/"+csv_path, "a") as myfile:
myfile.write(str(epoch)+","+str(logs)+"\n")
classifier.fit_generator(..., callbacks=[HistoryCallback()])
On my drive, I sometimes get strange output, like this:
25 {'acc': 0.963835932997043 loss': 0.10425430848152908 val_acc': 0.7071953016230713 val_loss': 1.1386645126622854 lr': 2.3961632e-06}
2366427 loss': 1.1117452404459112 val_acc': 0.5577092514076597 val_loss': 1.0548135743792362 lr': 4.980681e-06}
2 {'acc': 0.6330444829612712 loss': 0.9205646275682026 val_acc': 0.5994126287500939 val_loss': 0.9518886575614829 lr': 4.956604e-06}
3 {'acc': 0.6983824379057777 loss': 0.7753418573921365 val_acc': 0.6314243757777277 val_loss': 0.8930798317542336 lr': 4.923029e-06}
4 {'acc': 0.7437319468601393 loss': 0.6659318362681732 val_acc': 0.6464023491359492 val_loss': 0.866631023106596 lr': 4.8800885e-06}
5 {'acc': 0.7798526863016054 loss': 0.5813610065455109 val_acc': 0.6637298091742786 val_loss': 0.8554121221564764 lr': 4.8279526e-06}
6 {'acc': 0.8090410167584868 loss': 0.5098161270401851 val_acc': 0.6657856092579039 val_loss': 0.8496283279291509 lr': 4.7668264e-06}
7 {'acc': 0.8317157712132858 loss': 0.45465362796302755 val_acc': 0.6734214392352559 val_loss': 0.8745797048056179 lr': 4.6969512e-06}
8 {'acc': 0.8491478913819287 loss': 0.4042509938624124 val_acc': 0.688986784105959 val_loss': 0.8465897937878288 lr': 4.6186033e-06}
....
As you can see, the ordering is messed up, epoch 0 is missing, and 2366427 is a portion of the accuracy with missing epoch number and a few preceding numbers
Has anyone encountered this before and knows how to deal with it?
EDIT:
I have noticed that after some epochs, the file on the GDrive is also 0 bytes and then on the next epoch it is filled back in

A guess: your training is proceeding in multiple threads/subprocesses, so your on_epoch_end is fighting itself - concurrently-executing copies of it are all opening the same file in 'a'ppend mode, and (partially) overwriting each other's outputs.
Does the problem go away if you write to a per-epoch file, or serialize output another way?

Related

PowerShell one counter behaves differently at some part of a script

I am running a PS script to create an excel file with some text and the corresponding pictures to the text. Till now everything works as expected but when i want to choose the right column for the pictures with a counter something weird happens.
Before the if statement the counter behaves like it should, but as soon i let it output the counter inside the if statement the counter does something weird what i cannot explain.
Part of the Script
pictures is an list of file names
rows_sorted is an list of camera names
foreach ($picture in $pictures) {
foreach ($row in $rows_sorted) {
$xlcounter += 1
Write $xlcounter -> counter looks fine
if ($picture -match $row ) {
$live_picture = "C:\Snapshots\pictures\$picture"
$image = [System.Drawing.Image]::FromFile($live_picture)
$sheet | Add-ExcelImage -Image $image -Row $xlcounter -Column 4 -ResizeCell
$image.Dispose()
Write $xlcounter -> Weird things happen
}
}
}
Example output of counter outside the if statement:
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Example output of counter inside the if statement:
2 205 408 611 814 1017 1220 1423 1626 1829 2032 2235 2438 2641 2844 3047 3251 3454 3657 3860 4063
I tried to rename the counter because the same name was used in a few lines before, but that did not help either.

how to use if in matlab for 2 matrix condition? [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 3 years ago.
Improve this question
i have 2 reflection coefficient equations R1 and R2 from K with condition absolute must below 1, i use if command for this situation .But when i plot the graph the absolute reflection coefficient still above 1. (K is matrix with 1 column and 201 row)
R1=K+sqrt(K.^2-1);
R2=K-sqrt(K.^2-1);
if abs(R1)<1
r=R1;
else
r=R2;
end
this is the K in excel
real imaginer
-0.7536 0.0512
-0.802 0.0426
-0.8496 0.0408
-0.8872 0.0327
-0.927 0.0338
-0.9575 0.0242
-0.979 0.0174
-0.9977 0.0113
-10,031 0.0029
-10,012 -0.007
-0.9876 -0.0167
-0.9654 -0.0249
-0.9299 -0.0401
-0.8797 -0.0488
-0.8176 -0.0623
-0.7297 -0.0782
-0.6458 -0.0865
-0.5351 -0.1051
-0.4098 -0.1197
-0.2701 -0.1349
-0.1177 -0.1489
0.0536 -0.1699
0.213 -0.1853
0.3933 -0.1921
0.5519 -0.1857
0.7128 -0.1896
0.8511 -0.1712
0.9468 -0.1452
10,222 -0.0943
10,375 -0.04
10,134 0.0365
0.9361 0.1255
0.8122 0.2168
0.6622 0.3108
0.4657 0.3774
0.2577 0.4497
0.0431 0.4775
-0.1463 0.5093
-0.3442 0.4999
-0.5203 0.4782
-0.6692 0.4417
-0.7781 0.3822
-0.8856 0.3293
-0.9703 0.2615
-10,187 0.193
-10,524 0.1254
-10,614 0.0557
-10,539 -0.0016
-10,297 -0.0698
-0.9879 -0.1212
-0.9355 -0.1829
-0.8721 -0.2298
-0.8011 -0.2783
-0.7232 -0.325
-0.6401 -0.3586
-0.5455 -0.4008
-0.4429 -0.43
-0.3524 -0.4433
-0.2455 -0.4769
-0.1336 -0.4863
-0.0391 -0.5073
0.0779 -0.5105
0.1776 -0.5196
0.2869 -0.5152
0.3893 -0.5084
0.4831 -0.4978
0.5888 -0.4907
0.6822 -0.4574
0.7614 -0.4381
0.8484 -0.4017
0.9098 -0.3585
0.9771 -0.3172
10,268 -0.2607
10,667 -0.2102
10,969 -0.1464
11,115 -0.0724
11,141 -0.0019
10,981 0.0838
10,645 0.1546
10,135 0.2457
0.9409 0.3332
0.8657 0.4061
0.7519 0.4973
0.6426 0.5635
0.5072 0.6302
0.3633 0.6782
0.2148 0.7161
0.0382 0.7573
-0.1051 0.7395
-0.273 0.7359
-0.4273 0.7154
-0.5653 0.6794
-0.6971 0.6279
-0.8202 0.555
-0.905 0.493
-0.9996 0.4155
-10,716 0.3239
-11,006 0.2549
-11,444 0.1479
-11,464 0.0722
-11,493 -0.0031
-11,282 -0.0814
-11,040 -0.1603
-10,645 -0.2219
-10,187 -0.2787
-0.9514 -0.3223
-0.8878 -0.3841
-0.8225 -0.42
-0.7415 -0.4606
-0.6607 -0.4889
-0.5577 -0.5319
-0.482 -0.5512
-0.3775 -0.5614
-0.2918 -0.5798
-0.1621 -0.5712
-0.0979 -0.5917
0.0149 -0.5559
0.1062 -0.5734
0.2142 -0.5648
0.3159 -0.5363
0.3844 -0.5302
0.5019 -0.5066
0.5805 -0.4709
0.6626 -0.4506
0.7482 -0.4117
0.8005 -0.363
0.8799 -0.3378
0.9349 -0.2889
0.9883 -0.2449
10,306 -0.1946
10,643 -0.1373
10,870 -0.1025
10,935 -0.0389
10,840 0.0184
10,732 0.0639
10,333 0.1274
0.9906 0.1739
0.9243 0.2293
0.8455 0.2752
0.7527 0.3035
0.6292 0.3394
0.5384 0.3524
0.3808 0.3845
0.2509 0.4067
0.0931 0.4004
-0.0423 0.3839
-0.2123 0.377
-0.3666 0.3537
-0.4838 0.3309
-0.6157 0.288
-0.7211 0.2604
-0.8322 0.2172
-0.8947 0.1791
-0.9618 0.1366
-10,024 0.0932
-10,299 0.0493
-10,415 0.0099
-10,333 -0.0243
-10,092 -0.0612
-0.9798 -0.0906
-0.9321 -0.1302
-0.8796 -0.1472
-0.8121 -0.17
-0.7414 -0.1886
-0.6649 -0.2019
-0.5907 -0.2149
-0.4793 -0.2271
-0.4011 -0.2224
-0.3121 -0.2408
-0.1948 -0.2343
-0.0997 -0.2322
0.008 -0.2328
0.1304 -0.2224
0.2662 -0.2213
0.4093 -0.2298
0.553 -0.2406
0.7094 -0.3018
0.8613 -0.383
0.9745 -0.5634
0.9796 -0.8226
0.7781 -0.9412
0.6424 -0.8495
0.6264 -0.8147
0.6071 -0.6706
0.6682 -0.6029
0.6759 -0.5596
0.71 -0.5218
0.7479 -0.4825
0.7691 -0.4476
0.8264 -0.4056
0.8412 -0.3912
0.8511 -0.3813
0.8689 -0.3425
0.899 -0.3375
0.8827 -0.3198
0.9024 -0.3164
0.929 -0.2876
0.9106 -0.2855
0.9695 -0.2079
10,342 -0.5353
0.8692 -0.5046
I am not 100% sure exactly what you are asking, but I believe the problem you are experiencing is that r is above 1?
K is an imaginary number, where the first column is the real part and the second column is the imaginary part, do I have that correctly? So the first K value is -0.7536+0.0512i, right?
Ok, so did you perhaps intend to cycle through each position of the R1 matrix and see if each one was less than 1. Because right now what you are doing is saying if any values in the entire R1 vector are less than 1, then r equals to the entire R2 vector.
If you want to go through each position in the vector, you should do this:
R1=K+sqrt(K.^2-1);
R2=K-sqrt(K.^2-1);
l=length(R1);
for p=1:l
if abs(R1(p))<1
r(p)=R1(p);
else
r(p)=R2(p);
end
end

Why does an argument $args after = sign not get expanded? ($args as parameter to jvm typical -D switch)

I have the following definition in my *profile.ps1 file:
if(Test-Path $env:M2_HOME){
function mvn{
$cmd = "$env:M2_HOME\bin\mvn.bat"
& $cmd $args
}
}
When I define a function using this function in powershell like:
function d { mvn help:describe $args }
using like:
d -Dplugin=jar
everything is fine as opposed to defining the latter as:
function d { mvn help:describe -Dplugin=$args }
using like:
d jar
Is there some builtin to handle this corner case?
It looks like you just need to make sure you're passing the arguments as strings and ensure they're evaluated first:
function mvn{
$cmd = "$env:M2_HOME\bin\mvn.bat"
& $cmd $args
}
function d { mvn "help:describe" "-Dplugin=$($args)" }
for get argument through call function you should use like this
function test {
write-host $args[0]
write-host $args[1]
}
test stackoverflow powershell
output
stackoverflow
powershell
stackoverflow is first argument passed to function and powershell is second argument passed to function
for every argument passed to function
function test {
foreach ($a in $args){
write-host "output:$args"
}
}
test 1 2 3 4 5 6 7 8
output:
test 1 2 3 4 5 6 7 8
output:1 2 3 4 5 6 7 8
output:1 2 3 4 5 6 7 8
output:1 2 3 4 5 6 7 8
output:1 2 3 4 5 6 7 8
output:1 2 3 4 5 6 7 8
output:1 2 3 4 5 6 7 8
output:1 2 3 4 5 6 7 8
output:1 2 3 4 5 6 7 8
for your function
if(Test-Path $env:M2_HOME){
function mvn{
$cmd = "$env:M2_HOME\bin\mvn.bat"
foreach ($arg in $args) {
& $cmd $args}
}
}
Someone seems to have made a similar observation:
The reason seems -D being a special character in powershell more or less marking where exactly each particluar special option string determined by -D ends, at least this modification to my calling function then works for me:
function x {mvn help:describe `-Dplugin=$args}
Sure it would be nice to handle such occasions in the hosting function somehow (mvn definition in my *profile.ps1), but that solutions rather seems out of scope of my question.

Split File into chunks keeping complete lines in solaris

How can I split a file into 3 with equal (or almost equal) number of lines without breaking a line.
for example split a file of 25 lines into 3 files of 9,8 and 8 lines each.
I know of split -n l/3 but does not work on Solaris10.
Tried some stuff i got online but did not give desired result like:
!/usr/bin/ksh
fspec=~/input.list
num_files=3
total_lines=$(wc -l <${fspec})
((lines_per_file = (total_lines + num_files - 1) / num_files))
split -l ${lines_per_file} ${fspec} files.
Here is a generic solution for you in awk
awk '{a[NR]=$0} END {t=int (NR/s);r=((NR/s-t)*s);while (n<s) for (i=t*n+++1;i<=t*n;i++) print a[i] > "file"n;while (i++<=NR) print a[i-1] > "file"n}' s=3 infile
This splits the infile to s numbers of file. If you set s=3 you get file1 file2 file3
The data that does not divide up, ends up in last file.
Example
cat number
1 one
2 two
3 three
4 four
5 five
6 six
7 seven
8 eight
9 nine
10 ten
awk '{a[NR]=$0} END {t=int (NR/s);r=((NR/s-t)*s);while (n<s) for (i=t*n+++1;i<=t*n;i++) print a[i] > "file"n;while (i++<=NR) print a[i-1] > "file"n}' s=3 number
cat file1
1 one
2 two
3 three
cat file2
4 four
5 five
6 six
cat file3
7 seven
8 eight
9 nine
10 ten

Merge columns of data from multiple text files by row from each seperate file using Powershell

I have output from a numerical modelling code. I needed to extract a specific value from a series of files. I used the following code to get it (I derived this from an example that would extract IP addresses from logfiles):
$input_path = ‘C:\_TEST\Input_PC\out5.txt’
$output_file = ‘C:\_TEST\Output_PC_All\out5.txt’
$regex = ‘\bHEAD(.+)\s+[\-]*\d{1,3}\.\d{6,6}\s?\b’
select-string -Path $input_path -Pattern $regex -AllMatches | % { $_.Matches } | % { $_.Value } > $output_file
So I now have got a number of text files which contain measurements (the number of files may be variable, currently there are 50) with one column of numeric data (with a number of rows which currently equals 7302 but which may vary depending on the length of the time series modelled) and which may be positive or negative as per the example data below.
Note a semicolon preceding the text indicates that what follows is a comment I am using to explain the order of the dataset and does not appear in the data to be processed...
out1.txt
-1.000000 ; 1st line of out1.txt
2.000000 ; 2nd line of out1.txt
-3.000000 ; 3rd line of out1.txt
...
5.000000 ; nth line of out1.txt
out2.txt
-1.200000 ; 1st line of out2.txt
-2.200000 ; 2nd line of out2.txt
3.200000 ; 3rd line of out2.txt
...
-5.20000 ; nth line of out2.txt
outn.txt
1.300000 ; 1st line of outn.txt
-2.300000 ; 2nd line of outn.txt
-3.300000 ; 3rd line of outn.txt
...
10.300000 ; nth line of outn.txt
I need to merge them into a single text file (for this example lets call it "Combined_Output.txt") using Powershell with the data ordered so that the first row of values from the differing output files appear first, then repeat this for row 2 and so on as below:
Combined_Output.txt
-1.000000 ; 1st line of out1.txt
-1.200000 ; 1st line of out2.txt
1.300000 ; 1st line of outn.txt
2.000000 ; 2nd line of out1.txt
-2.200000 ; 2nd line of out2.txt
-2.300000 ; 2nd line of outn.txt
-3.000000 ; 3rd line of out1.txt
3.200000 ; 3rd line of out2.txt
-3.300000 ; 3rd line of outn.txt
...
5.000000 ; nth line of out1.txt
-5.200000 ; nth line of out2.txt
10.300000 ; nth line of outN.txt
Just to say that I'm very new to this sort of thing so I hope that the explanation above makes sense and also to say any help that you can provide would be much appreciated.
EDIT
Having now run the models, when using this code for the large data files created, there seems to be an issue of sorting of the imported data. This seems to occur primarily when there are repeated values for example the second row of data from each outfile has been combined in the following order by the script. It looks like there is some sorting based on the value of the data and not just based on the out file name:
Value ; out file text number
-1.215809 ; 1
-0.480543 ; 18
-0.480541 ; 19
-0.48054 ; 2
-0.480539 ; 20
-0.480538 ; 21
-0.480537 ; 22
-0.480536 ; 23
-0.480535 ; 24
-0.480534 ; 25
-0.480534 ; 26
-0.480688 ; 10
-0.480533 ; 27
-0.480532 ; 3
-0.480776 ; 4
-0.48051 ; 5
-0.48051 ; 6
-0.48051 ; 7
-0.48051 ; 8
-0.48051 ; 9
-0.48051 ; 11
-0.48051 ; 12
-0.48051 ; 13
I feel like I might have over complicated this answer but lets see how we do. Consider the following dummy data similar to your samples
Out1.txt Out2.txt Out3.txt
-0.40000 0.800000 4.100000
3.500000 0.300000 -0.90000
-2.60000 0.800000 2.200000
0.500000 1.800000 -1.40000
3.600000 1.800000 1.400000
40000000 -0.70000 1.500000
The file contents are arranged side by side for answer brevity and to help understand the output. The code is as follows:
$allTheFiles = #()
Get-ChildItem c:\temp\out*.txt | ForEach-Object{
$allTheFiles += ,(Get-Content $_.FullName)
}
For ($lineIndex=0; $lineIndex -lt $allTheFiles[0].Count; $lineIndex++){
For($fileIndex=0; $fileIndex -lt $allTheFiles.Count; $fileIndex++){
$allTheFiles[$fileIndex][$lineIndex]
}
} | Out-File -FilePath c:\temp\file.txt -Encoding ascii
Gather all out*.txt files the code creates an array of arrays which are the file contents themselves. Using nested For loops cycle though each single file outputting one line from each file at a time. While I am having a hard time being clear on what is happening if you compare the sample data to the output you should see that the first line or every file is outputted together followed by the next line...etc.
This code will produce the following output
-0.40000
0.800000
4.100000
3.500000
0.300000
-0.90000
-2.60000
0.800000
2.200000
0.500000
1.800000
-1.40000
3.600000
1.800000
1.400000
40000000
-0.70000
1.500000
Caveats
The code assumes that all files are of the same size. The number of lines is determined by the first file. If other files contain more data it would be lost in this model.