I have an issue regarding saving animations in matplotlib. In the middle of the process, this error emerges:
anim.save(adress, writer=writervideo)
File "C:\\Users\\Shayan\\anaconda3\\lib\\site-packages\\matplotlib\\animation.py", line 1085, in save
writer.grab_frame(\*\*savefig_kwargs)
File "C:\\Users\\Shayan\\anaconda3\\lib\\site-packages\\matplotlib\\animation.py", line 357, in grab_frame
self.fig.savefig(self.\_proc.stdin, format=self.frame_format,
File "C:\\Users\\Shayan\\anaconda3\\lib\\site-packages\\matplotlib\\figure.py", line 3274, in savefig
self.canvas.print_figure(fname, \*\*kwargs)
File "C:\\Users\\Shayan\\anaconda3\\lib\\site-packages\\matplotlib\\backends\\backend_qtagg.py", line 81, in print_figure
super().print_figure(\*args, \*\*kwargs)
File "C:\\Users\\Shayan\\anaconda3\\lib\\site-packages\\matplotlib\\backend_bases.py", line 2338, in print_figure
result = print_method(
File "C:\\Users\\Shayan\\anaconda3\\lib\\site-packages\\matplotlib\\backend_bases.py", line 2204, in \<lambda\>
print_method = functools.wraps(meth)(lambda \*args, \*\*kwargs: meth(
File "C:\\Users\\Shayan\\anaconda3\\lib\\site-packages\\matplotlib_api\\deprecation.py", line 410, in wrapper
return func(\*inner_args, \*\*inner_kwargs)
File "C:\\Users\\Shayan\\anaconda3\\lib\\site-packages\\matplotlib\\backends\\backend_agg.py", line 454, in print_raw
fh.write(renderer.buffer_rgba())
OSError: \[Errno 22\] Invalid argument
The code work in a loop and this happens in the middle of processing and each time in a different step. The last lines of my code are:
anim = animation.FuncAnimation(fig=fig, func=update, interval=interval, frames=frame_range)
writervideo = animation.FFMpegWriter(fps=60)
adress = f"Animations/{record_id}.mp4"
anim.save(adress, writer=writervideo)
plt.close()
I have tried changing the fps but the issue still exists. I have also checked my code for any argument issues but everything is ok (at list seems ok). When I reduce the number of frames the code works and I thought maybe it is a buffer issue; but I am not sure.
Any help would be highly appreciated.
Regards.
Related
I want to manage a subprocess with the subprocess module, and I need to pipe a (really) large numbers of lines to the child stdin. I'm creating the input with a generator, and passing onto the subprocess like this:
def my_gen (end): # simplified example
for i in range(0, end):
yield f"line {i}"
with subprocess.Popen(["command", "-o", "option_value"], # simplified example
stdin = subprocess.PIPE, stdout = sys.stdout, stderr = sys.stderr) as process:
for line in my_gen(1e7):
process.stdin.write(line.encode()) # This is apparently not safe
out, err = process.communicate() # out and err will be None,
# but this closes the process gracefully, which "with" does too
This results in a Broken Pipe Error, although it does't happen all the time on every machine I've tried:
Traceback (most recent call last):
File "my_script", line 170, in <module>
process.stdin.write(line.encode())
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "path/tolib/python3.8/subprocess.py", line 171, in <module>
File "path/tolib/python3.8/subprocess.py", line 914, in __exit__
self.stdin.close()
BrokenPipeError: [Errno 32] Broken pipe
So, what's the safe way to pass input line by line from a generator to a subprocess?
Edit: I've been getting suggestions about using communicate, which is of course in the docs. That answers how to communicate safely, but it doesn't accept a generator as input.
Edit2: as Booboo pointed out, the example will throw a runtime error (not the one I was finding in my code), the call to range should be range(0, int(end)) so my_gen can accept numbers in 1e7 notation.
First of all, if you want stdout and stderr to not be piped, then either do not specify these arguments to the Popen call at all or specify their values as None, the default value if not specified (but do not specify these as sys.stdout and sys.stderr).
Why not? Looking at the source for the Popen.communicate method I can see that there is special optimized code for the case where there is only one non-None argument and when that argument is the sysin argument then Popen.communicate is implemented by simply doing a write of the past input string to the pipe and ignores any BrokenPipeError error that might occur. But by passing the stdout and stderr arguments as you are, I suspect that communicate is confused and is now starting threads to handle the processing and this is ultimately intermittently leading to your exception.
Now I believe that you can execute your writes without using communicate and also ignore the BrokenPipeError. When I tried the following code (substituting my own command being executed by Popen that writes what is being piped in to a file and using text mode), I, in fact, did not encounter any BrokenPipeError exceptions (nor do I expect to with the proper setting of stdout and stderr). So I can't swear to whether the output will still be correct if such an exception should occur.
As an aside, the range built-in function does not take a float object (at least not for me), so I don't know how you are able to specify 1e7.
I have also modified the code to add terminating newline characters at the end of each line and to process in text mode, but you should not feel constrained to do so.
import subprocess
import sys
def my_gen (end): # simplified example
for i in range(0, end):
yield f"line {i}\n"
with subprocess.Popen(["command", "-o", "option_value"], stdin=subprocess.PIPE, text=True) as process: # simplified example
for line in my_gen(10_000_000):
try:
process.stdin.write(line)
except BrokenPipeError as e:
pass
out, err = process.communicate()
Docs say to use .communicate:
Warning: Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate
PS C:\OIDv4_ToolKit> python convert_annotations.py
Currently in subdirectory: train
Converting annotations for class: Vehicle registration plate
0%| | 0/400 [00:00<?, ?it/s]0317.44 497.91974400000004 413.44 526.08
0%| | 0/400 [00:00<?, ?it/s]
Traceback (most recent call last):
File "C:\OIDv4_ToolKit\convert_annotations.py", line 66, in <module>
coords = np.asarray([float(labels[1]), float(labels[2]), float(labels[3]), float(labels[4])])
IndexError: list index out of range
python file: this is the error it refers to as line 66 (Line 7 here)
with open(filename) as f:
for line in f:
for class_type in classes:
line = line.replace(class_type, str(classes.get(class_type)))
print(line)
labels = line.split()
coords = np.asarray([float(labels[1]), float(labels[2]), float(labels[3]), float(labels[4])])
coords = convert(filename_str, coords)
This doesn't look like a PowerShell issue; the python interpreter looks like it is being run correctly. I suggest adding the python tag to your question to get the right people involved.
Having located the source, it seems as if some of the text files in the following directory aren't in the format expected by convert_annotations.py:
C:\OIDv4_ToolKit\OID\Dataset\train\Vehicle registration plate\Label\
You can verify this with:
print("labels length =", len(labels))
after the line.split() method. If you get a length of 1, it is likely the items on a line somewhere aren't separated with whitespace, for example with commas. You can also inspect the files manually to determine the format. To find them, you can use:
print(os.path.join(os.getcwd(), filename))
inside the the for loop, which is on Line 54 in the source I linked above. Note also that the string split() method supports a custom separator as the first argument, should the files be in a different format.
This issue occurs when you don't put the class name in classes.txt
The class name should be same in classes.txt as downloaded class.
I have a file foo that I'd like to copy into a table.
\copy stage.from_csv FROM '/path/foo.dat' CSV;
If foo has an error like column mismatch or bad type, the return error is normal:
CONTEXT: COPY from_csv, line 5, column report_year: "aa"
However, if the error is caused by an extraneous quotation mark, the reported line number is always one greater than the size of the file.
CONTEXT: COPY from_csv, line 11: "02,2004,"05","123","09228","00","SUSX","PR",30,,..."
The source file has 10 lines, and I placed the error in line 5. If you examine the information in the CONTEXT message, it contains the line 5 data, so I know postgres can identify the row itself. However, it cannot identify the row by number. I have done this with a few different file lengths and the returned line number behavior is consistent.
Anyone know the cause and/or how to get around this?
That is because the error manifests only at the end of the file.
Look at this CSV file:
"google","growing",1,2
"google","growing",2,3
"google","falling","3,4
"google","growing",4,5
"yahoo","growing",1,2
There is a bug in the third line, an extra " has been added before the 3.
Now to parse a CSV file, you first read until you hit the next line break.
But be careful, line breaks that are within double quotes don't count!
So all of the line breaks are part of a quoted string, and the line continues until the end of file.
Now that we have read our line, we can continue parsing it and notice that the number of quotes is unbalanced. Hence the error message:
ERROR: unterminated CSV quoted field
CONTEXT: COPY stock, line 6: ""google","falling","3,4
"google","growing",4,5
"yahoo","growing",1,2
"
In a nutshell: the error does not occur until we have reached the end of file.
I want to read and write the same file simultaneously. Here is a simplified code:
clc;
close all;
clearvars;
fd = fopen ('abcd.txt','r+'); %opening file abcd.txt given below
while ~feof(fd)
nline = fgetl(fd);
find1 = strfind(nline,'abcd'); %searching for matching string
chk1 = isempty(find1);
if(chk1==0)
write = '0000'; %in this case, matching pattern found
% then replace that line by 0000
fprintf(fd,'%s \n',write);
else
continue;
end
end
File abcd.txt
abcde
abcd23
abcd2
abcd355
abcd65
I want to find text abcd in string of each line and replace the entire line by 0000. However, there is no change in the text file abcd.txt. The program doesn't write anything in the text file.
Someone can say read each line and write a separate text file line by line. However, there is a problem in this approach. In the original problem, instead of finding matching text `abcd, there is array of string with thousands of elements. In that case, I want to read the file, parse the file for find matching string, replace string as per condition, go to next iteration to search next matching string and so on. So in this approach, line by line reading original file and simultaneously writing another file does not work.
Another approach can be reading the entire file in memory, replacing the string and iterate. But I am not very sure how will it work. Another issue is memory usage.
Any comments?
What you are attempting to do is not possible in a efficient way. Replacing abcde with 0000, which should be done for the first line, would require all remaining text forward because you remove one char.
Instead solve this reading one file and write to a second, then remove the original file and rename the new one.
I'm trying to solve the bilateral problem on Spotify's Tech Puzzles. http://www.spotify.com/us/jobs/tech/bilateral-projects/ I have something that is working on my computer that reads input from a file input.txt, and it outputs to ouput.txt. My problem is that I cannot figure out how to make my code work when I submit it where it must read from stdin. I have looked at several other posts and I don't see anything that makes sense to me. I see some people just use raw_input - but this produces a user prompt?? Not sure what to do. Here is the protion of my code that is suposed to read the input, and write the output. Any suggestions on how this might need changed? Also how would I test the code once it is changed to read from stdin? How can I put test data in stdin? The error i get back from spotify says Run Time Error - NameError.
import sys
# Read input
Input = []
for line in sys.stdin.readlines():
if len(line) <9:
teamCount = int(line)
if len(line) > 8:
subList = []
a = line[0:4]
b = line[5:9]
subList.append(a)
subList.append(b)
Input.append(subList)
##### algorithm here
#write output
print listLength
for empWin in win:
print empWin
You are actually doing ok.
for line in sys.stdin.readlines():
will read lines from stdin. It can however be shortened to:
for line in sys.stdin:
I don't use Windows, but to test your solution from a command line, you should run it like this:
python bilateral.py < input.txt > output.txt
If I run your code above like that, I see the error message
Traceback (most recent call last):
File "bilateral.py", line 20, in <module>
print listLength
NameError: name 'listLength' is not defined
which by accident (because I guess you didn't send in that) was the error the Spotify puzzle checker discovered. You have probably just misspelled a variable somewhere.