I'm writing a CL library to read MS Excel(tm) spreadsheets called "xlmanip" (not ready for prime time yet -- only reads "xlsx" spreadsheets, works for the 80% use case of "I want to operate on cell contents"... but I digress.)
One thing that worries me when reading "xlsx" (ZIP archives in XML format) is that the current ZIP handling library, Common Lisp ZIP, unpacks the compressed contents as a (vector (unsigned-byte 8)). For a large spreadsheet, that'll cause an issue for the end user.
One alternative I've thought about is delayed loading -- let-over-lambda a closure that effectively demand-loads the worksheet when needed. However, that's just delaying the inevitable.
Are there any ZIP file CL libraries out there that return a Gray stream to a ZIP component's contents as opposed to a (potentially large) (vector (unsigned-byte 8))?
Edit: Clarification
I'm looking for a ZIP component function that returns a stream, not one that takes a stream. The functions that take a stream write the ZIP component's contents directly to the file associated with the stream. I'd rather that xlmanip reads from a stream directly as if the ZIP component were (implicitly, virtually) a file.
Chipz can decompress a ZIP to a stream. It provides a decompress functions where you give it an output stream and a input stream to decompress and it returns the output stream where the decompressed contents can be read.
Related
Streaming execute, -11!, does not work on named pipes, so an obvious solution of redirecting the gzip -cd output to a named pipe and passing it to -11! does not work.
-11! accepts a compressed file and streams it so long as it was compressed with -19! (using 2 as the compression algorithm parameter, which is gzip).
The only difference between a normal gzipped file and a kdb compressed one is a few bytes at the beginning of the file.
EDIT (see comment) Thanks, this isn't true - the bytes are different at the end of the file
So a possible solution is to prepend your gzipped files (if they weren't produced by -19!) with the appropriate byte array first.
For anyone using kdb v3.4+ streaming execution for named pipes was introduced as the function .Q.fps.
Here is a simple example of .Q.fps in action, first create the pipe via command line:
echo "aa" > test.pipe
Next in a q session:
q).Q.fps[0N!]`:test.pipe
,"aa"
Where 0N! is a function used to display the contents of the file.
I'm relatively new to C# and am attempting to adapt a text encryption algorithm I designed in wxMaxima into a Binary encryption program in C# using Visual Studio forms. Because I am new to reading/writing binary files, I am lacking in knowledge regarding what happens when I try to read or write to a filestream.
For example, instead of encrypting a text file as I've done in the past, say I want to encrypt an executable or any other form of binary file.
Here are a few questions I don't understand:
When I open a file stream and use binaryreader will it read in an absolute duplicate of absolutely everything in the file? I want to be able to, for example, read in an entire file, delete the original file, then create a new file with the old name and write the entire binary stream back. Will this reproduce the original file exactly or will there be some sort of corruption that must otherwise be accounted for?
Because it's an encryption program, I was hoping to add in a feature that would low-level "format" the original file before deleting it so it would be theoretically inaccessible by combing the physical data of a harddisk. If I use binarywriter to overwrite parts of the original file with gibberish will it be put on the same spot on the harddisk or will the file become fragmented and actually just redirect via the FAT to some other portion of the harddisk? Obviously there's no point in overwriting the original file with gibberish if it's not over-writing the original cluster on the harddisk.
For your first question: A BinaryReader is not what you want. The name is a bit misleading: it "Reads primitive data types as binary values in a specific encoding." You probably want a FileStream.
Regarding the second question: That will not be easy: please see the "How SDelete Works" section of SDelete for an explanation. Brief extract in case that link breaks in the future:
"Securely deleting a file that has no special attributes is relatively straight-forward: the secure delete program simply overwrites the file with the secure delete pattern. What is more tricky is securely deleting Windows NT/2K compressed, encrypted and sparse files, and securely cleansing disk free spaces.
Compressed, encrypted and sparse are managed by NTFS in 16-cluster blocks. If a program writes to an existing portion of such a file NTFS allocates new space on the disk to store the new data and after the new data has been written, deallocates the clusters previously occupied by the file."
i have found an instruction to send a file (xml in my case) to my device: Socket.SendFile(myXml) My problem is sendFile() does not exist in compact framework.
So how could i send a file with compact framework on the same way?
You would use a combination of Socket.Send() and Filestream.Read() for that.
Just read N bytes from the file and then write N bytes to your socket until you've covered the entire file. The second link has an example for reading files. Replace the section labeled "// Write the byte array to the other FileStream." with your Socket.Send() calls.
I would like to play some kind of text-to-speech with only numbers. I can record 10 wav files, but how can I combine them programmatically ?
For instance, the user types 1234, and the text-to-speech combines 1.wav with 2.wav, 3.wav and 4.wav to produce 1234.wav that plays "one two three four".
1) create a new destination sample buffer (you will want to know the sizes).
2) read the samples (e.g. using AudioFile and ExtAudioFile APIs) and write them in sequence to the buffer. You may want to add silence between the files.
It will help if your files are all the same bit depth (the destination bit depth - 16 should be fine) and sample rate.
Alternatively, if you have fixed, known, sample rates and bit depths for all files, you could just save them as raw sample data and be done in much less time because you could simply append the data as is without writing all the extra audio file reading programs.
The open source project wavtools provides a good reference for this sort of work, if you're ok with perl. Otherwise there is a similar question with some java examples.
The simplist common .wav (RIFF) file format just has a 44 byte header in front of raw PCM samples. So, for these simple types of .wav files, you could just try reading the files as raw bytes, removing the 44 byte header from all but the first file, and concatening the samples. Or just play the concatenated samples directly using the Audio Queue API.
I have a situation where I can save a post-processing pass through the audio by taking some manipulated buffer from the end of the track and writing them to the beginning of my output file.
I originally thought I could do this by resetting the write pointer using ExtAudioFileSeek, and was about to implement it when I saw this line in the docs
Ensure that the file you are seeking in is open for reading only. This function’s behavior with files open for writing is undefined.
Now I know I could close the file for writing then reopen it, but the process is a little more complicated than that. Part of the manipulation I am doing is reading from buffers that are in the file I am writing to. The overall process looks like this:
Read buffers from the end of the read file
Read buffers from the beginning of the write file
Process the buffers
Write the buffers back to the beginning of the write file, overwriting the buffers I read in step 2
Logically, this can be done in 1 pass no problem. Programmatically, how can I achieve the same thing without corrupting my data, becoming less-efficient (opposite of my goal) or potentially imploding the universe?
Yes, using a single audio file for both reading and writing may, as you put it, implode the universe, or at least lead to other nastiness. I think that the key to solving this problem is in step 4, where you should write the output to a new file instead of trying to "recycle" the initial write file. After your processing is complete, you can simply scrap the intermediate write file.
Or have I misunderstood the problem?
Oh, and also, you should use ExtAudioFileWriteAsync instead of ExtAudioFileWrite for your writes if you are doing this in realtime. Otherwise the I/O load will cause audio dropouts.