Is there a maximum amount of packets for pcap file (beyond OS limits?
Looking at the specification I don't see anything in the pcap file format that would limit the number of packets.
Perhaps will libpcap or tcpdump limit the number of packets written?
No, there is nothing in the pcap file format that limits the number of packets in a file. (The pcap-ng file format allows a maximum of 18446744073709551614 bytes in a section, if the Section Header Block must contain the number of bytes in a section, but, with more than 18446744073709551615 bytes in a file, even current 64-bit operating systems will have a problem with the file. :-))
Tcpdump just uses libpcap to write out files, so it would be limited only by libpcap's limitations.
libpcap 1.0 and later uses the AC_SYS_LARGEFILE macro in the configure script, so, on UN*X, if the underlying operating system supports files larger than 2GB (which they should all do for 64-bit machines, and which most if not all of them have done for several years on 32-bit machines), and if the configure script properly sets things up on 32-bit machines to allow large files (or if no such setup is necessary - it's not necessary on *BSD and OS X, for example), and if whoever built the library didn't somehow prevent that from working, libpcap should have no problem writing files larger than 2GB (as long as you have enough disk space, of course...).
I don't know whether the C support library that comes with whatever compiler was used to build WinPcap will support large files on Windows NT (NT 3.x, NT 4.0, 2000, XP, Vista, 7, 8, and various Server versions), but I suspect it would. I don't think Windows OT (95, 98, Me) supports large files.
Related
This question already has answers here:
Creating a bootable ISO image with custom bootloader
(2 answers)
Closed 2 years ago.
I have finished writing my simple Operating System and I want to test it on a real hardware(PC), not bochs or qemu. My OS has a custom bootloader and a kernel and I used cat to concat them into one single bin file. But I spent hours on finding out a way to convert the bin file to a bootable iso file but failed each time. According to OSDev.org, I think I need to use genisoimage(mkisofs) to do the convert, but I don't know exactly how this command works, I finally outputted a iso file but this one is not working.(I think I used the wrong command, can someone explain a little bit more to me?)
Other Approaches I tried:
Directly burn the bin file to a CD. Error: Missing Operating System.
Convert the bin file to ISO using winbin2iso and other windows platform software. Error: Could not boot. Not even in qemu.
Also, what is El-Torito?
When a CD is being booted, the firmware checks the CD for a "boot catalogue" in the CD's meta-data. This is a list of entries with (up to) one entry for each kind of computer; so that it's possible to create a single CD that works for 80x86 BIOS and 80x86 UEFI (and PowerPC and Sparc and ...).
For 80x86 there are 4 different types of entries:
UEFI. The rest of the entry tells the firmware the starting sector and size of a FAT file system image. UEFI figures out which file it wants from the FAT file system based on converting the architecture into a file name (e.g. for 80x86 it'll probably want the file \EFI\BOOT\BOOTX64.EFI).
"No emulation, 80x86 BIOS". The rest of the entry tells the firmware the starting sector and size of a boot loader. The boot loader can be any size you like (up to about 639 KiB); and the "device number" the BIOS tells you (in dl) will be for the CD drive itself, so you can load more stuff from the same disk using it.
"Hard disk emulation, 80x86 BIOS". The rest of the entry tells the firmware the starting sector and size of a disk image. The disk image should have an MBR with a BPB and partition/s, with an active partition pointing to where the operating system's boot loader is at the start of its partition. In this case the BIOS will create a fake "device 0x80" (from info in the BPB in the MBR) and mess up the device numbers for any real hard drives (e.g. the first hard drive which would've been "device 0x80" will become "device 0x81" instead, etc). Note that this is inefficient because sectors on a CD are 2048 bytes but the BIOS will emulate 512 byte sectors, so every time you try to read a 512 byte sector the BIOS will actually read 2048 bytes and throw the "wrong" 75% of the data away. It's also very annoying (trying to squeeze anything good in 512 bytes is impossible). It's mostly only used for obsolete junk (e.g. MS-DOS).
"Floppy disk emulation, 80x86 BIOS". The rest of the entry tells the firmware the starting sector and size of a disk image. The disk image should have a boot loader in the first sector with a BPB. In this case the BIOS will create a fake "device 0x00" (from info in the BPB in the MBR) and mess up the device numbers for any real floppy drives. Just like hard disk emulation, this is inefficient, even more "very annoying" (because it also limits the OS to the size of a floppy which is nowhere near enough space), and only intended for obsolete junk.
The best way to deal with CDs is to write 2 new boot loaders (one for "no emulation, 80x86 BIOS" and another for UEFI); then use any tool you like to create the appropriate structures on the CD (e.g. genisoimage with the -no-emul-boot option for your "no emulation, 80x86 BIOS" boot loader plus some other option for UEFI that doesn't seem to exist!?).
Note that it's easy to write your own utility that is "more clever". You'd want to create a FAT file system (for UEFI) and an ISO9660 file system (for the rest of the operating system's file - help/docs, drivers, etc), but most tools won't create the FAT file system for you, and it's possible for files in both file systems (FAT and ISO9660) to use the same sectors (so that the same files appear in both file systems without costing twice as much disk space). Something like this would probably only take you 1 week to write yourself (and you'll learn a lot about CDs and ISO9660 that you're going to have to learn eventually anyway). The relevant documentation (for booting from CD, ISO9660 file systems, FAT file systems, and UEFI) are all easily obtained online.
Also, what is El-Torito?
El-Torito is another name for the "Bootable CD-ROM Specification" that describes the structures needed on a CD to make it bootable.
Strawberry Perl lists SHA1 digests on its download page.
However, looking at download page snapshots on archive.org, their SHA1 digests for the same perl version and build seem to change over time.
Example: in the download page snapshot from 2013-05-10, strawberry-perl-5.16.3.1-32bit-portable.zip is shown to be 86.8 MB long with an SHA1 digest of 3b9c4c32bf29e141329c3be417d9c425a7f6c2ff.
In the download page snapshot from 2017-02-14, the same strawberry-perl-5.16.3.1-32bit-portable.zip is shown to be 87.3 MB long with an SHA1 digest of 7f6da2c3e1b7a808f27969976777f47a7a7c6544.
And on the current download page, the same strawberry-perl-5.16.3.1-32bit-portable.zip is shown to be 91.0 MB long with an SHA1 digest of 00ba29e351e2f74a7dbceaad5d9bc20159dd7003
I thought they might have recompiled the package for some reason, but the current strawberry-perl-5.10.0.6-portable.zip has only one file dated later than 2009 (it's portable.perl), so this doesn't explain why the archive grew over time. Sadly, I don't have older zip files, so I have no way of knowing what changed inside the archive.
What's going on here? Why do past builds change over time?? I am kind of concerned that some hackers might be injecting malicious code or something into binary perl packages...
Is there a rational explanation here ? Thanks...
An hash such as a SHA1 digest is good to defend against communication errors, that is to ensure integrity of what you downloaded (basically proving: "file on your hard disk" = "file on the webserver"), but just by itself does not help to ensure authentication.
For that, files should be signed with some PGP signature, or using X.509 certificates. This is the only way you could verify that the file was indeed produced by the true intended authors.
So just by itself your observation neither signal an attack nor helps you defend against one in fact.
Like #ikegami said, you can even configure compressors with a different RAM/time ratio and the same on will produce different results.
See for example in Unix zip:
-Z cm
--compression-method cm
Set the default compression method. Currently the main methods supported by zip are store and deflate. Compression method can be set to:
store - Setting the compression method to store forces zip to store entries with no compression. This is generally faster than compressing entries, but results in no space savings. This is the same as using -0 (compression level zero).
deflate - This is the default method for zip. If zip determines that storing is better than deflation, the entry will be stored instead.
bzip2 - If bzip2 support is compiled in, this compression method also becomes available. Only some modern unzips currently support the bzip2 compression method, so test the unzip you will be using before relying on archives using this method (compression method 12).
and
-#
(-0, -1, -2, -3, -4, -5, -6, -7, -8, -9)
Regulate the speed of compression using the specified digit #, where -0 indicates no compression (store all files), -1 indicates the fastest compression speed (less compression) and -9 indicates the slowest compression speed (optimal compression, ignores the suffix list). The default compression level is -6.
Though still being worked, the intention is this setting will control compression speed for all compression methods. Currently only deflation is controlled.
The same source code could have been recompressed over time when the website is regenerated. So same content but different archive results.
I have downloaded all three files. In fact they are the same sizes exactly (or the WayBack machine did not store them correctly, it does not do a redirection), and all in fact with a SHA1 hash of 00ba29e351e2f74a7dbceaad5d9bc20159dd7003.
Your best bet is probably to ask StrawberryPerl organization directly.
I have a data file (6.3GB) that I'm attempting to work on in MATLAB, but I'm unable to get it to load, and I think it may be a memory issue. I've tried loading in a smaller "sample" file (39MB) and that seems to work, but my actual file won't load at all. Here's my code:
filename = 'C://Users/Andrew/Documents/filename.mat';
load(filename);
??? Error using ==> load
Can't read file C://Users/Andrew/Documents/filename.mat.
exist(filename);
EDU>> ans = 2
Well, at least the file exists. When I check the memory...
memory
Maximum possible array: 2046 MB (2.146e+009 bytes) *
Memory available for all arrays: 3442 MB (3.609e+009 bytes) **
Memory used by MATLAB: 296 MB (3.103e+008 bytes)
Physical Memory (RAM): 8175 MB (8.572e+009 bytes)
* Limited by contiguous virtual address space available.
** Limited by virtual address space available.
So since I have enough RAM, do I need to increase the maximum possible array size? If so, how can I do that without adding more RAM?
System specifics: I'm running 64-bit Windows, 8GB of RAM, MATLAB Version 7.10.0.499 (R2010a). I think I can't update to a newer version since I'm on a student license.
As the size might be the issue, you could try load('fileName.mat', 'var1'); load('fileName.mat', 'var2'); etc. For this, you'll have to know the variable names though.
An option would be to use the matfile object to load/index directly into the file instead of loading into ram.
doc matfile
But one limitation is that you can not index directly into a struct. So you would need to find a friend to convert the struct in your mat file and save it with the version option
save(filename, variables, '-v7.3')
May be you can load part by part your data to do your stuff using load part of variables from mat file. You must have matlab 7.3 or newer.
From your file path I can see you are using Windows. Matlab is only 32 bit for Windows and Linux (there is no 64 bit for these OSes at least for older releases, please see my edit), which means you are limited to <4GB ram total for a single application (no matter how much you have in your system), this is a 32 bit application issue so there is nothing you can do to remedy it. Interestingly the Mac version is 64 bit and you can use as much ram as you want (in my computer vision class we often used my mac to do our big video projects because windows machines would just say "out of memory")
As you can see from your memory output you can only have ~3.4GB total for matrix storage, this is far less than the 6.3GB file. You'll also notice, you can only use ~2GB for one particular matrix (that number changes as you use more memory).
Typically when working with large files you can read the file line by line, rather than loading the entire file into memory. But since this is a .mat file that likely wouldn't work. If the file contains multiple variables maybe separate them each into their own individual files that are small enough to load
The take home message here is you can't read the entire file at once unless you hop onto a Mac with enough RAM. Even then the size for a single matrix is still likely less than 6.3GB
EDIT
Current Matlab student versions can be purchased in 64 bit for all OSes as of 2014 see here so a newer release of Matlab might allow you to read the entire file at once. I should also add there has been a 64 bit version before 2014, but not for the student license
I have a Redhat 5.5 Linux machine using Perl Net::SSH2 to sftp a file. It appears that once the filehandle is opened for writing, I can only write a maximum buffer length of 32500 bytes for each call to $fh->write($buffer). I've had to limit my file read to 30*1024 bytes before writing it to the Net::SSH2::File filehandle. Shouldn't I be able to write a larger buffer? The filesize I am copying is about 800k. The buffer I am talking about is the size of the write chunk. I can continually write this chunk size.
It's apparently a hard-coded limit in Net::SSH2. See here:
http://www.libssh2.org/mail/libssh2-devel-archive-2010-06/0104.shtml
Most SSH servers limit the size of the packets they can handle to some size around 32Kb and so clients do the same for interoperability.
You can use Net::SFTP::Foreign, that doesn't have that limitation (its buffers will grow to accommodate data of any length as far as memory is available) and it can run on top of Net::SSH2 via the Net_SSH2 backend Net::SFTP::Foreign::Backend::Net_SSH2 or just use the OpenSSH binary client.
Usually both files are availble for running some diff tool but I need to find the differences in 2 binary files when one of them resides in the server and another is in the mobile device. Then only the different parts can be sent to the server and file updated.
There is the bsdiff tool. Debian has a bsdiff package, too, and there are high-level programming language interfaces like python-bsdiff.
I think that a jailbreaked iPhone, Android or similar mobile device can run bsdiff, but maybe you have to compile the software yourself.
But note! If you use the binary diff only to decide which part of the file to update, better use rsync. rsync has a built-in binary diff algorithm.
You're probably using the name generically, because diff expects its arguments to be text files.
If given binary files, it can only say they're different, not what the differences are.
But you need to update only the modified parts of binary files.
This is how the Open Source program called Rsync works, but I'm not aware of any version running on mobile devices.
To find the differences, you must compare. If you cannot compare, you cannot compute the minimal differences.
What kind of changes do you do to the local file?
Inserts?
Deletions?
Updates?
If only updates, ie. the size and location of unchanged data is constant, then a block-type checksum solution might work, where you split the file up into blocks, compute the checksum of each, and compare with a list of previous checksums. Then you only have to send the modified blocks.
Also, if possible, you could store two versions of the file locally, the old and modified.
Sounds like a job for rsync. See also librsync and pyrsync.
Cool thing about the rsync algorithm is that you don't need both files to be accessible on the same machine.