I need to copy a file that will be modified later on an iOS device. For performance reasons, it would be great if this would work copy-on-write, i.e. the file is not really duplicated, and only modified blocks of the copy are written later.
As pointed out in the comments, this probably has to be supported by the file system (HFS+?). A link is not sufficient, since both the old (A) and new (B) file name will point to the same file, and if I modify A, B will also change.
A "lazy" copy also would not help, since on first write the whole file would still need to be copied.
I was thinking more about a solution like the one described by #Hot Licks that would start with A and B using the same blocks on disk, and when I write to file B, only the modified blocks would be stored on disk, while identical parts in A and B go on using the same blocks on disk.
Is this possible on iOS?
Regards,
Jochen
There's no built-in mechanism for doing efficient partial copies of files, but if you're copying a file and making internal changes to the content, then the most efficient mechanism to use is mmap. You map the file into memory and modify it in-place. The changes are written back to the file automatically without needing to rewrite the file in pieces.
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/mman.h>
struct stat astat;
int fd = open("filename", O_RDWR);
if ((fd != -1) && (fstat(fd, &astat) != -1)) {
char *data = mmap(0, astat.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if (data != MAP_FAILED) {
self.data_ptr = data;
self.data_size = astat.st_size;
}
close(fd);
}
When you're done with the file, you use munmap to release the mapping back to the os:
munmap(self.data_ptr, self.data_size);
The usual caveats apply from modifying a shared resource.
Related
I'm trying to figure out how to use AFL, but I can't seem to make a simple example running.
Here is my C program:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <assert.h>
char *remove_white_space(char *s)
{
while (s && *s++)
if (*s == ' ')
return "moish";
return s;
}
int main(int argc, char **argv)
{
char buffer[256]={0};
FILE *fl = fopen(argv[1],"rt");
if (fl == NULL) return 0;
assert(fscanf(fl,"%s",buffer) > 0);
char *res = remove_white_space(buffer);
if (strcmp(res,"a b c d") == 0)
{
assert(0);
}
fclose(fl);
return 0;
}
My input seed is a text file with a single line abhgsd.
Here is what I did:
$ afl-gcc main.c -o main
afl-cc 2.56b by <lcamtuf#google.com>
afl-as 2.56b by <lcamtuf#google.com>
[+] Instrumented 62 locations (64-bit, non-hardened mode, ratio 100%).
$ afl-fuzz -i INPUTS/ -o OUTPUTS ./main ##
And I got this red CAPITAL CRASH message:
afl-fuzz 2.56b by <lcamtuf#google.com>
[+] You have 8 CPU cores and 1 runnable tasks (utilization: 12%).
[+] Try parallel jobs - see /usr/local/share/doc/afl/parallel_fuzzing.txt.
[*] Checking CPU core loadout...
[+] Found a free CPU core, binding to #0.
[*] Checking core_pattern...
[-] Hmm, your system is configured to send core dump notifications to an
external utility. This will cause issues: there will be an extended delay
between stumbling upon a crash and having this information relayed to the
fuzzer via the standard waitpid() API.
To avoid having crashes misinterpreted as timeouts, please log in as root
and temporarily modify /proc/sys/kernel/core_pattern, like so:
echo core >/proc/sys/kernel/core_pattern
[-] PROGRAM ABORT : Pipe at the beginning of 'core_pattern'
Location : check_crash_handling(), afl-fuzz.c:7316
I'm a bit reluctant to change something unless I'm sure what I'm doing.
What's going on here? Should I listen to what AFL is saying?
You should probably change your core pattern, but you can change it back later. Many linux distros have a crash reporting service like apport, which relies on having core dumps from crashing processes piped to it via a core pattern like |/usr/share/apport/apport %p %s %c %d %P (see man 5 core) When the core pattern is set up this way, every time a program crashes, apport is run and the core is fed to it as standard input. So if you change the core pattern to just core, do your fuzzing, and then change the core pattern back to whatever it is currently, your distro's crash reporter should resume its normal operation.
AFL may have an environment variable to disable this check, as I know there exist environment variables to disable other pre-fuzzing checks (like AFL_SKIP_CRASHES allowing crashing input in the initial seeds), but this one is pretty low-cost to toggle.
The answer is right there in front of you.
log in as root and echo core >/proc/sys/kernel/core_pattern
I'm currently integrating libFuzzer in a project which parses files on the hard drive. I have some prior experience with AFL, where a command line like this one was used:
afl-fuzz -m500 -i input/ -o output/ -t100 -- program_to_fuzz ##
...where ## was a path to the generated input.
Looking at libFuzzer however, I see that the fuzz targets look like this:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
DoSomethingInterestingWithMyAPI(Data, Size);
return 0; // Non-zero return values are reserved for future use.
}
I understand that the input isn't provided in the form of a file, but as a buffer in-memory instead. The problem is that the program I'm trying to fuzz works with files and obtains its data through fread() calls. At no point in time is the whole input supposed to be loaded in memory (where, in the general case, it might not even fit); so there's not much I can do with a const uint8_t*.
Writing the buffer back to the hard drive to get back a file seems extremely inefficient. Is there a way around this?
You can do as in this example from google security team.
The buf_to_file defined here takes your buffer and returns a char* pathname you can then pass to you target:
(from https://github.com/google/security-research-pocs/blob/master/autofuzz/fuzz_utils.h#L27 )
// Write the data provided in buf to a new temporary file. This function is
// meant to be called by LLVMFuzzerTestOneInput() for fuzz targets that only
// take file names (and not data) as input.
//
// Return the path of the newly created file or NULL on error. The caller should
// eventually free the returned buffer (see delete_file).
extern "C" char *buf_to_file(const uint8_t *buf, size_t size);
Be sure to free the ressource with the delete_file function.
You could use LD_PRELOAD and override fread.
I am trying to provide DMA via PCI. For that purpose I have an example of sysfs driver. I succesfully stored data to RAM but unfortunately I cant read them. I have a functions store_dmaread and show_dmaread. I acces them via c code like this. The write function works fine but the show function which I open via read() works (reads the DMA data, prints them) but the user space buffer is not visible in that function.
char buf[2] = {3,3};
fw = open("/sys/bus/pci/devices/0000\:01\:00.0/dmaread", O_RDWR);
read (fw,buf, 2);
write (fw, buf, 2);
close(fw);
the function in the driver looks like this:
static ssize_t show_dmaread(struct device *dev, struct device_attribute *attr, char *buf)
{
printk("User space buffer value %d \n", buf[0]) // PRINTS 0
// MORE CODE WHICH WORKS
}
static ssize_t store_dmaread(struct device *dev, struct device_attribute *attr, const char *buf, size_t count)
{
// WORKS FINE THE ATTRIBUTE CHANGES ITS VALUE
}
Thanks a lot for help
From your question, it appears you are expecting that the char * buf passed to your show_dmaread function points directly to the userspace buffer passed to read (or at the very least has been populated with the data in the user-side buffer):
However, looking in Documentation/filesystem/sysfs.txt it says:
sysfs allocates a buffer of size (PAGE_SIZE) and passes it to the
method. Sysfs will call the method exactly once for each read or
write. This forces the following behavior on the method
implementations:
On read(2), the show() method should fill the entire buffer. Recall that an attribute should only be exporting one value, or an
array of similar values, so this shouldn't be that expensive.
This allows userspace to do partial reads and forward seeks
arbitrarily over the entire file at will. If userspace seeks back to
zero or does a pread(2) with an offset of '0' the show() method will
be called again, rearmed, to fill the buffer.
Which leads me to believe you are getting a newly allocated buffer and that some other kernel code manages copying your buffer back over to userspace.
I want to implement the already defined system calls in PintOS ( halt(), create()...etc defined in pintos/src/lib/user/syscall.c ). The current system call handler in pintos/src/userprog/syscall.c does not do anything. How do I make a process that makes system calls. Further I need to myself add a few system calls. How do I proceed in that too. But first I need to implement the existing system calls.
The default implementation in pintos terminates the calling process.
goto this link.There is explanation on where to modify the code to implement the system calls.
The "src/examples" directory contains a few sample user programs.
The "Makefile" in this directory compiles the provided examples, and you can edit it compile your own programs as well.
This program/process when run will inturn make a system call.
Use gdb to follow the execution of one such program a simple printf statement will eventually call write system call to STDOUT file.
The link given also has pointers on how to run pintos on gdb, my guess is you are using either bochs or qemu.In any case just run the gdb once with a simple hello world program running on pintos.
This will give u an idea of how the system call is made.
static void
syscall_handler (struct intr_frame *f)// UNUSED)
{
int *p=f->esp;
switch(*p)
case *p=SYS_CREATE // NUMBER # DEFINED
const char *name=*(p+1); //extract the filename
if(name==NULL||*name==NULL)
exit(-1);
off_t size=(int32_t)*(p+2);//extract file size
f->eax=filesys_create(name,size,_FILE); //call filesys_create
//eax will have the return value
}
This is pseudo code for sys_create .. all file system related system call are very trivial,
Filesys realted system calls like open read write close needs you to translate file to their corresponding fd (file descriptor). You need to add a file table for each process to keep track this, this can either be preprocess data or a global data.(UR choice),
case (*p==SYS_WRITE)
{
// printf("wite syscall\n");
char *buffer=*(p+2);
unsigned size=*(p+3);
int fd=*(p+1);
// getiing the fd of specified file
struct file *fil= thread_current()->fdtable[fd];/ my per thread fdtable
if(fd==1) goto here;
if(is_directory(fil->inode)){
exit(-1);
goto done;
}
here:
if(buffer>=PHYS_BASE)exit(-1);
if(fd<0||fd>=128){exit(-1);}
if(fd==0){exit(-1);} // writing to STDIN
if(fd==1) //writing to STDOUT
{
int a=(int)size;
while(a>=100)
{
putbuf(buffer,100);
buffer=buffer+100;
a-=100;
}
putbuf(buffer,a);
f->eax=(int)size;
}
else
if(thread_current()->fdtable[fd]==NULL)
{f->eax=-1;}
else
{
f->eax=file_write(thread_current()->fdtable[fd],buffer,(off_t)size);
}
done: ;
}//printf("write");} /* Write to a file. */
Open - adds anew entry to fdtable and return the fd number u give to the file,
close - remove that entry from fd table
read - similar to write.
The process_create ,wait are not simple to implement...
Cheers :)
I'm using the perl module Image:Imlib2 to resize photos. Here is the code:
#create thumbnail
my $old = Image::Imlib2->load("$upload_dir/$name");
my $new = $old->create_scaled_image(80, 80);
$new->save("$upload_dir/$thumbnail_name");
This code works fine when I am saving a jpg or png file, but whenever I save a gif, I get an internal server error. Here is the error I get in my apache log file:
Image::Imlib2 save error: Unknown error at /path/to/script/script.pl
Any Ideas?
Thanks!
I don't think Imlib2 supports writing GIF files at all. From a rather old mailing list posting:
I discovered that Imlib2 has absolutely NO support for writing out gif files.
Furthermore, if you look at the source, you'll see a couple files of interest:
imlib2-1.4.5/src/modules/loaders/loader_png.c
imlib2-1.4.5/src/modules/loaders/loader_gif.c
Inside loader_png.c you'll find this:
char
load(ImlibImage * im, ImlibProgressFunction progress,
char progress_granularity, char immediate_load)
{
/*...*/
}
char
save(ImlibImage * im, ImlibProgressFunction progress, char progress_granularity)
{
/*...*/
}
and inside loader_gif.c, you'll find:
char
load(ImlibImage * im, ImlibProgressFunction progress, char progress_granularity,
char immediate_load)
{
/* ... */
}
but no save implementation. So it looks like Imlib2 can read GIFs but can't write them and that's where your trouble lies.
I'd recommend that you switch to GraphicsMagick and Graphics::Magick. GraphicsMagick is a fork of ImageMagick that is faster and has fewer bugs, this is what Flickr uses internally so it should be good enough for you. GraphicsMagick unfortunately uses the somewhat strange ImageMagick API but you can hide the ugly details behind a wrapper without too much difficulty.
Alternatively, save all your thumbnails as JPEGs or PNGs.