[patch] Nasty mmap problem with huge files

Hi, I've been tracking down a very odd bug this afternoon. As it turns
out it is caused by enabling xine's mmap() support for the input_file.c.

I'm running 32 bit linux 2.6.21. The file in question is 0x10e4da000
bytes long (you can probably guess what kind of bug this is by now :)

Anyway, the issue stems from the definition of mmap():

void *mmap(void *start, size_t length, int prot, int flags, int fd,
off_t offset);

compare this to the definition of st_size in struct stat:
off_t     st_size;    /* total size, in bytes */

On my machine (in input_file.c) sizeof(size_t) ==4, whilst
sizeof(off_t) == 8. However the compiler doesn't generate a warning
when the following is done in xine's code:

     if ( (this->mmap_base = mmap(NULL, sbuf.st_size, PROT_READ,
MAP_SHARED, this->fh, 0)) != (void*)-1

So it silently truncates the upper part of the length. Obviously you
cannot mmap() a file that large into (32 bit) memory anyway, but as it turns
out, mmapping() 0xe4da000 succeeds, which causes... problems.

The patch (against xine-lib 1.1.6) does two things:
* Check that the length will not be truncated, while still allowing
for mmap()s of large files under 64 bit OSes.
* A correctness fix: if mmap() fails, this->mmap_base will be set to
0xffffffff. Later on when the file is closed, this means it was
attempting to do munmap(0xffffffff).

0 files changed, 0 insertions, 0 deletions