C++

Forum

 
mmap with files >2GB

Jun 17, 2012 at 1:32pm
Eric99 (3)
I've been trying to get mmap to work on a 3GB file (Linux Ubuntu 12.04), but it keeps returning MAP_FAILED. If I ignore it, I eventually get a segmentation fault, so I'm sure something is wrong somewhere.

I'm compling with 64 bit file options:
gcc -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 test.c

Here's an example I made for testing:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
 
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <inttypes.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <stdint.h>


int main(int argc, char **argv)
{
    uint8_t *base;

    int writable=0;
 
    if(argc != 2)    
        return 1;
 
    int file=0;
        if((file=open(argv[1],O_RDONLY)) < -1)
            return 1;
 
    struct stat fileStat;
    if(fstat(file,&fileStat) < 0)    
        return 1;
 
    printf("Information for %s\n",argv[1]);
    printf("---------------------------\n");
    printf("File Size: \t\t%" PRIuPTR " bytes\n", (uintptr_t) fileStat.st_size);
    printf("Number of Links: \t%d\n",fileStat.st_nlink);
    printf("File inode: \t\t%" PRIuPTR " bytes\n", (uintptr_t) fileStat.st_ino);
 
    printf("File Permissions: \t");
    printf( (S_ISDIR(fileStat.st_mode)) ? "d" : "-");
    printf( (fileStat.st_mode & S_IRUSR) ? "r" : "-");
    printf( (fileStat.st_mode & S_IWUSR) ? "w" : "-");
    printf( (fileStat.st_mode & S_IXUSR) ? "x" : "-");
    printf( (fileStat.st_mode & S_IRGRP) ? "r" : "-");
    printf( (fileStat.st_mode & S_IWGRP) ? "w" : "-");
    printf( (fileStat.st_mode & S_IXGRP) ? "x" : "-");
    printf( (fileStat.st_mode & S_IROTH) ? "r" : "-");
    printf( (fileStat.st_mode & S_IWOTH) ? "w" : "-");
    printf( (fileStat.st_mode & S_IXOTH) ? "x" : "-");
    printf("\n\n");
 
    printf("The file %s a symbolic link\n\n", (S_ISLNK(fileStat.st_mode)) ? "is" : "is not");
  
    fprintf(stderr,"file=%d flags=%d\n",file,writable);
    if (file == -1) {
        printf("Cannot open file %s\n", argv[1]);
        return 1;
    }

    if (fstat(file,&fileStat) == -1) {
        printf("Cannot get size of file %s\n", argv[1]);
        return 1;
    }
    if (fileStat.st_size == 0) {
        printf("file %s is an empty file\n", argv[1]);
        return 1;
    }
    
    base = mmap(NULL, fileStat.st_size, PROT_READ | (PROT_WRITE * writable),
            MAP_SHARED, file, 0);
    fprintf(stderr,"base: \t\t%" PRIuPTR " \n", (uintptr_t) base);
    if (base == MAP_FAILED) {
        fprintf(stderr, "Cannot map file %s into memory\n", argv[1]);
        return 1;
    }

    return 0;
}
Edit & run on cpp.sh


Running it on a test file >2GB produces this example output
1
2
3
4
5
6
7
8
9
10
 
Information for largefile.iso
 
File Size: 		3019309056 bytes
Number of Links: 	1
File inode: 		14948955 bytes
File Permissions: 	-rw-rw-r--

The file is not a symbolic link

file=3 flags=0
base: 		4294967295 
Cannot map file largefile.iso into memory


The line base == MAP_FAILED is comparing the 4294967295 to (void *) -1

Can someone clue me as to how to properly handle this large file with mmap? Am I running up against a physical memory limit or am I making some kind of integer size error?

Thanks in advance!
Last edited on Jun 17, 2012 at 1:33pm
Jun 17, 2012 at 1:41pm
Athar (4466)
Am I running up against a physical memory limit

If it's a 32-bit program, yes. Your address space is 4 GB, of which you can use at most 3 GB.
Jun 17, 2012 at 5:14pm
Eric99 (3)
I think you are right. I did some memory allocation tests on my 32-bit machine and found the max size I could allocate was 2.8G.

How do I handle files greater than 2.8G? Do I need to load the file in chunks? Or is there another method that can handle paging large files?
Last edited on Jun 17, 2012 at 5:16pm
Jun 17, 2012 at 5:39pm
Athar (4466)
Either map it in smaller chunks or just use regular file read/write operations instead of mapping it.
Jun 17, 2012 at 7:32pm
Eric99 (3)
Strange I thought that mmap used some kind of virtual memory map and didn't put it into RAM.

I think fseek methods will be slow, but I'll try rewriting the code and see.
Jun 17, 2012 at 7:41pm
Athar (4466)
mmap maps the file into your virtual address space, which has a size of 2^32 bytes=4 GB in a 32-bit application.
The contents of the file do not necessarily reside in physical RAM.
Topic archived. No new replies allowed.