feat(filestore): add mmap reader option

[Dreamacro]

Motivation

I have a custom IPFS implementation based on boxo, which uses FileManager on Windows because I need to store files directly rather than file blocks. The pull logic is roughly as follows:

1. Pull the entire ProtoNode tree.
2. Create a sparse file, which can be larger than 100 GB.
3. Randomly pull RawNode file blocks and write them to the system. Meanwhile, this Windows node also communicates and synchronizes with other nodes.

When hundreds of Windows nodes are working simultaneously, I noticed that the system's memory consumption becomes very high, and the memory is immediately released once the file download is complete. Interestingly, the memory consumption is not by the program itself but by the operating system. After some investigation, I found the root cause: Windows caches the read file blocks (even if the *os.File is already closed).

This can be reproduced with the following code:

package main

import (
	"crypto/rand"
	"fmt"
	"io"
	"os"
	"runtime"
	"sync"
	"time"
	"unsafe"

	"sparse/ratelimit"

	"golang.org/x/sys/windows"
)

func readBuf(path string, idx int, size int) []byte {
	f, err := os.Open(path)
	if err != nil {
		panic(err)
	}
	defer f.Close()

	buf := make([]byte, size)
	if _, err := f.ReadAt(buf, int64(idx)); err != nil {
		panic(err)
	}

	return buf
}

type FILE_SET_SPARSE_BUFFER struct {
	SetSparse bool
}

// SetSparse makes the file be a sparse file
func SetSparse(out *os.File, open bool) error {
	lpInBuffer := FILE_SET_SPARSE_BUFFER{
		SetSparse: open,
	}

	var bytesReturned uint32
	err := windows.DeviceIoControl(
		windows.Handle(out.Fd()),
		windows.FSCTL_SET_SPARSE,
		(*byte)(unsafe.Pointer(&lpInBuffer)),
		uint32(unsafe.Sizeof(lpInBuffer)),
		nil, 0, &bytesReturned, nil,
	)
	if err != nil {
		return fmt.Errorf("DeviceIoControl FSCTL_SET_SPARSE: %w", err)
	}
	return nil
}

func main() {
	filename := "test.bin"
	size := 1024 * 1024 * 1024 * 100 // 100GB
	defer os.Remove(filename)

	f, err := os.OpenFile(filename, os.O_RDWR|os.O_CREATE, 0o644)
	if err != nil {
		panic(err)
	}
	defer f.Close()

	if err := SetSparse(f, true); err != nil {
		panic(err)
	}

	if err := f.Truncate(int64(size)); err != nil {
		panic(err)
	}

	chunkSize := 256 * 1024
	offset := make([]int, size/chunkSize)
	for i := 0; i < int(size); i += chunkSize {
		offset = append(offset, i)
	}

	mrand.Shuffle(len(offset), func(i, j int) {
		offset[i], offset[j] = offset[j], offset[i]
	})

	bucket := ratelimit.NewFromMbps(200)

	ch := make(chan int)

	cache := map[int]struct{}{}

	rwMux := sync.RWMutex{}

	numCpu := runtime.GOMAXPROCS(0)

	println(numCpu)

	rf, err := os.Open(filename)
	if err != nil {
		panic(err)
	}
	defer rf.Close()

	for i := 0; i < numCpu; i++ {
		go func() {
			for {
				rwMux.RLock()
				var key int
				for k := range cache {
					key = k
					break
				}
				rwMux.RUnlock()
				readBuf(filename, key, chunkSize)
				time.Sleep(time.Millisecond * 1)
			}
		}()
	}

	wg := sync.WaitGroup{}
	for i := 0; i < numCpu; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			buf := make([]byte, chunkSize)
			for idx := range ch {
				io.ReadFull(rand.Reader, buf)
				bucket.Wait(int64(chunkSize))

				if _, err := f.WriteAt(buf, int64(idx)); err != nil {
					panic(err)
				}

				// if err := f.Sync(); err != nil {
				// 	panic(err)
				// }

				rwMux.Lock()
				cache[idx] = struct{}{}
				rwMux.Unlock()
			}
		}()
	}

	for _, start := range offset {
		ch <- start
	}

	wg.Wait()
}

Solution

Using mmap can solve this problem (implemented with CreateFileMapping on Windows). To maintain backward compatibility, a WithMMapReader option has been added to FileManager. Enabling this option on Windows can prevent excessive memory consumption.

The downside is that it relies on the exp package, but it only changes from an indirect dependency to a direct dependency. Alternatively, the code from this package can be copied directly into the project.

[codecov]

Codecov Report

Attention: Patch coverage is 83.78378% with 6 lines in your changes missing coverage. Please review.

Project coverage is 60.41%. Comparing base (29598b2) to head (06b91df).

Files with missing lines	Patch %	Lines
filestore/filereader.go	70.00%	4 Missing and 2 partials ⚠️

@@            Coverage Diff             @@
##             main     #665      +/-   ##
==========================================
+ Coverage   60.39%   60.41%   +0.02%     
==========================================
  Files         243      244       +1     
  Lines       31025    31054      +29     
==========================================
+ Hits        18737    18761      +24     
- Misses      10621    10629       +8     
+ Partials     1667     1664       -3     

Files with missing lines	Coverage Δ
filestore/fsrefstore.go	41.90% <100.00%> (+4.09%)	⬆️
filestore/filereader.go	70.00% <70.00%> (ø)

... and 10 files with indirect coverage changes

[lidel]

Thank you @Dreamacro!

Triage notes:

• this seems to solve Windows-specific perf. problem, and should not be applied to all platforms.
• please refactor to leverage build flags, see prior art in https://github.com/ipfs/boxo/blob/main/files/filewriter_windows.go

[Dreamacro]

@lidel Yes, mmap was introduced to solve the Windows-specific problem, but WithMMapReader is an option-in and the default behavior is still to use go std's os.File, so I implemented WithMMapReader to let the user choose whether to use mmap or not, I don't think a build tag is needed here, mmap is worked well to all major platforms.

[lidel]

Triage notes:

• overall sounds good, as long it remains opt-in
  • we will dogfood this in Kubo, will circle back to this as part of 0.31 activities Release 0.31 kubo#10499
  • because mmap impl. is in golang.org/x/exp plan is to have it as opt-in flag for now, similar to IPFS_FD_MAX
• small ask to rename func inline

[Dreamacro]

done

[Dreamacro]

@lidel Hi, should I do anything else?

[hsanjuan]

Eyeballed this and it looks good to me. I defer to @lidel for merging.