-
- Downloads
Parallel scanning: early proof-of-concept implementation
And it's not looking well; this implementation seems to be 3x slower in the hot cache scenario with -J8, which is a major regression. There's way too much lock contention and context switching. Haven't tested with actual disk I/O yet and I've not yet measured how much parallelism this approach will actually get us in practice, nor whether the disk access patterns of this approach make a whole lot of sense. Maybe this low-memory approach will not work out and I'll end up rewriting this to scan disjoint subtrees after all. TODO: - Validate how much parallelism we can actually get with this algorithm - Lots of benchmarking and tuning (and most likely some re-architecting) - Re-implement exclude pattern matching - Document -J option - Make OOM handling thread-safe
Loading
Please register or sign in to comment