Messages in this thread | | | Date | Mon, 28 Jul 2014 11:13:29 -0400 | Subject | Re: Multi Core Support for compression in compression.c | From | Nick Krause <> |
| |
On Mon, Jul 28, 2014 at 6:10 AM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote: > On 07/27/2014 11:21 PM, Nick Krause wrote: >> On Sun, Jul 27, 2014 at 10:56 PM, Austin S Hemmelgarn >> <ahferroin7@gmail.com> wrote: >>> On 07/27/2014 04:47 PM, Nick Krause wrote: >>>> This may be a bad idea , but compression in brtfs seems to be only >>>> using one core to compress. >>>> Depending on the CPU used and the amount of cores in the CPU we can >>>> make this much faster >>>> with multiple cores. This seems bad by my reading at least I would >>>> recommend for writing compression >>>> we write a function to use a certain amount of cores based on the load >>>> of the system's CPU not using >>>> more then 75% of the system's CPU resources as my system when idle has >>>> never needed more >>>> then one core of my i5 2500k to run when with interrupts for opening >>>> eclipse are running. For reading >>>> compression on good core seems fine to me as testing other compression >>>> software for reads , it's >>>> way less CPU intensive. >>>> Cheers Nick >>> We would probably get a bigger benefit from taking an approach like >>> SquashFS has recently added, that is, allowing multi-threaded >>> decompression fro reads, and decompressing directly into the pagecache. >>> Such an approach would likely make zlib compression much more scalable >>> on large systems. >>> >>> >> >> Austin, >> That seems better then my idea as you seem to be more up to date on >> brtfs devolopment. >> If you and the other developers of brtfs are interested in adding this >> as a feature please let >> me known as I would like to help improve brtfs as the file system as >> an idea is great just >> seems like it needs a lot of work :). >> Nick > I wouldn't say that I am a BTRFS developer (power user maybe?), but I > would definitely say that parallelizing compression on writes would be a > good idea too (especially for things like lz4, which IIRC is either in > 3.16 or in the queue for 3.17). Both options would be a lot of work, > but almost any performance optimization would. I would almost say that > it would provide a bigger performance improvement to get BTRFS to > intelligently stripe reads and writes (at the moment, any given worker > thread only dispatches one write or read to a single device at a time, > and any given write() or read() syscall gets handled by only one worker). >
I will look into this idea and see if I can do this for writes. Regards Nick
| |