Messages in this thread | | | Subject | Re: [PATCHSET v5] Make background writeback great again for the first time | From | Jens Axboe <> | Date | Fri, 13 May 2016 12:29:10 -0600 |
| |
On 05/11/2016 10:36 AM, Jan Kara wrote: > On Tue 03-05-16 14:17:19, Jan Kara wrote: >> The question remains how common a pattern where throttling of background >> writeback delays also something else is. I'll schedule a couple of >> benchmarks to measure impact of your patches for a wider range of workloads >> (but sadly pretty limited set of hw). If ext3 is the only one seeing >> issues, I would be willing to accept that ext3 takes the hit since it is >> doing something rather stupid (but inherent in its journal design) and we >> have a way to deal with this either by enabling delayed allocation or by >> turning off the writeback throttling... > > So I've run some benchmarks on a machine with 6 GB of RAM and SSD with > queue depth 32. The filesystem on the disk was XFS this time. I've found > couple of regressions. A clear one is with dbench (version 4). The average > throughput numbers look like: > > Baseline WBT > Hmean mb/sec-1 30.26 ( 0.00%) 18.67 (-38.28%) > Hmean mb/sec-2 40.71 ( 0.00%) 31.25 (-23.23%) > Hmean mb/sec-4 52.67 ( 0.00%) 46.83 (-11.09%) > Hmean mb/sec-8 69.51 ( 0.00%) 64.35 ( -7.42%) > Hmean mb/sec-16 91.07 ( 0.00%) 86.46 ( -5.07%) > Hmean mb/sec-32 115.10 ( 0.00%) 110.29 ( -4.18%) > Hmean mb/sec-64 145.14 ( 0.00%) 134.97 ( -7.00%) > Hmean mb/sec-512 93.99 ( 0.00%) 133.85 ( 42.41%) > > There were also some losses in a filebench webproxy workload (I can give > you exact details of the settings if you want to reproduce it). > > Also, and this really puzzles me, I've seen higher read latencies in some > cases (I've verified they are not just noise by rerunning the test for > kernel with writeback throttling patches). For example with the following > fio job file: > > [global] > direct=0 > ioengine=sync > runtime=300 > time_based > invalidate=1 > blocksize=4096 > size=10g # Just random value, we are running time based workload > log_avg_msec=10 > group_reporting=1 > > [writer] > nrfiles=1 > filesize=1g > fdatasync=256 > readwrite=randwrite > numjobs=4 > > [reader] > # Simulate random reading from different files, switching to different file > # after 16 ios. This somewhat simulates application startup. > new_group > filesize=100m > nrfiles=20 > file_service_type=random:16 > readwrite=randread > > I get the following results: > > Throughput Baseline WBT > Hmean kb/sec-writer-write 591.60 ( 0.00%) 507.00 (-14.30%) > Hmean kb/sec-reader-read 211.81 ( 0.00%) 137.53 (-35.07%) > > So both read and write throughput have suffered. And latencies don't offset > for the loss either: > > FIO read latency > Min latency-read 1383.00 ( 0.00%) 1519.00 ( -9.83%) > 1st-qrtle latency-read 3485.00 ( 0.00%) 5235.00 (-50.22%) > 2nd-qrtle latency-read 4708.00 ( 0.00%) 15028.00 (-219.20%) > 3rd-qrtle latency-read 10286.00 ( 0.00%) 57622.00 (-460.20%) > Max-90% latency-read 195834.00 ( 0.00%) 167149.00 ( 14.65%) > Max-93% latency-read 273145.00 ( 0.00%) 200319.00 ( 26.66%) > Max-95% latency-read 335434.00 ( 0.00%) 220695.00 ( 34.21%) > Max-99% latency-read 537017.00 ( 0.00%) 347174.00 ( 35.35%) > Max latency-read 991101.00 ( 0.00%) 485835.00 ( 50.98%) > Mean latency-read 51282.79 ( 0.00%) 49953.95 ( 2.59%) > > So we have reduced the extra high read latencies which is nice but on > average there is no change. > > And another fio jobfile which doesn't look great: > > [global] > direct=0 > ioengine=sync > runtime=300 > blocksize=4096 > invalidate=1 > time_based > ramp_time=5 # Let the flusher thread start before taking measurements > log_avg_msec=10 > group_reporting=1 > > [writer] > nrfiles=1 > filesize=$((MEMTOTAL_BYTES*2)) > readwrite=randwrite > > [reader] > # Simulate random reading from different files, switching to different file > # after 16 ios. This somewhat simulates application startup. > new_group > filesize=100m > nrfiles=20 > file_service_type=random:16 > readwrite=randread > > The throughput numbers look like: > Hmean kb/sec-writer-write 24707.22 ( 0.00%) 19912.23 (-19.41%) > Hmean kb/sec-reader-read 886.65 ( 0.00%) 905.71 ( 2.15%) > > So we've got significant hit in writes not really offset by a big increase > in reads. Read latency numbers look like (I show the WBT numbers for two runs > just so that one can see how variable the latency numbers are because I was > puzzled by very high max latency for WBT kernels - quartiles seem rather > stable higher percentiles and min/max are rather variable): > > Baseline WBT WBT > Min latency-read 1230.00 ( 0.00%) 1560.00 (-26.83%) 1100.00 ( 10.57%) > 1st-qrtle latency-read 3357.00 ( 0.00%) 3351.00 ( 0.18%) 3351.00 ( 0.18%) > 2nd-qrtle latency-read 4074.00 ( 0.00%) 4056.00 ( 0.44%) 4022.00 ( 1.28%) > 3rd-qrtle latency-read 5198.00 ( 0.00%) 5145.00 ( 1.02%) 5095.00 ( 1.98%) > Max-90% latency-read 6594.00 ( 0.00%) 6370.00 ( 3.40%) 6130.00 ( 7.04%) > Max-93% latency-read 11251.00 ( 0.00%) 9410.00 ( 16.36%) 6654.00 ( 40.86%) > Max-95% latency-read 14769.00 ( 0.00%) 13231.00 ( 10.41%) 10306.00 ( 30.22%) > Max-99% latency-read 27826.00 ( 0.00%) 28728.00 ( -3.24%) 25077.00 ( 9.88%) > Max latency-read 80202.00 ( 0.00%) 186491.00 (-132.53%) 141346.00 (-76.24%) > Mean latency-read 5356.12 ( 0.00%) 5229.00 ( 2.37%) 4927.23 ( 8.01%) > > I have run also other tests but they have mostly shown no significant > difference.
Thanks Jan, this is great and super useful! I'm revamping certain parts of it to deal with write back caching better, and I'll take a look at the regressions that you reported.
What kind of SSD is this? I'm assuming it's SATA (QD=32), and then it would probably be a safe assumption that it's flagging itself as having a volatile write back cache, would that be a correct assumption?
Are you using scsi-mq, or do you have an IO scheduler attached to it?
-- Jens Axboe
| |