lkml.org 
[lkml]   [2009]   [Jul]   [14]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [RESEND] [PATCH] readahead:add blk_run_backing_dev
Ronald Moesbergen, on 07/13/2009 04:12 PM wrote:
> 2009/7/10 Vladislav Bolkhovitin <vst@vlnb.net>:
>> Vladislav Bolkhovitin, on 07/10/2009 12:43 PM wrote:
>>> Ronald Moesbergen, on 07/10/2009 10:32 AM wrote:
>>>>> I've also long ago noticed that reading data from block devices is
>>>>> slower
>>>>> than from files from mounted on those block devices file systems. Can
>>>>> anybody explain it?
>>>>>
>>>>> Looks like this is strangeness #2 which we uncovered in our tests (the
>>>>> first
>>>>> one was earlier in this thread why the context RA doesn't work with
>>>>> cooperative I/O threads as good as it should).
>>>>>
>>>>> Can you rerun the same 11 tests over a file on the file system, please?
>>>> I'll see what I can do. Just te be sure: you want me to run
>>>> blockdev-perftest on a file on the OCFS2 filesystem which is mounted
>>>> on the client over iScsi, right?
>>> Yes, please.
>> Forgot to mention that you should also configure your backend storage as a
>> big file on a file system (preferably, XFS) too, not as direct device, like
>> /dev/vg/db-master.
>
> Ok, here are the results:
>
> client kernel: 2.6.26-15lenny3 (debian)
> server kernel: 2.6.29.5 with readahead patch
>
> Test done with XFS on both the target and the initiator. This confirms
> your findings, using files instead of block devices is faster, but
> only when using the io_context patch.

Seems, correct, except case (2), which is still 10% faster.

> Without io_context patch:
> 1) client: default, server: default
> blocksize R R R R(avg, R(std R
> (bytes) (s) (s) (s) MB/s) ,MB/s) (IOPS)
> 67108864 18.327 18.327 17.740 56.491 0.872 0.883
> 33554432 18.662 18.311 18.116 55.772 0.683 1.743
> 16777216 18.900 18.421 18.312 55.229 0.754 3.452
> 8388608 18.893 18.533 18.281 55.156 0.743 6.895
> 4194304 18.512 18.097 18.400 55.850 0.536 13.963
> 2097152 18.635 18.313 18.676 55.232 0.486 27.616
> 1048576 18.441 18.264 18.245 55.907 0.267 55.907
> 524288 17.773 18.669 18.459 55.980 1.184 111.960
> 262144 18.580 18.758 17.483 56.091 1.767 224.365
> 131072 17.224 18.333 18.765 56.626 2.067 453.006
> 65536 18.082 19.223 18.238 55.348 1.483 885.567
> 32768 17.719 18.293 18.198 56.680 0.795 1813.766
> 16384 17.872 18.322 17.537 57.192 1.024 3660.273
>
> 2) client: default, server: 64 max_sectors_kb, RA default
> blocksize R R R R(avg, R(std R
> (bytes) (s) (s) (s) MB/s) ,MB/s) (IOPS)
> 67108864 18.738 18.435 18.400 55.283 0.451 0.864
> 33554432 18.046 18.167 17.572 57.128 0.826 1.785
> 16777216 18.504 18.203 18.377 55.771 0.376 3.486
> 8388608 22.069 18.554 17.825 53.013 4.766 6.627
> 4194304 19.211 18.136 18.083 55.465 1.529 13.866
> 2097152 18.647 17.851 18.511 55.866 1.071 27.933
> 1048576 19.084 18.177 18.194 55.425 1.249 55.425
> 524288 18.999 18.553 18.380 54.934 0.763 109.868
> 262144 18.867 18.273 18.063 55.668 1.020 222.673
> 131072 17.846 18.966 18.193 55.885 1.412 447.081
> 65536 18.195 18.616 18.482 55.564 0.530 889.023
> 32768 17.882 18.841 17.707 56.481 1.525 1807.394
> 16384 17.073 18.278 17.985 57.646 1.689 3689.369
>
> 3) client: default, server: default max_sectors_kb, RA 2MB
> blocksize R R R R(avg, R(std R
> (bytes) (s) (s) (s) MB/s) ,MB/s) (IOPS)
> 67108864 18.658 17.830 19.258 55.162 1.750 0.862
> 33554432 17.193 18.265 18.517 56.974 1.854 1.780
> 16777216 17.531 17.681 18.776 56.955 1.720 3.560
> 8388608 18.234 17.547 18.201 56.926 1.014 7.116
> 4194304 18.057 17.923 17.901 57.015 0.218 14.254
> 2097152 18.565 17.739 17.658 56.958 1.277 28.479
> 1048576 18.393 17.433 17.314 57.851 1.550 57.851
> 524288 18.939 17.835 18.972 55.152 1.600 110.304
> 262144 18.562 19.005 18.069 55.240 1.141 220.959
> 131072 19.574 17.562 18.251 55.576 2.476 444.611
> 65536 19.117 18.019 17.886 55.882 1.647 894.115
> 32768 18.237 17.415 17.482 57.842 1.200 1850.933
> 16384 17.760 18.444 18.055 56.631 0.876 3624.391
>
> 4) client: default, server: 64 max_sectors_kb, RA 2MB
> blocksize R R R R(avg, R(std R
> (bytes) (s) (s) (s) MB/s) ,MB/s) (IOPS)
> 67108864 18.368 17.495 18.524 56.520 1.434 0.883
> 33554432 18.209 17.523 19.146 56.052 2.027 1.752
> 16777216 18.765 18.053 18.550 55.497 0.903 3.469
> 8388608 17.878 17.848 18.389 56.778 0.774 7.097
> 4194304 18.058 17.683 18.567 56.589 1.129 14.147
> 2097152 18.896 18.384 18.697 54.888 0.623 27.444
> 1048576 18.505 17.769 17.804 56.826 1.055 56.826
> 524288 18.319 17.689 17.941 56.955 0.816 113.910
> 262144 19.227 17.770 18.212 55.704 1.821 222.815
> 131072 18.738 18.227 17.869 56.044 1.090 448.354
> 65536 19.319 18.525 18.084 54.969 1.494 879.504
> 32768 18.321 17.672 17.870 57.047 0.856 1825.495
> 16384 18.249 17.495 18.146 57.025 1.073 3649.582
>
> With io_context patch:
> 5) client: default, server: default
> blocksize R R R R(avg, R(std R
> (bytes) (s) (s) (s) MB/s) ,MB/s) (IOPS)
> 67108864 12.393 11.925 12.627 83.196 1.989 1.300
> 33554432 11.844 11.855 12.191 85.610 1.142 2.675
> 16777216 12.729 12.602 12.068 82.187 1.913 5.137
> 8388608 12.245 12.060 14.081 80.419 5.469 10.052
> 4194304 13.224 11.866 12.110 82.763 3.833 20.691
> 2097152 11.585 12.584 11.755 85.623 3.052 42.811
> 1048576 12.166 12.144 12.321 83.867 0.539 83.867
> 524288 12.019 12.148 12.160 84.568 0.448 169.137
> 262144 12.014 12.378 12.074 84.259 1.095 337.036
> 131072 11.840 12.068 11.849 85.921 0.756 687.369
> 65536 12.098 11.803 12.312 84.857 1.470 1357.720
> 32768 11.852 12.635 11.887 84.529 2.465 2704.931
> 16384 12.443 13.110 11.881 82.197 3.299 5260.620
>
> 6) client: default, server: 64 max_sectors_kb, RA default
> blocksize R R R R(avg, R(std R
> (bytes) (s) (s) (s) MB/s) ,MB/s) (IOPS)
> 67108864 13.033 12.122 11.950 82.911 3.110 1.295
> 33554432 12.386 13.357 12.082 81.364 3.429 2.543
> 16777216 12.102 11.542 12.053 86.096 1.860 5.381
> 8388608 12.240 11.740 11.789 85.917 1.601 10.740
> 4194304 11.824 12.388 12.042 84.768 1.621 21.192
> 2097152 11.962 12.283 11.973 84.832 1.036 42.416
> 1048576 12.639 11.863 12.010 84.197 2.290 84.197
> 524288 11.809 12.919 11.853 84.121 3.439 168.243
> 262144 12.105 12.649 12.779 81.894 1.940 327.577
> 131072 12.441 12.769 12.713 81.017 0.923 648.137
> 65536 12.490 13.308 12.440 80.414 2.457 1286.630
> 32768 13.235 11.917 12.300 82.184 3.576 2629.883
> 16384 12.335 12.394 12.201 83.187 0.549 5323.990
>
> 7) client: default, server: default max_sectors_kb, RA 2MB
> blocksize R R R R(avg, R(std R
> (bytes) (s) (s) (s) MB/s) ,MB/s) (IOPS)
> 67108864 12.017 12.334 12.151 84.168 0.897 1.315
> 33554432 12.265 12.200 11.976 84.310 0.864 2.635
> 16777216 12.356 11.972 12.292 83.903 1.165 5.244
> 8388608 12.247 12.368 11.769 84.472 1.825 10.559
> 4194304 11.888 11.974 12.144 85.325 0.754 21.331
> 2097152 12.433 10.938 11.669 87.911 4.595 43.956
> 1048576 11.748 12.271 12.498 84.180 2.196 84.180
> 524288 11.726 11.681 12.322 86.031 2.075 172.062
> 262144 12.593 12.263 11.939 83.530 1.817 334.119
> 131072 11.874 12.265 12.441 84.012 1.648 672.093
> 65536 12.119 11.848 12.037 85.330 0.809 1365.277
> 32768 12.549 12.080 12.008 83.882 1.625 2684.238
> 16384 12.369 12.087 12.589 82.949 1.385 5308.766
>
> 8) client: default, server: 64 max_sectors_kb, RA 2MB
> blocksize R R R R(avg, R(std R
> (bytes) (s) (s) (s) MB/s) ,MB/s) (IOPS)
> 67108864 12.664 11.793 11.963 84.428 2.575 1.319
> 33554432 11.825 12.074 12.442 84.571 1.761 2.643
> 16777216 11.997 11.952 10.905 88.311 3.958 5.519
> 8388608 11.866 12.270 11.796 85.519 1.476 10.690
> 4194304 11.754 12.095 12.539 84.483 2.230 21.121
> 2097152 11.948 11.633 11.886 86.628 1.007 43.314
> 1048576 12.029 12.519 11.701 84.811 2.345 84.811
> 524288 11.928 12.011 12.049 85.363 0.361 170.726
> 262144 12.559 11.827 11.729 85.140 2.566 340.558
> 131072 12.015 12.356 11.587 85.494 2.253 683.952
> 65536 11.741 12.113 11.931 85.861 1.093 1373.770
> 32768 12.655 11.738 12.237 83.945 2.589 2686.246
> 16384 11.928 12.423 11.875 84.834 1.711 5429.381
>
> 9) client: 64 max_sectors_kb, default RA. server: 64 max_sectors_kb, RA 2MB
> blocksize R R R R(avg, R(std R
> (bytes) (s) (s) (s) MB/s) ,MB/s) (IOPS)
> 67108864 13.570 13.491 14.299 74.326 1.927 1.161
> 33554432 13.238 13.198 13.255 77.398 0.142 2.419
> 16777216 13.851 13.199 13.463 75.857 1.497 4.741
> 8388608 13.339 16.695 13.551 71.223 7.010 8.903
> 4194304 13.689 13.173 14.258 74.787 2.415 18.697
> 2097152 13.518 13.543 13.894 75.021 0.934 37.510
> 1048576 14.119 14.030 13.820 73.202 0.659 73.202
> 524288 13.747 14.781 13.820 72.621 2.369 145.243
> 262144 14.168 13.652 14.165 73.189 1.284 292.757
> 131072 14.112 13.868 14.213 72.817 0.753 582.535
> 65536 14.604 13.762 13.725 73.045 2.071 1168.728
> 32768 14.796 15.356 14.486 68.861 1.653 2203.564
> 16384 13.079 13.525 13.427 76.757 1.111 4912.426
>
> 10) client: default max_sectors_kb, 2MB RA. server: 64 max_sectors_kb, RA 2MB
> blocksize R R R R(avg, R(std R
> (bytes) (s) (s) (s) MB/s) ,MB/s) (IOPS)
> 67108864 20.372 18.077 17.262 55.411 3.800 0.866
> 33554432 17.287 17.620 17.828 58.263 0.740 1.821
> 16777216 16.802 18.154 17.315 58.831 1.865 3.677
> 8388608 17.510 18.291 17.253 57.939 1.427 7.242
> 4194304 17.059 17.706 17.352 58.958 0.897 14.740
> 2097152 17.252 18.064 17.615 58.059 1.090 29.029
> 1048576 17.082 17.373 17.688 58.927 0.838 58.927
> 524288 17.129 17.271 17.583 59.103 0.644 118.206
> 262144 17.411 17.695 18.048 57.808 0.848 231.231
> 131072 17.937 17.704 18.681 56.581 1.285 452.649
> 65536 17.927 17.465 17.907 57.646 0.698 922.338
> 32768 18.494 17.820 17.719 56.875 1.073 1819.985
> 16384 18.800 17.759 17.575 56.798 1.666 3635.058
>
> 11) client: 64 max_sectors_kb, 2MB. RA server: 64 max_sectors_kb, RA 2MB
> blocksize R R R R(avg, R(std R
> (bytes) (s) (s) (s) MB/s) ,MB/s) (IOPS)
> 67108864 20.045 21.881 20.018 49.680 2.037 0.776
> 33554432 20.768 20.291 20.464 49.938 0.479 1.561
> 16777216 21.563 20.714 20.429 49.017 1.116 3.064
> 8388608 21.290 21.109 21.308 48.221 0.205 6.028
> 4194304 22.240 20.662 21.088 48.054 1.479 12.013
> 2097152 20.282 21.098 20.580 49.593 0.806 24.796
> 1048576 20.367 19.929 20.252 50.741 0.469 50.741
> 524288 20.885 21.203 20.684 48.945 0.498 97.890
> 262144 19.982 21.375 20.798 49.463 1.373 197.853
> 131072 20.744 21.590 19.698 49.593 1.866 396.740
> 65536 21.586 20.953 21.055 48.314 0.627 773.024
> 32768 21.228 20.307 21.049 49.104 0.950 1571.327
> 16384 21.257 21.209 21.150 48.289 0.100 3090.498

The drop with 64 max_sectors_kb on the client is a consequence of how
CFQ is working. I can't find the exact code responsible for this, but
from all signs, CFQ stops delaying requests if amount of outstanding
requests exceeds some threshold, which is 2 or 3. With 64 max_sectors_kb
and 5 SCST I/O threads this threshold is exceeded, so CFQ doesn't
recover order of requests, hence the performance drop. With default 512
max_sectors_kb and 128K RA the server sees at max 2 requests at time.

Ronald, can you perform the same tests with 1 and 2 SCST I/O threads,
please?

You can limit amount of SCST I/O threads by num_threads parameter of
scst_vdisk module.

Thanks,
Vlad


\
 
 \ /
  Last update: 2009-07-14 20:55    [from the cache]
©2003-2011 Jasper Spaans