Messages in this thread | | | Subject | RE: 2.6.13.2 aacraid regression | Date | Tue, 4 Oct 2005 08:44:21 -0400 | From | "Salyzyn, Mark" <> |
| |
I am sorry you are experiencing problems with the driver and the Adapter.
The controller is failing to complete commands within 130 seconds (total) under load; yes, the adapter is *DEAD*. Although the changes in the driver between these two releases is numerous, the primary goal of those changes increased the performance of the system which could take a marginal controller over the edge. We have to determine if you have a marginal controller, cables and drives first before we can explore the possibility that these changes are stressing the controller's firmware past some breaking point that we can mitigate with adjustments in the driver. Tuning the driver with a bad controller or set of drives will only lead to frustration and inconsistent results.
Please confirm with Adaptec Technical support that you have a compatible set of drives with the controller. If compatible, please ensure that your SATA cables are clean and not mechanically stressed. Make sure you have the latest Firmware release in the Adapter, but we may ask you to hold off on this if you are willing to experiment.
One of the changes in this driver is to negotiate with later firmware some of the maximum characteristics (largest transfer, largest sg table size, largest queue of outstanding commands). These characteristics map to internal adapter limits and can be tuned by the adapter at initialization time. The negotiated values can be changed by the RAID management software to aid in increased performance based on the array requirements.
Earlier firmware without these negotiation would have its transfer size hard coded and increased from 64K to 256K, its sg table size moved from 16 to 34, 38 or 57 depending on controller model and environment, and its queue of outstanding commands increased from 100 to 512. These limits were consulted with the firmware team and tested out in our labs. Nothing is perfect, as we discovered the maximum transfer size had to be reduced recently from 1G to 256K as that corner did not appear to be fully tested (oops). Any one, or combination, of these characteristics could be a trigger for the failure for older firmware without the negotiations.
If you are willing to take some time to explore these limits for the benefit of the community, I would beg you to hold off the step above where I request you to update the firmware (if it is not already) and permit these changes to be explored. We would like to find a combination that 'just' works and back off by a small amount after that.
I understand if you are unwilling to explore these limits on a production server. Upgrading the firmware to the latest may be the easiest and fastest path to resolution.
If you are game, let me demonstrate how to tune the driver (sorry, none of these were made insmod parameters, that was short-sighted of me).
To adjust the outstanding commands tune the value of AAC_NUM_IO_FIB in the .../drivers/scsi/aacraid/aacraid.h file and recompile the driver. Values between (108 - AAC_NUM_MGT_FIB) and (512 - AAC_NUM_MGT_FIB) should be explored.
To adjust the maximum command size, adjust the value of AAC_MAX_32BIT_SGBCOUNT in the aacraid.h file. Values of 256 & 128 should be tried.
To adjust the maximum number of sg elements the AAC_QUIRK_34SG may need to be added in the linit.c file for the 2610 adapter, or the formulas in the aachba.c file will need to be modified (see code fragment below). I was assured that the 2610 was never shipped with firmware that had this 34 sg element hard limit, but for this test, lets cast doubt on that.
/* * 57 scatter gather elements */ dev->scsi_host_ptr->sg_tablesize = (dev->max_fib_size - sizeof(struct aac_fibhdr) - sizeof(struct aac_write) + sizeof(struct sgmap)) / sizeof(struct sgmap); if (dev->dac_support) { /* * 38 scatter gather elements */ dev->scsi_host_ptr->sg_tablesize = (dev->max_fib_size - sizeof(struct aac_fibhdr) - sizeof(struct aac_write64) + sizeof(struct sgmap64)) / sizeof(struct sgmap64); } dev->scsi_host_ptr->max_sectors = AAC_MAX_32BIT_SGBCOUNT; if(!(dev->adapter_info.options & AAC_OPT_NEW_COMM)) { /* * Worst case size that could cause sg overflow when * we break up SG elements that are larger than 64KB. * Would be nice if we could tell the SCSI layer what * the maximum SG element size can be. Worst case is * (sg_tablesize-1) 4KB elements with one 64KB * element. * 32bit -> 468 or 238KB 64bit -> 424 or 212KB */ dev->scsi_host_ptr->max_sectors = (dev->scsi_host_ptr->sg_tablesize * 8) + 112; }
I look forward to your correspondence.
Sincerely -- Mark Salyzyn
-----Original Message----- From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Juan D Ch Sent: Monday, October 03, 2005 5:46 PM To: linux-kernel@vger.kernel.org Subject: 2.6.13.2 aacraid regression
Hi this is my first post in the list. I will try to make this mail short
and useful, sorry but english is not my first language.
I have a client that has a HP Proliant ML 150 G2 with a 6 port SATA RAID Controler Adaptec 2610SA, with HP part number 372953-B21. I tried to install debian, but 2.6.8 kernel doesn't support this controler. So I installed the system in a PATA drive and then compiled the 2.6.13.2 kernel from kernel.org. This kernel recognizes the controler so i formatted the array and start coping data. I copied some dirs in /dev/sda1 and then, after transferring a random amount of data (sometimes a couple of MB, sometimes a few GB) the following messages appear:
------
Sep 30 10:16:59 syrah kernel: aacraid: Host adapter reset request. SCSI hang ? Sep 30 10:16:59 syrah kernel: aacraid: Host adapter appears dead Sep 30 10:16:59 syrah kernel: scsi: Device offlined - not ready after error recovery: host 0 channel 0 id 0 lun 0 Sep 30 10:16:59 syrah last message repeated 2 times Sep 30 10:16:59 syrah kernel: SCSI error : <0 0 0 0> return code = 0x6000000 Sep 30 10:16:59 syrah kernel: end_request: I/O error, dev sda, sector 895 Sep 30 10:16:59 syrah kernel: scsi0 (0:0): rejecting I/O to offline device
-----
When this happens, I need to reboot in order to see the device again.
I tried 2.6.12.6 with the same options and the controller works fine.
As this is a production server it's difficult for me to track down the exact change that introduced this error; so any hints would be highly appreciated.
Thanks a lot
juandie
-- - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |