[Bug 85101] New: hpsa + P410 does not show connected HP SAS port expanders

[b***@bugzilla.kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

Bug ID: 85101
Summary: hpsa + P410 does not show connected HP SAS port
expanders
Product: SCSI Drivers
Version: 2.5
Kernel Version: 3.17
Hardware: x86-64
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
Assignee: scsi_drivers-***@kernel-bugs.osdl.org
Reporter: linux-***@crashplan.pro
Regression: No

ASRock H81 + Intel Celeron G1840
HP P410, firmware 6.60
HP SAS Expander, firmware 2.08

In the need for upgrading the firmware on the HP SAS expanders, the HP P410 is
attached to this setup. However it does not see the any of the two connected HP
SAS expander cards, only the boot drive and P410 are shown in the lsscsi
output.

Steps to reproduce:
1. blacklist hpsa
2. reboot
3. modprobe -v hpsa
4. lsscsi

# lsscsi
[0:0:0:0] disk ATA OCZ-VERTEX 1.3 /dev/sda
[4:3:0:0] storage HP P410 6.60 -

# modprobe -v hpsa dmesg:
[ 49.567169] HP HPSA Driver (v 3.4.4-1)
[ 49.567251] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[ 49.567389] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[ 49.567576] hpsa 0000:01:00.0: MSIX
[ 49.567649] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[ 49.567705] hpsa 0000:01:00.0: irq 31 for MSI/MSI-X
[ 49.567799] hpsa 0000:01:00.0: Logical aborts not supported
[ 49.587966] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 30 using DAC
[ 55.125143] scsi host4: hpsa
[ 55.129439] hpsa 0000:01:00.0: RAID device c4b3t0l0 added.
[ 55.129832] scsi 4:3:0:0: RAID HP P410 6.60
PQ: 0 ANSI: 5
[ 55.130585] scsi 4:3:0:0: Attached scsi generic sg1 type 12

Regression:
1. changed back to kernel 3.13: no difference
2. disconnect all other SAS (to SATA) cables: no difference
3. attach only one HP SAS expander: no difference
4. switch cable between P410 and SAS expander card: no differnce
5. replace HP P410 (and cables) with Adapter 7805H or Marvell 88SE9485 SAS/SATA
6Gb/s controller [1b4b:9485]: HP SAS expanders are shown

[b***@bugzilla.kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

Robert Elliott <***@hp.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@hp.com

--- Comment #1 from Robert Elliott <***@hp.com> ---
Smart Array P410 is a RAID controller that does not expose anything about the
SAS fabric (expanders or physical drives) to the OS; it just presents logical
drives. You need to use HP management tools to download firmware to expanders
and drives behind the controller.

If you attach the expander to an HBA, then you can use sg_write_buffer to
download microcode. Version 1.39 added a --bpw option to download in small
chunks - an expander might require a small chunk size like 4 KiB (see
http://sg.danny.cz/sg for the latest sg3_utils code).

Although they sound like DOS programs, .scexe files from HP are really linux
executable scripts that you can view in a text editor (at least the first
part). You can do this to extract the files:
mkdir tmp
./CPxyzzy.scexe --unpack=tmp

One of those is the binary image that can be sent with sg_write_buffer.

BTW, use lsscsi -g to see the /dev/sgNN paths for all the devices presented by
the LLD, including expanders.

[b***@bugzilla.kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

--- Comment #2 from linux-***@crashplan.pro ---
In this case the hardware seems broken.

The P410 controller is only used for upgrading the HP SAS expander firmware.
This time we started with upgrading the controller firmware:
===
./ccissflash

This program consists of two phases: device discovery and device update.
No device will be updated until you answer.

Do you want to run device discovery?
(yes/no) yes
Finding hardware. This may take a few minutes.
Found 1 devices.

Do you want to upgrade the device that has older ROM?
(yes/no) yes
1 devices will be updated.
Updating: P410 Slot: 2 from [5.70] to [6.60]
Updating: P410 Slot: 2 from [5.70] to [6.60]

As part of the reboot process, you must power cycle the server and any external
array storage devices.
===

After power cycling the first three reboots were ok. Then a controller lockup
0x00130000 occurred 468 seconds after loading the hpsa module. Since then only
one boot/module loading was ok and since then the "time outs" are occurring:

===
[ 121.613269] HP HPSA Driver (v 3.4.0-1)
[ 121.613344] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[ 121.613447] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[ 121.613669] hpsa 0000:01:00.0: MSIX
[ 121.613770] hpsa 0000:01:00.0: irq 46 for MSI/MSI-X
[ 121.613843] hpsa 0000:01:00.0: irq 47 for MSI/MSI-X
[ 121.613909] hpsa 0000:01:00.0: irq 48 for MSI/MSI-X
[ 121.613981] hpsa 0000:01:00.0: irq 49 for MSI/MSI-X
[ 121.614046] hpsa 0000:01:00.0: irq 50 for MSI/MSI-X
[ 121.614117] hpsa 0000:01:00.0: irq 51 for MSI/MSI-X
[ 121.614182] hpsa 0000:01:00.0: irq 52 for MSI/MSI-X
[ 121.614251] hpsa 0000:01:00.0: irq 53 for MSI/MSI-X
[ 121.634560] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 46 using DAC
[ 127.171906] scsi4 : hpsa
[ 127.174883] hpsa 0000:01:00.0: RAID device c4b3t0l0 added.
[ 127.175961] scsi 4:3:0:0: RAID HP P410 6.60
PQ: 0 ANSI: 5
[ 127.176335] scsi 4:3:0:0: Attached scsi generic sg1 type 12
[ 424.473998] HP HPSA Driver (v 3.4.0-1)
===
[ 16.127857] HP HPSA Driver (v 3.4.0-1)
[ 16.127927] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[ 16.128020] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[ 16.128231] hpsa 0000:01:00.0: MSIX
[ 16.128327] hpsa 0000:01:00.0: irq 46 for MSI/MSI-X
[ 16.128394] hpsa 0000:01:00.0: irq 47 for MSI/MSI-X
[ 16.128460] hpsa 0000:01:00.0: irq 48 for MSI/MSI-X
[ 16.128524] hpsa 0000:01:00.0: irq 49 for MSI/MSI-X
[ 16.128590] hpsa 0000:01:00.0: irq 50 for MSI/MSI-X
[ 16.128660] hpsa 0000:01:00.0: irq 51 for MSI/MSI-X
[ 16.128726] hpsa 0000:01:00.0: irq 52 for MSI/MSI-X
[ 16.128791] hpsa 0000:01:00.0: irq 53 for MSI/MSI-X
[ 16.149110] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 46 using DAC
[ 34.767566] scsi4 : hpsa
[ 34.770583] hpsa 0000:01:00.0: RAID device c4b3t0l0 added.
[ 34.771054] scsi 4:3:0:0: RAID HP P410 6.60
PQ: 0 ANSI: 5
===
[ 48.389920] HP HPSA Driver (v 3.4.4-1)
[ 48.389985] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[ 48.390059] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[ 48.390239] hpsa 0000:01:00.0: MSIX
[ 48.390312] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[ 48.390362] hpsa 0000:01:00.0: irq 31 for MSI/MSI-X
[ 48.390453] hpsa 0000:01:00.0: Logical aborts not supported
[ 48.410617] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 30 using DAC
[ 53.947809] scsi host4: hpsa
[ 53.953488] hpsa 0000:01:00.0: RAID device c4b3t0l0 added.
[ 53.953986] scsi 4:3:0:0: RAID HP P410 6.60
PQ: 0 ANSI: 5
===
[ 17.076307] HP HPSA Driver (v 3.4.4-1)
[ 17.076388] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[ 17.076483] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[ 17.076695] hpsa 0000:01:00.0: MSIX
[ 17.076782] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[ 17.076837] hpsa 0000:01:00.0: irq 31 for MSI/MSI-X
[ 17.076942] hpsa 0000:01:00.0: Logical aborts not supported
[ 17.097108] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 30 using DAC
[ 33.659110] scsi host4: hpsa
[ 33.663958] hpsa 0000:01:00.0: RAID device c4b3t0l0 added.
[ 33.664360] scsi 4:3:0:0: RAID HP P410 6.60
PQ: 0 ANSI: 5
[ 33.665053] scsi 4:3:0:0: Attached scsi generic sg1 type 12
[ 485.066696] hpsa 0000:01:00.0: Controller lockup detected: 0x00130000
===
[ 49.567169] HP HPSA Driver (v 3.4.4-1)
[ 49.567251] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[ 49.567389] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[ 49.567576] hpsa 0000:01:00.0: MSIX
[ 49.567649] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[ 49.567705] hpsa 0000:01:00.0: irq 31 for MSI/MSI-X
[ 49.567799] hpsa 0000:01:00.0: Logical aborts not supported
[ 49.587966] hpsa 0000:01:00.0: hpsa0: <0x323a> at IRQ 30 using DAC
[ 55.125143] scsi host4: hpsa
[ 55.129439] hpsa 0000:01:00.0: RAID device c4b3t0l0 added.
[ 55.129832] scsi 4:3:0:0: RAID HP P410 6.60
PQ: 0 ANSI: 5
===
[ 38.305788] HP HPSA Driver (v 3.4.4-1)
[ 38.305854] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[ 38.306052] hpsa 0000:01:00.0: MSIX
[ 38.306127] hpsa 0000:01:00.0: irq 29 for MSI/MSI-X
[ 38.306179] hpsa 0000:01:00.0: irq 30 for MSI/MSI-X
[ 163.163631] hpsa 0000:01:00.0: board not ready, timed out.
===
[ 23.324890] HP HPSA Driver (v 3.4.0-1)
[ 23.324957] hpsa 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM
control
[ 23.325060] hpsa 0000:01:00.0: enabling device (0000 -> 0003)
[ 23.325281] hpsa 0000:01:00.0: MSIX
[ 23.325379] hpsa 0000:01:00.0: irq 45 for MSI/MSI-X
[ 23.325453] hpsa 0000:01:00.0: irq 46 for MSI/MSI-X
[ 23.325518] hpsa 0000:01:00.0: irq 47 for MSI/MSI-X
[ 23.325591] hpsa 0000:01:00.0: irq 48 for MSI/MSI-X
[ 23.325655] hpsa 0000:01:00.0: irq 49 for MSI/MSI-X
[ 23.325728] hpsa 0000:01:00.0: irq 50 for MSI/MSI-X
[ 23.325793] hpsa 0000:01:00.0: irq 51 for MSI/MSI-X
[ 23.325865] hpsa 0000:01:00.0: irq 52 for MSI/MSI-X
[ 148.185212] hpsa 0000:01:00.0: board not ready, timed out.
===

Right now loading the hpsa module permanently errors out with "board not ready,
timed out" and the "DS1: Diagnostics Error" LED turns Amber (=One of the server
diagnostics utilities has detected a controller error.)

Seems like a hardware issue, therefore this case can be closed.

[b***@bugzilla.kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

--- Comment #3 from Robert Elliott <***@hp.com> ---
Controller lockup means the firmware crashed (e.g. an ASSERT statement
triggered). That can be due to hardware problems or firmware bugs. You might
see if it boots with nothing attached; then with one expander; then with two;
then with drives attached to the expanders.

If you're not really using the P410, though, just use sg_write_buffer through
the Adaptec 7805H HBA.

[b***@bugzilla.kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

--- Comment #4 from linux-***@crashplan.pro ---
@Robert Elliott
Thanks for all your help.
The issue turns out not to be the HP P410 controller.
Using another mainboard the same HP P410 controller is ok.

[b***@bugzilla.kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

linux-***@crashplan.pro changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |UNREPRODUCIBLE
Severity|normal |low

[b***@bugzilla.kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

linux-***@crashplan.pro changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|UNREPRODUCIBLE |---

--- Comment #5 from linux-***@crashplan.pro ---
After replacing the faulty hardware, the HP P410 boots normally again.

However the P410+hpsa still does not detect either of the HP SAS expander
cards, neither does ./ccissflash detect any connected HP SAS expander.

LED status of the HP SAS expander card:
CR1 = solid green
CR2 = solid green
CR3 = solid green
CR4 = off
CR5 = off
CR6 = blinks green

Note: the Adaptec pm80xx module fails loading in sas expander discovery, I will
file a separate bug report for that.

[b***@bugzilla.kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

--- Comment #6 from linux-***@crashplan.pro ---
The HP Smart Array P410 still has its issues: sometimes the diagnostics LED
turns amber, other times it works ok.

As a fall back a Marvell controller detects the HP SAS expander card fine:
# lsscsi -g | grep HP
[6:0:0:0] enclosu HP HP SAS EXP Card 2.08 - /dev/sg1

I am not able to figure out a correct sg_write_buffer command. Run from the
directory where the unpacked binary image is stored:
# sg_write_buffer --bpw=4096 -v --in=PUF21000.bin /dev/sg1
tried to read 8388608 bytes from PUF21000.bin, got 1509632 bytes
will write 1509632 bytes
sending write buffer, mode=0x0, mspec=0, id=0, offset=0, len=4096
Write buffer cmd: 3b 00 00 00 00 00 00 10 00 00
write buffer: Fixed format, current; Sense key: Illegal Request
Additional sense: Invalid field in cdb
Write buffer failed: Illegal request sense key, apart from Invalid opcode

Any suggestions for the correct sg_write_buffer statement for writing to an HP
SAS expander card?

[b***@bugzilla.kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

--- Comment #7 from Robert Elliott <***@hp.com> ---
You need to specify a download microcode mode. Using deferred activation, the
most modern approach:

1. sg_write_buffer --mode=dmc_offs_defer --bpw=4096 --in=xxx.bin /dev/sgXX
2. sg_write_buffer --mode=activate_mc /dev/sgXX

Older devices might only support mode 0x7.

[b***@bugzilla.kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

--- Comment #8 from linux-***@crashplan.pro ---
Confirmation of successful firmware download/update/flash to the HP SAS
expander card using the commands:
# sg_write_buffer --mode=dmc_offs_defer --bpw=4096 --in=PUF21000.bin /dev/sg1
# sg_write_buffer --mode=activate_mc /dev/sg1
# reboot

# lsscsi -g | grep HP
[6:0:0:0] enclosu HP HP SAS EXP Card 2.10 - /dev/sg1

[b***@bugzilla.kernel.org]

https://bugzilla.kernel.org/show_bug.cgi?id=85101

linux-***@crashplan.pro changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|REOPENED |RESOLVED
Resolution|--- |DOCUMENTED

--- Comment #9 from linux-***@crashplan.pro ---
This is expected behavior and not a bug because the HP P410 device is a SAS
RAID controller and not a serial attached scsi host bus adapter (SAS HBA).

A SAS RAID controller
quote: "does not expose anything about the SAS fabric (expanders or physical
drives) to the OS"