[PATCH] scsi-mq: fix hw queue hang caused by timeout

[Ming Lei]

If there are two requests or more timed out, the dispatch queue
is put into stopped state and never be recoverd, and there
is no such problem in non-mq mode.

This patch trys to recover the stopped queue when the queue
becomes unbusy, then the following retries can move on.

Basically this patch maintains same behavior for this situation
with non-mq mode.

Signed-off-by: Ming Lei <***@canonical.com>
---
drivers/scsi/scsi_lib.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 3b92c39..dfbc028 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -142,6 +142,8 @@ static void __scsi_queue_insert(struct scsi_cmnd *cmd, int reason, int unbusy)
struct scsi_device *device = cmd->device;
struct request_queue *q = device->request_queue;
unsigned long flags;
+ bool restart = false;
+ bool blocked = !!atomic_read(&device->device_blocked);

SCSI_LOG_MLQUEUE(1, scmd_printk(KERN_INFO, cmd,
"Inserting command %p into mlqueue\n", cmd));
@@ -152,9 +154,14 @@ static void __scsi_queue_insert(struct scsi_cmnd *cmd, int reason, int unbusy)
* Decrement the counters, since these commands are no longer
* active on the host/device.
*/
- if (unbusy)
+ if (unbusy) {
scsi_device_unbusy(device);

+ /* need to restart hw queue if it was stopped */
+ if (!atomic_read(&device->device_busy) && blocked)
+ restart = true;
+ }
+
/*
* Requeue this command. It will go before all other commands
* that are already in the queue. Schedule requeue work under
@@ -164,6 +171,8 @@ static void __scsi_queue_insert(struct scsi_cmnd *cmd, int reason, int unbusy)
cmd->result = 0;
if (q->mq_ops) {
scsi_mq_requeue_cmd(cmd);
+ if (restart)
+ blk_mq_start_stopped_hw_queues(q, true);
return;
}
spin_lock_irqsave(q->queue_lock, flags);

[Christoph Hellwig]

Post by Ming Lei
If there are two requests or more timed out, the dispatch queue
is put into stopped state and never be recoverd, and there
is no such problem in non-mq mode.
This patch trys to recover the stopped queue when the queue
becomes unbusy, then the following retries can move on.
Basically this patch maintains same behavior for this situation
with non-mq mode.

This looks somewhat similar to the issues that Doug reported, and I remember
when he was last running into boot problems it was timeout related, too.

As far as the implementation is concerned I think the correct fix is
to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list.

[Jens Axboe]

Post by Christoph Hellwig

Post by Ming Lei
If there are two requests or more timed out, the dispatch queue
is put into stopped state and never be recoverd, and there
is no such problem in non-mq mode.
This patch trys to recover the stopped queue when the queue
becomes unbusy, then the following retries can move on.
Basically this patch maintains same behavior for this situation
with non-mq mode.

This looks somewhat similar to the issues that Doug reported, and I remember
when he was last running into boot problems it was timeout related, too.
As far as the implementation is concerned I think the correct fix is
to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list.

Since that's the kick part of the requeue, auto-starting the queue for
that makes a lot of sense. I say that's the way we go.

[Ming Lei]

Post by Christoph Hellwig

Post by Ming Lei
If there are two requests or more timed out, the dispatch queue
is put into stopped state and never be recoverd, and there
is no such problem in non-mq mode.
This patch trys to recover the stopped queue when the queue
becomes unbusy, then the following retries can move on.
Basically this patch maintains same behavior for this situation
with non-mq mode.

This looks somewhat similar to the issues that Doug reported, and I remember
when he was last running into boot problems it was timeout related, too.
As far as the implementation is concerned I think the correct fix is
to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list.

Since that's the kick part of the requeue, auto-starting the queue for that
makes a lot of sense. I say that's the way we go.

Yeah, that looks better.

But it doesn't work after the simple change, and I need to
investigate further.

Thanks,

[Ming Lei]

Post by Ming Lei

Post by Christoph Hellwig

Post by Ming Lei
If there are two requests or more timed out, the dispatch queue
is put into stopped state and never be recoverd, and there
is no such problem in non-mq mode.
This patch trys to recover the stopped queue when the queue
becomes unbusy, then the following retries can move on.
Basically this patch maintains same behavior for this situation
with non-mq mode.

This looks somewhat similar to the issues that Doug reported, and I remember
when he was last running into boot problems it was timeout related, too.
As far as the implementation is concerned I think the correct fix is
to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list.

Since that's the kick part of the requeue, auto-starting the queue for that
makes a lot of sense. I say that's the way we go.

Yeah, that looks better.
But it doesn't work after the simple change, and I need to
investigate further.

It is because of the timer miss, now it starts to work.

Thanks,

[Jens Axboe]

Post by Ming Lei

Post by Ming Lei

Post by Christoph Hellwig

Post by Ming Lei
If there are two requests or more timed out, the dispatch queue
is put into stopped state and never be recoverd, and there
is no such problem in non-mq mode.
This patch trys to recover the stopped queue when the queue
becomes unbusy, then the following retries can move on.
Basically this patch maintains same behavior for this situation
with non-mq mode.

This looks somewhat similar to the issues that Doug reported, and I remember
when he was last running into boot problems it was timeout related, too.
As far as the implementation is concerned I think the correct fix is
to clear the BLK_MQ_S_STOPPED queue flags in blk_mq_kick_requeue_list.

Since that's the kick part of the requeue, auto-starting the queue for that
makes a lot of sense. I say that's the way we go.

Yeah, that looks better.
But it doesn't work after the simple change, and I need to
investigate further.

It is because of the timer miss, now it starts to work.

Excellent. I think most new issues should be fixed in for-linus for
inclusion in this round. It's much bigger than I hoped for this late in
the cycle, but lots of us have run a lot of testing, so that's not a
huge worry.