sbitmap: optimize wakeup check

Even if we have no waiters on any of the sbitmap_queue wait states, we
still have to loop every entry to check. We do this for every IO, so
the cost adds up.

Shift a bit of the cost to the slow path, when we actually have waiters.
Wrap prepare_to_wait_exclusive() and finish_wait(), so we can maintain
an internal count of how many are currently active. Then we can simply
check this count in sbq_wake_ptr() and not have to loop if we don't
have any sleepers.

Convert the two users of sbitmap with waiting, blk-mq-tag and iSCSI.

Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

1 files changed, 7 insertions, 5 deletions

diff --git a/drivers/target/iscsi/iscsi_target_util.c b/drivers/target/iscsi/iscsi_target_util.c
index 36b742932c72..86987da86dd6 100644
--- a/drivers/target/iscsi/iscsi_target_util.c
+++ b/drivers/target/iscsi/iscsi_target_util.c
@@ -150,24 +150,26 @@ void iscsit_free_r2ts_from_list(struct iscsi_cmd *cmd)
 static int iscsit_wait_for_tag(struct se_session *se_sess, int state, int *cpup)
 {
         int tag = -1;
-        DEFINE_WAIT(wait);
+        DEFINE_SBQ_WAIT(wait);
         struct sbq_wait_state *ws;
+        struct sbitmap_queue *sbq;
 
         if (state == TASK_RUNNING)
                 return tag;
 
-        ws = &se_sess->sess_tag_pool.ws[0];
+        sbq = &se_sess->sess_tag_pool;
+        ws = &sbq->ws[0];
         for (;;) {
-                prepare_to_wait_exclusive(&ws->wait, &wait, state);
+                sbitmap_prepare_to_wait(sbq, ws, &wait, state);
                 if (signal_pending_state(state, current))
                         break;
-                tag = sbitmap_queue_get(&se_sess->sess_tag_pool, cpup);
+                tag = sbitmap_queue_get(sbq, cpup);
                 if (tag >= 0)
                         break;
                 schedule();
         }
 
-        finish_wait(&ws->wait, &wait);
+        sbitmap_finish_wait(sbq, ws, &wait);
         return tag;
 }