aio: implement request batching

Hi,

Some workloads issue batches of small I/O, and the performance is poor
due to the call to blk_run_address_space for every single iocb.  Nathan
Roberts pointed this out, and suggested that by deferring this call
until all I/Os in the iocb array are submitted to the block layer, we
can realize some impressive performance gains (up to 30% for sequential
4k reads in batches of 16).

Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>

1 files changed, 4 insertions, 4 deletions

diff --git a/fs/direct-io.c b/fs/direct-io.c
index c86d35f142de..3af761c8c5cc 100644
--- a/fs/direct-io.c
+++ b/fs/direct-io.c
@@ -1028,9 +1028,6 @@ direct_io_worker(int rw, struct kiocb *iocb, struct inode *inode,
         if (dio->bio)
                 dio_bio_submit(dio);
 
-        /* All IO is now issued, send it on its way */
-        blk_run_address_space(inode->i_mapping);
-
         /*
          * It is possible that, we return short IO due to end of file.
          * In that case, we need to release all the pages we got hold on.
@@ -1057,8 +1054,11 @@ direct_io_worker(int rw, struct kiocb *iocb, struct inode *inode,
             ((rw & READ) || (dio->result == dio->size)))
                 ret = -EIOCBQUEUED;
 
-        if (ret != -EIOCBQUEUED)
+        if (ret != -EIOCBQUEUED) {
+                /* All IO is now issued, send it on its way */
+                blk_run_address_space(inode->i_mapping);
                 dio_await_completion(dio);
+        }
 
         /*
          * Sync will always be dropping the final ref and completing the