[PATCH] optimize o_direct on block devices

Implement block device specific .direct_IO method instead of going through
generic direct_io_worker for block device.

direct_io_worker() is fairly complex because it needs to handle O_DIRECT on
file system, where it needs to perform block allocation, hole detection,
extents file on write, and tons of other corner cases.  The end result is
that it takes tons of CPU time to submit an I/O.

For block device, the block allocation is much simpler and a tight triple
loop can be written to iterate each iovec and each page within the iovec in
order to construct/prepare bio structure and then subsequently submit it to
the block layer.  This significantly speeds up O_D on block device.

[akpm@osdl.org: small speedup]
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Zach Brown <zach.brown@oracle.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

2 files changed, 2 insertions, 0 deletions

diff --git a/include/linux/aio.h b/include/linux/aio.h
index 3372ec6bf53a..a30ef13c9e62 100644
--- a/include/linux/aio.h
+++ b/include/linux/aio.h
@@ -105,6 +105,7 @@ struct kiocb {
         wait_queue_t            ki_wait;
         loff_t                  ki_pos;
 
+        atomic_t                ki_bio_count;   /* num bio used for this iocb */
         void                    *private;
         /* State that we remember to be able to restart/retry  */
         unsigned short          ki_opcode;
diff --git a/include/linux/bio.h b/include/linux/bio.h
index 092dbd0e7658..08daf3272c02 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -309,6 +309,7 @@ extern struct bio *bio_map_kern(struct request_queue *, void *, unsigned int,
                                 gfp_t);
 extern void bio_set_pages_dirty(struct bio *bio);
 extern void bio_check_pages_dirty(struct bio *bio);
+extern void bio_release_pages(struct bio *bio);
 extern struct bio *bio_copy_user(struct request_queue *, unsigned long, unsigned int, int);
 extern int bio_uncopy_user(struct bio *);
 void zero_fill_bio(struct bio *bio);