fuse: support writable mmap

Quoting Linus (3 years ago, FUSE inclusion discussions):

  "User-space filesystems are hard to get right. I'd claim that they
   are almost impossible, unless you limit them somehow (shared
   writable mappings are the nastiest part - if you don't have those,
   you can reasonably limit your problems by limiting the number of
   dirty pages you accept through normal "write()" calls)."

Instead of attempting the impossible, I've just waited for the dirty page
accounting infrastructure to materialize (thanks to Peter Zijlstra and
others).  This nicely solved the biggest problem: limiting the number of pages
used for write caching.

Some small details remained, however, which this largish patch attempts to
address.  It provides a page writeback implementation for fuse, which is
completely safe against VM related deadlocks.  Performance may not be very
good for certain usage patterns, but generally it should be acceptable.

It has been tested extensively with fsx-linux and bash-shared-mapping.

Fuse page writeback design
--------------------------

fuse_writepage() allocates a new temporary page with GFP_NOFS|__GFP_HIGHMEM.
It copies the contents of the original page, and queues a WRITE request to the
userspace filesystem using this temp page.

The writeback is finished instantly from the MM's point of view: the page is
removed from the radix trees, and the PageDirty and PageWriteback flags are
cleared.

For the duration of the actual write, the NR_WRITEBACK_TEMP counter is
incremented.  The per-bdi writeback count is not decremented until the actual
write completes.

On dirtying the page, fuse waits for a previous write to finish before
proceeding.  This makes sure, there can only be one temporary page used at a
time for one cached page.

This approach is wasteful in both memory and CPU bandwidth, so why is this
complication needed?

The basic problem is that there can be no guarantee about the time in which
the userspace filesystem will complete a write.  It may be buggy or even
malicious, and fail to complete WRITE requests.  We don't want unrelated parts
of the system to grind to a halt in such cases.

Also a filesystem may need additional resources (particularly memory) to
complete a WRITE request.  There's a great danger of a deadlock if that
allocation may wait for the writepage to finish.

Currently there are several cases where the kernel can block on page
writeback:

  - allocation order is larger than PAGE_ALLOC_COSTLY_ORDER
  - page migration
  - throttle_vm_writeout (through NR_WRITEBACK)
  - sync(2)

Of course in some cases (fsync, msync) we explicitly want to allow blocking.
So for these cases new code has to be added to fuse, since the VM is not
tracking writeback pages for us any more.

As an extra safetly measure, the maximum dirty ratio allocated to a single
fuse filesystem is set to 1% by default.  This way one (or several) buggy or
malicious fuse filesystems cannot slow down the rest of the system by hogging
dirty memory.

With appropriate privileges, this limit can be raised through
'/sys/class/bdi/<bdi>/max_ratio'.

Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 305 insertions, 16 deletions

diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index 676b0bc8a86d..68051f3bdf91 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -210,6 +210,49 @@ u64 fuse_lock_owner_id(struct fuse_conn *fc, fl_owner_t id)
         return (u64) v0 + ((u64) v1 << 32);
 }
 
+/*
+ * Check if page is under writeback
+ *
+ * This is currently done by walking the list of writepage requests
+ * for the inode, which can be pretty inefficient.
+ */
+static bool fuse_page_is_writeback(struct inode *inode, pgoff_t index)
+{
+        struct fuse_conn *fc = get_fuse_conn(inode);
+        struct fuse_inode *fi = get_fuse_inode(inode);
+        struct fuse_req *req;
+        bool found = false;
+
+        spin_lock(&fc->lock);
+        list_for_each_entry(req, &fi->writepages, writepages_entry) {
+                pgoff_t curr_index;
+
+                BUG_ON(req->inode != inode);
+                curr_index = req->misc.write.in.offset >> PAGE_CACHE_SHIFT;
+                if (curr_index == index) {
+                        found = true;
+                        break;
+                }
+        }
+        spin_unlock(&fc->lock);
+
+        return found;
+}
+
+/*
+ * Wait for page writeback to be completed.
+ *
+ * Since fuse doesn't rely on the VM writeback tracking, this has to
+ * use some other means.
+ */
+static int fuse_wait_on_page_writeback(struct inode *inode, pgoff_t index)
+{
+        struct fuse_inode *fi = get_fuse_inode(inode);
+
+        wait_event(fi->page_waitq, !fuse_page_is_writeback(inode, index));
+        return 0;
+}
+
 static int fuse_flush(struct file *file, fl_owner_t id)
 {
         struct inode *inode = file->f_path.dentry->d_inode;
@@ -245,6 +288,21 @@ static int fuse_flush(struct file *file, fl_owner_t id)
         return err;
 }
 
+/*
+ * Wait for all pending writepages on the inode to finish.
+ *
+ * This is currently done by blocking further writes with FUSE_NOWRITE
+ * and waiting for all sent writes to complete.
+ *
+ * This must be called under i_mutex, otherwise the FUSE_NOWRITE usage
+ * could conflict with truncation.
+ */
+static void fuse_sync_writes(struct inode *inode)
+{
+        fuse_set_nowrite(inode);
+        fuse_release_nowrite(inode);
+}
+
 int fuse_fsync_common(struct file *file, struct dentry *de, int datasync,
                       int isdir)
 {
@@ -261,6 +319,17 @@ int fuse_fsync_common(struct file *file, struct dentry *de, int datasync,
         if ((!isdir && fc->no_fsync) || (isdir && fc->no_fsyncdir))
                 return 0;
 
+        /*
+         * Start writeback against all dirty pages of the inode, then
+         * wait for all outstanding writes, before sending the FSYNC
+         * request.
+         */
+        err = write_inode_now(inode, 0);
+        if (err)
+                return err;
+
+        fuse_sync_writes(inode);
+
         req = fuse_get_req(fc);
         if (IS_ERR(req))
                 return PTR_ERR(req);
@@ -340,6 +409,13 @@ static int fuse_readpage(struct file *file, struct page *page)
         if (is_bad_inode(inode))
                 goto out;
 
+        /*
+         * Page writeback can extend beyond the liftime of the
+         * page-cache page, so make sure we read a properly synced
+         * page.
+         */
+        fuse_wait_on_page_writeback(inode, page->index);
+
         req = fuse_get_req(fc);
         err = PTR_ERR(req);
         if (IS_ERR(req))
@@ -411,6 +487,8 @@ static int fuse_readpages_fill(void *_data, struct page *page)
         struct inode *inode = data->inode;
         struct fuse_conn *fc = get_fuse_conn(inode);
 
+        fuse_wait_on_page_writeback(inode, page->index);
+
         if (req->num_pages &&
             (req->num_pages == FUSE_MAX_PAGES_PER_REQ ||
              (req->num_pages + 1) * PAGE_CACHE_SIZE > fc->max_read ||
@@ -477,11 +555,10 @@ static ssize_t fuse_file_aio_read(struct kiocb *iocb, const struct iovec *iov,
 }
 
 static void fuse_write_fill(struct fuse_req *req, struct file *file,
-                            struct inode *inode, loff_t pos, size_t count,
-                            int writepage)
+                            struct fuse_file *ff, struct inode *inode,
+                            loff_t pos, size_t count, int writepage)
 {
         struct fuse_conn *fc = get_fuse_conn(inode);
-        struct fuse_file *ff = file->private_data;
         struct fuse_write_in *inarg = &req->misc.write.in;
         struct fuse_write_out *outarg = &req->misc.write.out;
 
@@ -490,7 +567,7 @@ static void fuse_write_fill(struct fuse_req *req, struct file *file,
         inarg->offset = pos;
         inarg->size = count;
         inarg->write_flags = writepage ? FUSE_WRITE_CACHE : 0;
-        inarg->flags = file->f_flags;
+        inarg->flags = file ? file->f_flags : 0;
         req->in.h.opcode = FUSE_WRITE;
         req->in.h.nodeid = get_node_id(inode);
         req->in.argpages = 1;
@@ -511,7 +588,7 @@ static size_t fuse_send_write(struct fuse_req *req, struct file *file,
                               fl_owner_t owner)
 {
         struct fuse_conn *fc = get_fuse_conn(inode);
-        fuse_write_fill(req, file, inode, pos, count, 0);
+        fuse_write_fill(req, file, file->private_data, inode, pos, count, 0);
         if (owner != NULL) {
                 struct fuse_write_in *inarg = &req->misc.write.in;
                 inarg->write_flags |= FUSE_WRITE_LOCKOWNER;
@@ -546,6 +623,12 @@ static int fuse_buffered_write(struct file *file, struct inode *inode,
         if (is_bad_inode(inode))
                 return -EIO;
 
+        /*
+         * Make sure writepages on the same page are not mixed up with
+         * plain writes.
+         */
+        fuse_wait_on_page_writeback(inode, page->index);
+
         req = fuse_get_req(fc);
         if (IS_ERR(req))
                 return PTR_ERR(req);
@@ -716,21 +799,225 @@ static ssize_t fuse_direct_write(struct file *file, const char __user *buf,
         return res;
 }
 
-static int fuse_file_mmap(struct file *file, struct vm_area_struct *vma)
+static void fuse_writepage_free(struct fuse_conn *fc, struct fuse_req *req)
 {
-        if ((vma->vm_flags & VM_SHARED)) {
-                if ((vma->vm_flags & VM_WRITE))
-                        return -ENODEV;
-                else
-                        vma->vm_flags &= ~VM_MAYWRITE;
+        __free_page(req->pages[0]);
+        fuse_file_put(req->ff);
+        fuse_put_request(fc, req);
+}
+
+static void fuse_writepage_finish(struct fuse_conn *fc, struct fuse_req *req)
+{
+        struct inode *inode = req->inode;
+        struct fuse_inode *fi = get_fuse_inode(inode);
+        struct backing_dev_info *bdi = inode->i_mapping->backing_dev_info;
+
+        list_del(&req->writepages_entry);
+        dec_bdi_stat(bdi, BDI_WRITEBACK);
+        dec_zone_page_state(req->pages[0], NR_WRITEBACK_TEMP);
+        bdi_writeout_inc(bdi);
+        wake_up(&fi->page_waitq);
+}
+
+/* Called under fc->lock, may release and reacquire it */
+static void fuse_send_writepage(struct fuse_conn *fc, struct fuse_req *req)
+{
+        struct fuse_inode *fi = get_fuse_inode(req->inode);
+        loff_t size = i_size_read(req->inode);
+        struct fuse_write_in *inarg = &req->misc.write.in;
+
+        if (!fc->connected)
+                goto out_free;
+
+        if (inarg->offset + PAGE_CACHE_SIZE <= size) {
+                inarg->size = PAGE_CACHE_SIZE;
+        } else if (inarg->offset < size) {
+                inarg->size = size & (PAGE_CACHE_SIZE - 1);
+        } else {
+                /* Got truncated off completely */
+                goto out_free;
         }
-        return generic_file_mmap(file, vma);
+
+        req->in.args[1].size = inarg->size;
+        fi->writectr++;
+        request_send_background_locked(fc, req);
+        return;
+
+ out_free:
+        fuse_writepage_finish(fc, req);
+        spin_unlock(&fc->lock);
+        fuse_writepage_free(fc, req);
+        spin_lock(&fc->lock);
 }
 
-static int fuse_set_page_dirty(struct page *page)
+/*
+ * If fi->writectr is positive (no truncate or fsync going on) send
+ * all queued writepage requests.
+ *
+ * Called with fc->lock
+ */
+void fuse_flush_writepages(struct inode *inode)
 {
-        printk("fuse_set_page_dirty: should not happen\n");
-        dump_stack();
+        struct fuse_conn *fc = get_fuse_conn(inode);
+        struct fuse_inode *fi = get_fuse_inode(inode);
+        struct fuse_req *req;
+
+        while (fi->writectr >= 0 && !list_empty(&fi->queued_writes)) {
+                req = list_entry(fi->queued_writes.next, struct fuse_req, list);
+                list_del_init(&req->list);
+                fuse_send_writepage(fc, req);
+        }
+}
+
+static void fuse_writepage_end(struct fuse_conn *fc, struct fuse_req *req)
+{
+        struct inode *inode = req->inode;
+        struct fuse_inode *fi = get_fuse_inode(inode);
+
+        mapping_set_error(inode->i_mapping, req->out.h.error);
+        spin_lock(&fc->lock);
+        fi->writectr--;
+        fuse_writepage_finish(fc, req);
+        spin_unlock(&fc->lock);
+        fuse_writepage_free(fc, req);
+}
+
+static int fuse_writepage_locked(struct page *page)
+{
+        struct address_space *mapping = page->mapping;
+        struct inode *inode = mapping->host;
+        struct fuse_conn *fc = get_fuse_conn(inode);
+        struct fuse_inode *fi = get_fuse_inode(inode);
+        struct fuse_req *req;
+        struct fuse_file *ff;
+        struct page *tmp_page;
+
+        set_page_writeback(page);
+
+        req = fuse_request_alloc_nofs();
+        if (!req)
+                goto err;
+
+        tmp_page = alloc_page(GFP_NOFS | __GFP_HIGHMEM);
+        if (!tmp_page)
+                goto err_free;
+
+        spin_lock(&fc->lock);
+        BUG_ON(list_empty(&fi->write_files));
+        ff = list_entry(fi->write_files.next, struct fuse_file, write_entry);
+        req->ff = fuse_file_get(ff);
+        spin_unlock(&fc->lock);
+
+        fuse_write_fill(req, NULL, ff, inode, page_offset(page), 0, 1);
+
+        copy_highpage(tmp_page, page);
+        req->num_pages = 1;
+        req->pages[0] = tmp_page;
+        req->page_offset = 0;
+        req->end = fuse_writepage_end;
+        req->inode = inode;
+
+        inc_bdi_stat(mapping->backing_dev_info, BDI_WRITEBACK);
+        inc_zone_page_state(tmp_page, NR_WRITEBACK_TEMP);
+        end_page_writeback(page);
+
+        spin_lock(&fc->lock);
+        list_add(&req->writepages_entry, &fi->writepages);
+        list_add_tail(&req->list, &fi->queued_writes);
+        fuse_flush_writepages(inode);
+        spin_unlock(&fc->lock);
+
+        return 0;
+
+err_free:
+        fuse_request_free(req);
+err:
+        end_page_writeback(page);
+        return -ENOMEM;
+}
+
+static int fuse_writepage(struct page *page, struct writeback_control *wbc)
+{
+        int err;
+
+        err = fuse_writepage_locked(page);
+        unlock_page(page);
+
+        return err;
+}
+
+static int fuse_launder_page(struct page *page)
+{
+        int err = 0;
+        if (clear_page_dirty_for_io(page)) {
+                struct inode *inode = page->mapping->host;
+                err = fuse_writepage_locked(page);
+                if (!err)
+                        fuse_wait_on_page_writeback(inode, page->index);
+        }
+        return err;
+}
+
+/*
+ * Write back dirty pages now, because there may not be any suitable
+ * open files later
+ */
+static void fuse_vma_close(struct vm_area_struct *vma)
+{
+        filemap_write_and_wait(vma->vm_file->f_mapping);
+}
+
+/*
+ * Wait for writeback against this page to complete before allowing it
+ * to be marked dirty again, and hence written back again, possibly
+ * before the previous writepage completed.
+ *
+ * Block here, instead of in ->writepage(), so that the userspace fs
+ * can only block processes actually operating on the filesystem.
+ *
+ * Otherwise unprivileged userspace fs would be able to block
+ * unrelated:
+ *
+ * - page migration
+ * - sync(2)
+ * - try_to_free_pages() with order > PAGE_ALLOC_COSTLY_ORDER
+ */
+static int fuse_page_mkwrite(struct vm_area_struct *vma, struct page *page)
+{
+        /*
+         * Don't use page->mapping as it may become NULL from a
+         * concurrent truncate.
+         */
+        struct inode *inode = vma->vm_file->f_mapping->host;
+
+        fuse_wait_on_page_writeback(inode, page->index);
+        return 0;
+}
+
+static struct vm_operations_struct fuse_file_vm_ops = {
+        .close          = fuse_vma_close,
+        .fault          = filemap_fault,
+        .page_mkwrite   = fuse_page_mkwrite,
+};
+
+static int fuse_file_mmap(struct file *file, struct vm_area_struct *vma)
+{
+        if ((vma->vm_flags & VM_SHARED) && (vma->vm_flags & VM_MAYWRITE)) {
+                struct inode *inode = file->f_dentry->d_inode;
+                struct fuse_conn *fc = get_fuse_conn(inode);
+                struct fuse_inode *fi = get_fuse_inode(inode);
+                struct fuse_file *ff = file->private_data;
+                /*
+                 * file may be written through mmap, so chain it onto the
+                 * inodes's write_file list
+                 */
+                spin_lock(&fc->lock);
+                if (list_empty(&ff->write_entry))
+                        list_add(&ff->write_entry, &fi->write_files);
+                spin_unlock(&fc->lock);
+        }
+        file_accessed(file);
+        vma->vm_ops = &fuse_file_vm_ops;
         return 0;
 }
 
@@ -940,10 +1227,12 @@ static const struct file_operations fuse_direct_io_file_operations = {
 
 static const struct address_space_operations fuse_file_aops  = {
         .readpage       = fuse_readpage,
+        .writepage      = fuse_writepage,
+        .launder_page   = fuse_launder_page,
         .write_begin    = fuse_write_begin,
         .write_end      = fuse_write_end,
         .readpages      = fuse_readpages,
-        .set_page_dirty = fuse_set_page_dirty,
+        .set_page_dirty = __set_page_dirty_nobuffers,
         .bmap           = fuse_bmap,
 };