aboutsummaryrefslogtreecommitdiffstats
path: root/block/cfq-iosched.c
Commit message (Collapse)AuthorAge
* [PATCH] completions: lockdep annotate on stack completionsPeter Zijlstra2006-10-01
| | | | | | | | | | | All on stack DECLARE_COMPLETIONs should be replaced by: DECLARE_COMPLETION_ONSTACK Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Acked-by: Ingo Molnar <mingo@elte.hu> Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* [PATCH] Update axboe@suse.de email addressJens Axboe2006-09-30
| | | | | | | As people often look for the copyright in files to see who to mail, update the link to a neutral one. Signed-off-by: Jens Axboe <axboe@kernel.dk>
* [PATCH] cfq-iosched: use metadata read flagJens Axboe2006-09-30
| | | | | | | | Give meta data reads preference over regular reads, as the process often needs to get that out of the way to do the io it was actually interested in. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: improve queue preemptionJens Axboe2006-09-30
| | | | | | | Don't touch the current queues, just make sure that the wanted queue is selected next. Simplifies the logic. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] Add blk_start_queueing() helperJens Axboe2006-09-30
| | | | | | | | | | CFQ implements this on its own now, but it's really block layer knowledge. Tells a device queue to start dispatching requests to the driver, taking care to unplug if needed. Also fixes the issue where as/cfq will invoke a stopped queue, which we really don't want. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: kill the empty_listJens Axboe2006-09-30
| | | | | | | No point in having a place holder list just for empty queues, so remove it. It's not used for anything other than to keep ->cfq_list busy. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: Kill O(N) runtime of cfq_resort_rr_list()Jens Axboe2006-09-30
| | | | | | | | | Currently it scales with number of processes in that priority group, which is potentially not very nice as it's called quite often. Basically we always need to do tail inserts, except for the case of a new process. So just mark/detect a queue as such. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] Make sure all block/io scheduler setups are node awareJens Axboe2006-09-30
| | | | | | | Some were kmalloc_node(), some were still kmalloc(). Change them all to kmalloc_node(). Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] Audit block layer inlinesJens Axboe2006-09-30
| | | | | | | Kill a few inlines that bring in too much code to more than one location Shrinks kernel text by about 300 bytes on 32-bit x86. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: use new io context counting mechanismJens Axboe2006-09-30
| | | | | | | | It's ok if the read path is a lot more costly, as long as inc/dec is really cheap. The inc/dec will happen for each created/freed io context, while the reading only happens when a disk queue exits. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: kill cfq_exit_lockJens Axboe2006-09-30
| | | | | | | | | | | | | | | | | | | | | cfq_exit_lock is protecting two things now: - The per-ioc rbtree of cfq_io_contexts - The per-cfqd linked list of cfq_io_contexts The per-cfqd linked list can be protected by the queue lock, as it is (by definition) per cfqd as the queue lock is. The per-ioc rbtree is mainly used and updated by the process itself only. The only outside use is the io priority changing. If we move the priority changing to not browsing the rbtree, we can remove any locking from the rbtree updates and lookup completely. Let the sys_ioprio syscall just mark processes as having the iopriority changed and lazily update the private cfq io contexts the next time io is queued, and we can remove this locking as well. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: cleanups, fixes, dead code removalJens Axboe2006-09-30
| | | | | | | | | | | | | | | | | | | | | | | | | | A collection of little fixes and cleanups: - We don't use the 'queued' sysfs exported attribute, since the may_queue() logic was rewritten. So kill it. - Remove dead defines. - cfq_set_active_queue() can be rewritten cleaner with else if conditions. - Several places had cfq_exit_cfqq() like logic, abstract that out and use that. - Annotate the cfqq kmem_cache_alloc() so the allocator knows that this is a repeat allocation if it fails with __GFP_WAIT set. Allows the allocator to start freeing some memory, if needed. CFQ already loops for this condition, so might as well pass the hint down. - Remove cfqd->rq_starved logic. It's not needed anymore after we dropped the crq allocation in cfq_set_request(). - Remove uneeded parameter passing. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] Drop useless bio passing in may_queue/set_request APIJens Axboe2006-09-30
| | | | | | It's not needed for anything, so kill the bio passing. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: kill crqJens Axboe2006-09-30
| | | | | | | | Get rid of the cfq_rq request type. With the added elevator_private2, we have enough room in struct request to get rid of any crq allocation/free for each request. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: remove the crq flag functions/variableJens Axboe2006-09-30
| | | | | | | There's just one flag currently (SYNC), and that one can be grabbed from the request. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: convert to using the FIFO elevator definesJens Axboe2006-09-30
| | | | Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: migrate to using the elevator rb functionsJens Axboe2006-09-30
| | | | | | This removes the rbtree handling from CFQ. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] elevator: move the backmerging logic into the elevator coreJens Axboe2006-09-30
| | | | | | | | | | | Right now, every IO scheduler implements its own backmerging (except for noop, which does no merging). That results in duplicated code for essentially the same operation, which is never a good thing. This patch moves the backmerging out of the io schedulers and into the elevator core. We save 1.6kb of text and as a bonus get backmerging for noop as well. Win-win! Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq_cic_link: fix usage of wrong cfq_io_contextOleg Nesterov2006-08-21
| | | | | | | | Obviously, cfq_cic_link() shouldn't free a just allocated cfq_io_context? The dead key is from __cic, so drop that. Signed-off-by: Oleg Nesterov <oleg@tv-sign.ru> Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: don't use a hard jiffies value, translate from msecsJens Axboe2006-07-25
| | | | | | | | | | | | The CIC_SEEKY() test really wants to use the minimum of either: - 2 msecs (not jiffies) - or, the pending slice time So code it like that. Signed-off-by: Jens Axboe <axboe@suse.de>
* Remove obsolete #include <linux/config.h>Jörn Engel2006-06-30
| | | | | Signed-off-by: Jörn Engel <joern@wohnheim.fh-wedel.de> Signed-off-by: Adrian Bunk <bunk@stusta.de>
* [PATCH] rbtree: support functions used by the io schedulersJens Axboe2006-06-23
| | | | | | | They all duplicate macros to check for empty root and/or node, and clearing a node. So put those in rbtree.h. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: rq update fixesJens Axboe2006-06-23
| | | | | | | | | - Remember to set ->last_sector so that the cfq_choose_req() logic works correctly. - Remove redundant call to cfq_choose_req() Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: many performance fixesJens Axboe2006-06-23
| | | | | | | | | | | | | | | | | | | | This is a collection of patches that greatly improve CFQ performance in some circumstances. - Change the idling logic to only kick in after a request is done and we are deciding what to do. Before the idling included the request service time, so it was hard to adjust. Now it's true think/idle time. - Take advantage of TCQ/NCQ/queueing for seeky sync workloads, but keep it in control for sync and sequential (or close to) workloads. - Expire queues immediately and move on to other busy queues, if we are not going to idle after the current one finishes. - Don't rearm idle timer if there are no busy queues. Just leave the system idle. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: correctly set ioprio on both targetsJens Axboe2006-06-23
| | | | | | | | | | | | | | | | | Patch originally from Vasily Tarasov <vtaras@sw.ru> If you set io-priority of process 1 using sys_ioprio_set system call by another process 2 (like ionice do), then cfq_init_prio_data() function sets priority of process 2 (current) on queue of process 1 and clears the flag, that designates change of ioprio. So the process 1 will work like with priority of process 2. I propose not to call cfq_init_prio_data() on io-priority change, but only mark queue as queue with changed prority. Every time when new request comes cfq-scheduler checks for this flag and atomaticaly changes priority of queue to new value. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] Kill PF_SYNCWRITE flagJens Axboe2006-06-23
| | | | | | | | | | | | | | | A process flag to indicate whether we are doing sync io is incredibly ugly. It also causes performance problems when one does a lot of async io and then proceeds to sync it. Part of the io will go out as async, and the other part as sync. This causes a disconnect between the previously submitted io and the synced io. For io schedulers such as CFQ, this will cause us lost merges and suboptimal behaviour in scheduling. Remove PF_SYNCWRITE completely from the fsync/msync paths, and let the O_DIRECT path just directly indicate that the writes are sync by using WRITE_SYNC instead. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: Don't set the queue batching limitsJens Axboe2006-06-23
| | | | | | | | We cannot update them if the user changes nr_requests, so don't set it in the first place. The gains are pretty questionable as well. The batching loss has been shown to decrease throughput. Signed-off-by: Jens Axboe <axboe@suse.de>
* Fix up CFQ scheduler for recent rbtree node shrinkageLinus Torvalds2006-06-20
| | | | | | | | The color is now in the low bits of the parent pointer, and initializing it to 0 happens as part of the whole memset above, so just remove the unnecessary RB_CLEAR_COLOR. Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* Merge git://git.infradead.org/~dwmw2/rbtree-2.6Linus Torvalds2006-06-20
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * git://git.infradead.org/~dwmw2/rbtree-2.6: [RBTREE] Switch rb_colour() et al to en_US spelling of 'color' for consistency Update UML kernel/physmem.c to use rb_parent() accessor macro [RBTREE] Update hrtimers to use rb_parent() accessor macro. [RBTREE] Add explicit alignment to sizeof(long) for struct rb_node. [RBTREE] Merge colour and parent fields of struct rb_node. [RBTREE] Remove dead code in rb_erase() [RBTREE] Update JFFS2 to use rb_parent() accessor macro. [RBTREE] Update eventpoll.c to use rb_parent() accessor macro. [RBTREE] Update key.c to use rb_parent() accessor macro. [RBTREE] Update ext3 to use rb_parent() accessor macro. [RBTREE] Change rbtree off-tree marking in I/O schedulers. [RBTREE] Add accessor macros for colour and parent fields of rb_node
| * [RBTREE] Change rbtree off-tree marking in I/O schedulers.David Woodhouse2006-04-21
| | | | | | | | | | | | | | | | | | They were abusing the rb_color field to mark nodes which weren't currently on the tree. Fix that to use the same method as eventpoll did -- setting the parent pointer to point back to itself. And use the appropriate accessor macros for setting and reading the parent. Signed-off-by: David Woodhouse <dwmw2@infradead.org>
* | [PATCH] cfq-iosched: fix crash in do_div()Jens Axboe2006-06-14
| | | | | | | | | | | | | | | | | | | | | | | | | | We don't clear the seek stat values in cfq_alloc_io_context(), and if ->seek_mean is unlucky enough to be set to -36 by chance, the first invocation of cfq_update_io_seektime() will oops with a divide by zero in do_div(). Just memset the entire cic instead of filling invididual values independently. Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* | [PATCH] elevator switching raceJens Axboe2006-06-08
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There's a race between shutting down one io scheduler and firing up the next, in which a new io could enter and cause the io scheduler to be invoked with bad or NULL data. To fix this, we need to maintain the queue lock for a bit longer. Unfortunately we cannot do that, since the elevator init requires to be run without the lock held. This isn't easily fixable, without also changing the mempool API. So split the initialization into two parts, and alloc-init operation and an attach operation. Then we can preallocate the io scheduler and related structures, and run the attach inside the lock after we detach the old one. This patch has survived 30 minutes of 1 second io scheduler switching with a very busy io load. Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* | [PATCH] cfq-iosched: busy_rr fairness fixJens Axboe2006-06-01
| | | | | | | | | | | | | | Now that we select busy_rr for possible service, insert entries at the back of that list instead of at the front. Signed-off-by: Jens Axboe <axboe@suse.de>
* | [PATCH] cfq-iosched: fix bug in timer handling for the idle classJens Axboe2006-06-01
| | | | | | | | | | | | | | | | | | There's a small window from when the timer is entered and we grab the queue lock, where cfq_set_active_queue() could be rearming the timer for us. Seen in the wild on a 12-way ppc box. Fix this by just using mod_timer(), which will do the right thing for us. Signed-off-by: Jens Axboe <axboe@suse.de>
* | [PATCH] cfq-iosched: Detect hardware queueingJens Axboe2006-06-01
| | | | | | | | | | | | | | | | If the hardware is doing real queueing, decide that it's worthless to idle the hardware. It does reasonable simultaneous io in that case anyways, and the idling hurts some work loads. Signed-off-by: Jens Axboe <axboe@suse.de>
* | [PATCH] cfq-iosched: Detect idle process issuing async requestJens Axboe2006-06-01
| | | | | | | | | | | | | | | | If we are anticipating a sync request from this process and we are waiting for that and see an async request come in, expire that slice and move on. Signed-off-by: Jens Axboe <axboe@suse.de>
* | [PATCH] cfq-iosched: check busy queues before deciding we are idleJens Axboe2006-06-01
| | | | | | | | | | | | | | | | For just one busy queue (like async write out), we often overlooked that we could queue more io and decided we were idle instead. This causes us quite a bit of performance loss. Signed-off-by: Jens Axboe <axboe@suse.de>
* | [PATCH] cfq-iosched: fixup locking and ->queue_list list managementJens Axboe2006-05-30
|/ | | | | | | | - Drop cic from the list when seen as dead. - Fixup the locking, just use a simple spinlock. Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* [PATCH] cfq: Further rbtree traversal and cfq_exit_queue() race fixOGAWA Hirofumi2006-04-18
| | | | | | | | | | | | In current code, we are re-reading cic->key after dead cic->key check. So, in theory, it may really re-read *after* cfq_exit_queue() seted NULL. To avoid race, we copy it to stack, then use it. With this change, I guess gcc will assign cic->key to a register or stack, and it wouldn't be re-readed. Signed-off-by: OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH 2/2] cfq: fix cic's rbtree traversalOGAWA Hirofumi2006-04-18
| | | | | | | | | When queue dies, we set cic->key=NULL as dead mark. So, when we traverse a rbtree, we must check whether it's still valid key. if it was invalidated, drop it, then restart the traversal from top. Signed-off-by: OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH 1/2] iosched: fix typo and barrier()OGAWA Hirofumi2006-04-18
| | | | | | | | | | | | | | | | | | | | | | On rmmod path, cfq/as waits to make sure all io-contexts was freed. However, it's using complete(), not wait_for_completion(). I think barrier() is not enough in here. To avoid the following case, this patch replaces barrier() with smb_wmb(). cpu0 visibility cpu1 [ioc_gnone=NULL,ioc_count=1] ioc_gnone = &all_gone NULL,ioc_count=1 atomic_read(&ioc_count) NULL,ioc_count=1 wait_for_completion() NULL,ioc_count=0 atomic_sub_and_test() NULL,ioc_count=0 if ( && ioc_gone) [ioc_gone==NULL, so doesn't call complete()] &all_gone,ioc_count=0 Signed-off-by: OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> Signed-off-by: Jens Axboe <axboe@suse.de>
* [BLOCK] cfq-iosched: seek and async performance fixesJens Axboe2006-03-28
| | | | | | | | | | | | | | Detect whether a given process is seeky and if so disable (mostly) the idle window if it is. We still allow just a little idle time, just enough to allow that process to submit a new request. That is needed to maintain fairness across priority groups. In some cases, we could setup several async queues. This is not optimal from a performance POV, since we want all async io in one queue to perform good sorting on it. It also impacted sync queues, as async io got too much slice time. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] cfq-iosched: small cfq_choose_req() optimizationAndreas Mohr2006-03-28
| | | | | | | | | | | | this is a small optimization to cfq_choose_req() in the CFQ I/O scheduler (this function is a semi-often invoked candidate in an oprofile log): by using a bit mask variable, we can use a simple switch() to check the various cases instead of having to query two variables for each check. Benefit: 251 vs. 285 bytes footprint of cfq_choose_req(). Also, common case 0 (no request wrapping) is now checked first in code. Signed-off-by: Andreas Mohr <andi@lisas.de> Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] [BLOCK] cfq-iosched: change cfq io context linking from list to treeJens Axboe2006-03-28
| | | | | | | | | On setups with many disks, we spend a considerable amount of time looking up the process-disk mapping on each queue of io. Testing with a NULL based block driver, this costs 40-50% reduction in throughput for 1000 disks. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] mempool: use mempool_create_slab_pool()Matthew Dobson2006-03-26
| | | | | | | | | | Modify well over a dozen mempool users to call mempool_create_slab_pool() rather than calling mempool_create() with extra arguments, saving about 30 lines of code and increasing readability. Signed-off-by: Matthew Dobson <colpatch@us.ibm.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* [PATCH] fix rmmod problems with elevator attributes, clean them upAl Viro2006-03-18
|
* [PATCH] elevator_t lifetime rules and sysfs fixesAl Viro2006-03-18
|
* [PATCH] noise removal: cfq-iosched.cAl Viro2006-03-18
| | | | Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
* [PATCH] don't bother with refcounting for cfq_dataAl Viro2006-03-18
| | | | Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
* [PATCH] fix cfq_get_queue()/ioprio_set(2) racesAl Viro2006-03-18
| | | | Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
generated by cgit v1.2.2 (git 2.25.0) at 2024-12-23 13:08:42 -0500