Messages in this thread Patch in this message | | | Date | Sun, 16 Jun 2002 23:52:42 -0700 | From | Andrew Morton <> | Subject | [patch 10/19] direct-to-BIO I/O for swapcache pages |
| |
This patch changes the swap I/O handling. The objectives are:
- Remove swap special-casing - Stop using buffer_heads -> direct-to-BIO - Make S_ISREG swapfiles more robust.
I've spent quite some time with swap. The first patches converted swap to use block_read/write_full_page(). These were discarded because they are still using buffer_heads, and a reasonable amount of otherwise unnecessary infrastructure had to be added to the swap code just to make it look like a regular fs. So this code just has a custom direct-to-BIO path for swap, which seems to be the most comfortable approach.
A significant thing here is the introduction of "swap extents". A swap extent is a simple data structure which maps a range of swap pages onto a range of disk sectors. It is simply:
struct swap_extent { struct list_head list; pgoff_t start_page; pgoff_t nr_pages; sector_t start_block; };
At swapon time (for an S_ISREG swapfile), each block in the file is bmapped() and the block numbers are parsed to generate the device's swap extent list. This extent list is quite compact - a 512 megabyte swapfile generates about 130 nodes in the list. That's about 4 kbytes of storage. The conversion from filesystem blocksize blocks into PAGE_SIZE blocks is performed at swapon time.
At swapon time (for an S_ISBLK swapfile), we install a single swap extent which describes the entire device.
The advantages of the swap extents are:
1: We never have to run bmap() (ie: read from disk) at swapout time. So S_ISREG swapfiles are now just as robust as S_ISBLK swapfiles.
2: All the differences between S_ISBLK swapfiles and S_ISREG swapfiles are handled at swapon time. During normal operation, we just don't care. Both types of swapfiles are handled the same way.
3: The extent lists always operate in PAGE_SIZE units. So the problems of going from fs blocksize to PAGE_SIZE are handled at swapon time and normal operating code doesn't need to care.
4: Because we don't have to fiddle with different blocksizes, we can go direct-to-BIO for swap_readpage() and swap_writepage(). This introduces the kernel-wide invariant "anonymous pages never have buffers attached", which cleans some things up nicely. All those block_flushpage() calls in the swap code simply go away.
5: The kernel no longer has to allocate both buffer_heads and BIOs to perform swapout. Just a BIO.
6: It permits us to perform swapcache writeout and throttling for GFP_NOFS allocations (a later patch).
(Well, there is one sort of anon page which can have buffers: the pages which are cast adrift in truncate_complete_page() because do_invalidatepage() failed. But these pages are never added to swapcache, and nobody except the VM LRU has to deal with them).
The swapfile parser in setup_swap_extents() will attempt to extract the largest possible number of PAGE_SIZE-sized and PAGE_SIZE-aligned chunks of disk from the S_ISREG swapfile. Any stray blocks (due to file discontiguities) are simply discarded - we never swap to those.
If an S_ISREG swapfile is found to have any unmapped blocks (file holes) then the swapon attempt will fail.
The extent list can be quite large (hundreds of nodes for a gigabyte S_ISREG swapfile). It needs to be consulted once for each page within swap_readpage() and swap_writepage(). Hence there is a risk that we could blow significant amounts of CPU walking that list. However I have implemented a "where we found the last block" cache, which is used as the starting point for the next search. Empirical testing indicates that this is wildly effective - the average length of the list walk in map_swap_page() is 0.3 iterations per page, with a 130-element list.
It _could_ be that some workloads do start suffering long walks in that code, and perhaps a tree would be needed there. But I doubt that, and if this is happening then it means that we're seeking all over the disk for swap I/O, and the list walk is the least of our problems.
rw_swap_page_nolock() now takes a page*, not a kernel virtual address. It has been renamed to rw_swap_page_sync() and it takes care of locking and unlocking the page itself. Which is all a much better interface.
Support for type 0 swap has been removed. Current versions of mkwap(8) seem to never produce v0 swap unless you explicitly ask for it, so I doubt if this will affect anyone. If you _do_ have a type 0 swapfile, swapon will fail and the message
version 0 swap is no longer supported. Use mkswap -v1 /dev/sdb3
is printed. We can remove that code for real later on. Really, all that swapfile header parsing should be pushed out to userspace.
This code always uses single-page BIOs for swapin and swapout. I have an additional patch which converts swap to use mpage_writepages(), so we swap out in 16-page BIOs. It works fine, but I don't intend to submit that. There just doesn't seem to be any significant advantage to it.
I can't see anything in sys_swapon()/sys_swapoff() which needs the lock_kernel() calls, so I deleted them.
If you ftruncate an S_ISREG swapfile to a shorter size while it is in use, subsequent swapout will destroy the filesystem. It was always thus, but it is much, much easier to do now. Not really a kernel problem, but swapon(8) should not be allowing the kernel to use swapfiles which are modifiable by unprivileged users.
Incidentally. The stale swapcache-page optimisation in this code:
static int swap_writepage(struct page *page) { if (remove_exclusive_swap_page(page)) { unlock_page(page); return 0; } rw_swap_page(WRITE, page); return 0; }
*never* seems to trigger. You can stick a printk in there and watch it. This is unrelated to my changes. So perhaps something has become broken in there somewhere??
--- 2.5.22/fs/buffer.c~swap-bio Sun Jun 16 22:50:18 2002 +++ 2.5.22-akpm/fs/buffer.c Sun Jun 16 23:22:45 2002 @@ -492,7 +492,7 @@ static void free_more_memory(void) } /* - * I/O completion handler for block_read_full_page() and brw_page() - pages + * I/O completion handler for block_read_full_page() - pages * which come unlocked at the end of I/O. */ static void end_buffer_async_read(struct buffer_head *bh, int uptodate) @@ -551,9 +551,8 @@ still_busy: } /* - * Completion handler for block_write_full_page() and for brw_page() - pages - * which are unlocked during I/O, and which have PageWriteback cleared - * upon I/O completion. + * Completion handler for block_write_full_page() - pages which are unlocked + * during I/O, and which have PageWriteback cleared upon I/O completion. */ static void end_buffer_async_write(struct buffer_head *bh, int uptodate) { @@ -1360,11 +1359,11 @@ int block_invalidatepage(struct page *pa { struct buffer_head *head, *bh, *next; unsigned int curr_off = 0; + int ret = 1; - if (!PageLocked(page)) - BUG(); + BUG_ON(!PageLocked(page)); if (!page_has_buffers(page)) - return 1; + goto out; head = page_buffers(page); bh = head; @@ -1386,12 +1385,10 @@ int block_invalidatepage(struct page *pa * The get_block cached value has been unconditionally invalidated, * so real IO is not possible anymore. */ - if (offset == 0) { - if (!try_to_release_page(page, 0)) - return 0; - } - - return 1; + if (offset == 0) + ret = try_to_release_page(page, 0); +out: + return ret; } EXPORT_SYMBOL(block_invalidatepage); @@ -2269,57 +2266,6 @@ int brw_kiovec(int rw, int nr, struct ki } /* - * Start I/O on a page. - * This function expects the page to be locked and may return - * before I/O is complete. You then have to check page->locked - * and page->uptodate. - * - * FIXME: we need a swapper_inode->get_block function to remove - * some of the bmap kludges and interface ugliness here. - */ -int brw_page(int rw, struct page *page, - struct block_device *bdev, sector_t b[], int size) -{ - struct buffer_head *head, *bh; - - BUG_ON(!PageLocked(page)); - - if (!page_has_buffers(page)) - create_empty_buffers(page, size, 0); - head = bh = page_buffers(page); - - /* Stage 1: lock all the buffers */ - do { - lock_buffer(bh); - bh->b_blocknr = *(b++); - bh->b_bdev = bdev; - set_buffer_mapped(bh); - if (rw == WRITE) { - set_buffer_uptodate(bh); - clear_buffer_dirty(bh); - mark_buffer_async_write(bh); - } else { - mark_buffer_async_read(bh); - } - bh = bh->b_this_page; - } while (bh != head); - - if (rw == WRITE) { - BUG_ON(PageWriteback(page)); - SetPageWriteback(page); - unlock_page(page); - } - - /* Stage 2: start the IO */ - do { - struct buffer_head *next = bh->b_this_page; - submit_bh(rw, bh); - bh = next; - } while (bh != head); - return 0; -} - -/* * Sanity checks for try_to_free_buffers. */ static void check_ttfb_buffer(struct page *page, struct buffer_head *bh) --- 2.5.22/include/linux/buffer_head.h~swap-bio Sun Jun 16 22:50:18 2002 +++ 2.5.22-akpm/include/linux/buffer_head.h Sun Jun 16 23:22:46 2002 @@ -181,7 +181,6 @@ struct buffer_head * __bread(struct bloc void wakeup_bdflush(void); struct buffer_head *alloc_buffer_head(int async); void free_buffer_head(struct buffer_head * bh); -int brw_page(int, struct page *, struct block_device *, sector_t [], int); void FASTCALL(unlock_buffer(struct buffer_head *bh)); /* --- 2.5.22/include/linux/swap.h~swap-bio Sun Jun 16 22:50:18 2002 +++ 2.5.22-akpm/include/linux/swap.h Sun Jun 16 22:50:18 2002 @@ -5,6 +5,7 @@ #include <linux/kdev_t.h> #include <linux/linkage.h> #include <linux/mmzone.h> +#include <linux/list.h> #include <asm/page.h> #define SWAP_FLAG_PREFER 0x8000 /* set if swap priority specified */ @@ -62,6 +63,21 @@ typedef struct { #ifdef __KERNEL__ /* + * A swap extent maps a range of a swapfile's PAGE_SIZE pages onto a range of + * disk blocks. A list of swap extents maps the entire swapfile. (Where the + * term `swapfile' refers to either a blockdevice or an IS_REG file. Apart + * from setup, they're handled identically. + * + * We always assume that blocks are of size PAGE_SIZE. + */ +struct swap_extent { + struct list_head list; + pgoff_t start_page; + pgoff_t nr_pages; + sector_t start_block; +}; + +/* * Max bad pages in the new format.. */ #define __swapoffset(x) ((unsigned long)&((union swap_header *)0)->x) @@ -83,11 +99,17 @@ enum { /* * The in-memory structure used to track swap areas. + * extent_list.prev points at the lowest-index extent. That list is + * sorted. */ struct swap_info_struct { unsigned int flags; spinlock_t sdev_lock; struct file *swap_file; + struct block_device *bdev; + struct list_head extent_list; + int nr_extents; + struct swap_extent *curr_swap_extent; unsigned old_block_size; unsigned short * swap_map; unsigned int lowest_bit; @@ -134,8 +156,9 @@ extern wait_queue_head_t kswapd_wait; extern int FASTCALL(try_to_free_pages(zone_t *, unsigned int, unsigned int)); /* linux/mm/page_io.c */ -extern void rw_swap_page(int, struct page *); -extern void rw_swap_page_nolock(int, swp_entry_t, char *); +int swap_readpage(struct file *file, struct page *page); +int swap_writepage(struct page *page); +int rw_swap_page_sync(int rw, swp_entry_t entry, struct page *page); /* linux/mm/page_alloc.c */ @@ -163,12 +186,13 @@ extern unsigned int nr_swapfiles; extern struct swap_info_struct swap_info[]; extern void si_swapinfo(struct sysinfo *); extern swp_entry_t get_swap_page(void); -extern void get_swaphandle_info(swp_entry_t, unsigned long *, struct inode **); extern int swap_duplicate(swp_entry_t); -extern int swap_count(struct page *); extern int valid_swaphandles(swp_entry_t, unsigned long *); extern void swap_free(swp_entry_t); extern void free_swap_and_cache(swp_entry_t); +sector_t map_swap_page(struct swap_info_struct *p, pgoff_t offset); +struct swap_info_struct *get_swap_info_struct(unsigned type); + struct swap_list_t { int head; /* head of priority-ordered swapfile list */ int next; /* swapfile to be used next */ --- 2.5.22/kernel/ksyms.c~swap-bio Sun Jun 16 22:50:18 2002 +++ 2.5.22-akpm/kernel/ksyms.c Sun Jun 16 23:22:46 2002 @@ -562,7 +562,6 @@ EXPORT_SYMBOL(buffer_insert_list); EXPORT_SYMBOL(make_bad_inode); EXPORT_SYMBOL(is_bad_inode); EXPORT_SYMBOL(event); -EXPORT_SYMBOL(brw_page); #ifdef CONFIG_UID16 EXPORT_SYMBOL(overflowuid); --- 2.5.22/kernel/suspend.c~swap-bio Sun Jun 16 22:50:18 2002 +++ 2.5.22-akpm/kernel/suspend.c Sun Jun 16 22:50:18 2002 @@ -319,14 +319,15 @@ static void mark_swapfiles(swp_entry_t p { swp_entry_t entry; union diskpage *cur; - - cur = (union diskpage *)get_free_page(GFP_ATOMIC); - if (!cur) + struct page *page; + + page = alloc_page(GFP_ATOMIC); + if (!page) panic("Out of memory in mark_swapfiles"); + cur = page_address(page); /* XXX: this is dirty hack to get first page of swap file */ entry = swp_entry(root_swap, 0); - lock_page(virt_to_page((unsigned long)cur)); - rw_swap_page_nolock(READ, entry, (char *) cur); + rw_swap_page_sync(READ, entry, page); if (mode == MARK_SWAP_RESUME) { if (!memcmp("SUSP1R",cur->swh.magic.magic,6)) @@ -344,10 +345,8 @@ static void mark_swapfiles(swp_entry_t p cur->link.next = prev; /* prev is the first/last swap page of the resume area */ /* link.next lies *no more* in last 4 bytes of magic */ } - lock_page(virt_to_page((unsigned long)cur)); - rw_swap_page_nolock(WRITE, entry, (char *)cur); - - free_page((unsigned long)cur); + rw_swap_page_sync(WRITE, entry, page); + __free_page(page); } static void read_swapfiles(void) /* This is called before saving image */ @@ -408,6 +407,7 @@ static int write_suspend_image(void) int nr_pgdir_pages = SUSPEND_PD_PAGES(nr_copy_pages); union diskpage *cur, *buffer = (union diskpage *)get_free_page(GFP_ATOMIC); unsigned long address; + struct page *page; PRINTS( "Writing data to swap (%d pages): ", nr_copy_pages ); for (i=0; i<nr_copy_pages; i++) { @@ -420,13 +420,8 @@ static int write_suspend_image(void) panic("\nPage %d: not enough swapspace on suspend device", i ); address = (pagedir_nosave+i)->address; - lock_page(virt_to_page(address)); - { - long dummy1; - struct inode *suspend_file; - get_swaphandle_info(entry, &dummy1, &suspend_file); - } - rw_swap_page_nolock(WRITE, entry, (char *) address); + page = virt_to_page(address); + rw_swap_page_sync(WRITE, entry, page); (pagedir_nosave+i)->swap_address = entry; } PRINTK(" done\n"); @@ -451,8 +446,8 @@ static int write_suspend_image(void) if (PAGE_SIZE % sizeof(struct pbe)) panic("I need PAGE_SIZE to be integer multiple of struct pbe, otherwise next assignment could damage pagedir"); cur->link.next = prev; - lock_page(virt_to_page((unsigned long)cur)); - rw_swap_page_nolock(WRITE, entry, (char *) cur); + page = virt_to_page((unsigned long)cur); + rw_swap_page_sync(WRITE, entry, page); prev = entry; } PRINTK(", header"); @@ -472,8 +467,8 @@ static int write_suspend_image(void) cur->link.next = prev; - lock_page(virt_to_page((unsigned long)cur)); - rw_swap_page_nolock(WRITE, entry, (char *) cur); + page = virt_to_page((unsigned long)cur); + rw_swap_page_sync(WRITE, entry, page); prev = entry; PRINTK( ", signature" ); --- 2.5.22/mm/page_io.c~swap-bio Sun Jun 16 22:50:18 2002 +++ 2.5.22-akpm/mm/page_io.c Sun Jun 16 22:50:18 2002 @@ -14,112 +14,163 @@ #include <linux/kernel_stat.h> #include <linux/pagemap.h> #include <linux/swap.h> -#include <linux/swapctl.h> -#include <linux/buffer_head.h> /* for brw_page() */ - +#include <linux/bio.h> +#include <linux/buffer_head.h> #include <asm/pgtable.h> +#include <linux/swapops.h> -/* - * Reads or writes a swap page. - * wait=1: start I/O and wait for completion. wait=0: start asynchronous I/O. - * - * Important prevention of race condition: the caller *must* atomically - * create a unique swap cache entry for this swap page before calling - * rw_swap_page, and must lock that page. By ensuring that there is a - * single page of memory reserved for the swap entry, the normal VM page - * lock on that page also doubles as a lock on swap entries. Having only - * one lock to deal with per swap entry (rather than locking swap and memory - * independently) also makes it easier to make certain swapping operations - * atomic, which is particularly important when we are trying to ensure - * that shared pages stay shared while being swapped. - */ +static int +swap_get_block(struct inode *inode, sector_t iblock, + struct buffer_head *bh_result, int create) +{ + struct swap_info_struct *sis; + swp_entry_t entry; -static int rw_swap_page_base(int rw, swp_entry_t entry, struct page *page) + entry.val = iblock; + sis = get_swap_info_struct(swp_type(entry)); + bh_result->b_bdev = sis->bdev; + bh_result->b_blocknr = map_swap_page(sis, swp_offset(entry)); + bh_result->b_size = PAGE_SIZE; + set_buffer_mapped(bh_result); + return 0; +} + +static struct bio * +get_swap_bio(int gfp_flags, struct page *page, bio_end_io_t end_io) { - unsigned long offset; - sector_t zones[PAGE_SIZE/512]; - int zones_used; - int block_size; - struct inode *swapf = 0; - struct block_device *bdev; + struct bio *bio; + struct buffer_head bh; - if (rw == READ) { + bio = bio_alloc(gfp_flags, 1); + if (bio) { + swap_get_block(NULL, page->index, &bh, 1); + bio->bi_sector = bh.b_blocknr * (PAGE_SIZE >> 9); + bio->bi_bdev = bh.b_bdev; + bio->bi_io_vec[0].bv_page = page; + bio->bi_io_vec[0].bv_len = PAGE_SIZE; + bio->bi_io_vec[0].bv_offset = 0; + bio->bi_vcnt = 1; + bio->bi_idx = 0; + bio->bi_size = PAGE_SIZE; + bio->bi_end_io = end_io; + } + return bio; +} + +static void end_swap_bio_write(struct bio *bio) +{ + const int uptodate = test_bit(BIO_UPTODATE, &bio->bi_flags); + struct page *page = bio->bi_io_vec[0].bv_page; + + if (!uptodate) + SetPageError(page); + end_page_writeback(page); + bio_put(bio); +} + +static void end_swap_bio_read(struct bio *bio) +{ + const int uptodate = test_bit(BIO_UPTODATE, &bio->bi_flags); + struct page *page = bio->bi_io_vec[0].bv_page; + + if (!uptodate) { + SetPageError(page); ClearPageUptodate(page); - kstat.pswpin++; - } else - kstat.pswpout++; - - get_swaphandle_info(entry, &offset, &swapf); - bdev = swapf->i_bdev; - if (bdev) { - zones[0] = offset; - zones_used = 1; - block_size = PAGE_SIZE; } else { - int i, j; - unsigned int block = offset - << (PAGE_SHIFT - swapf->i_sb->s_blocksize_bits); - - block_size = swapf->i_sb->s_blocksize; - for (i=0, j=0; j< PAGE_SIZE ; i++, j += block_size) - if (!(zones[i] = bmap(swapf,block++))) { - printk("rw_swap_page: bad swap file\n"); - return 0; - } - zones_used = i; - bdev = swapf->i_sb->s_bdev; - } - - /* block_size == PAGE_SIZE/zones_used */ - brw_page(rw, page, bdev, zones, block_size); - - /* Note! For consistency we do all of the logic, - * decrementing the page count, and unlocking the page in the - * swap lock map - in the IO completion handler. - */ - return 1; + SetPageUptodate(page); + } + unlock_page(page); + bio_put(bio); } /* - * A simple wrapper so the base function doesn't need to enforce - * that all swap pages go through the swap cache! We verify that: - * - the page is locked - * - it's marked as being swap-cache - * - it's associated with the swap inode + * We may have stale swap cache pages in memory: notice + * them here and get rid of the unnecessary final write. */ -void rw_swap_page(int rw, struct page *page) +int swap_writepage(struct page *page) { - swp_entry_t entry; + struct bio *bio; + int ret = 0; - entry.val = page->index; - - if (!PageLocked(page)) - PAGE_BUG(page); - if (!PageSwapCache(page)) - PAGE_BUG(page); - if (!rw_swap_page_base(rw, entry, page)) + if (remove_exclusive_swap_page(page)) { unlock_page(page); + goto out; + } + bio = get_swap_bio(GFP_NOIO, page, end_swap_bio_write); + if (bio == NULL) { + ret = -ENOMEM; + goto out; + } + kstat.pswpout++; + SetPageWriteback(page); + unlock_page(page); + submit_bio(WRITE, bio); +out: + return ret; +} + +int swap_readpage(struct file *file, struct page *page) +{ + struct bio *bio; + int ret = 0; + + ClearPageUptodate(page); + bio = get_swap_bio(GFP_KERNEL, page, end_swap_bio_read); + if (bio == NULL) { + ret = -ENOMEM; + goto out; + } + kstat.pswpin++; + submit_bio(READ, bio); +out: + return ret; +} +/* + * swapper_space doesn't have a real inode, so it gets a special vm_writeback() + * so we don't need swap special cases in generic_vm_writeback(). + * + * Swap pages are PageLocked and PageWriteback while under writeout so that + * memory allocators will throttle against them. + */ +static int swap_vm_writeback(struct page *page, int *nr_to_write) +{ + struct address_space *mapping = page->mapping; + + unlock_page(page); + return generic_writepages(mapping, nr_to_write); } +struct address_space_operations swap_aops = { + vm_writeback: swap_vm_writeback, + writepage: swap_writepage, + readpage: swap_readpage, + sync_page: block_sync_page, + set_page_dirty: __set_page_dirty_nobuffers, +}; + /* - * The swap lock map insists that pages be in the page cache! - * Therefore we can't use it. Later when we can remove the need for the - * lock map and we can reduce the number of functions exported. + * A scruffy utility function to read or write an arbitrary swap page + * and wait on the I/O. */ -void rw_swap_page_nolock(int rw, swp_entry_t entry, char *buf) +int rw_swap_page_sync(int rw, swp_entry_t entry, struct page *page) { - struct page *page = virt_to_page(buf); - - if (!PageLocked(page)) - PAGE_BUG(page); - if (page->mapping) - PAGE_BUG(page); - /* needs sync_page to wait I/O completation */ + int ret; + + lock_page(page); + + BUG_ON(page->mapping); page->mapping = &swapper_space; - if (rw_swap_page_base(rw, entry, page)) - lock_page(page); - if (page_has_buffers(page) && !try_to_free_buffers(page)) - PAGE_BUG(page); + page->index = entry.val; + + if (rw == READ) { + ret = swap_readpage(NULL, page); + wait_on_page_locked(page); + } else { + ret = swap_writepage(page); + wait_on_page_writeback(page); + } page->mapping = NULL; - unlock_page(page); + if (ret == 0 && (!PageUptodate(page) || PageError(page))) + ret = -EIO; + return ret; } --- 2.5.22/mm/swapfile.c~swap-bio Sun Jun 16 22:50:18 2002 +++ 2.5.22-akpm/mm/swapfile.c Sun Jun 16 22:50:18 2002 @@ -16,7 +16,7 @@ #include <linux/namei.h> #include <linux/shm.h> #include <linux/blkdev.h> -#include <linux/buffer_head.h> /* for try_to_free_buffers() */ +#include <linux/buffer_head.h> #include <asm/pgtable.h> #include <linux/swapops.h> @@ -294,13 +294,14 @@ int remove_exclusive_swap_page(struct pa struct swap_info_struct * p; swp_entry_t entry; - if (!PageLocked(page)) - BUG(); + BUG_ON(page_has_buffers(page)); + BUG_ON(!PageLocked(page)); + if (!PageSwapCache(page)) return 0; if (PageWriteback(page)) return 0; - if (page_count(page) - !!PagePrivate(page) != 2) /* 2: us + cache */ + if (page_count(page) != 2) /* 2: us + cache */ return 0; entry.val = page->index; @@ -313,14 +314,8 @@ int remove_exclusive_swap_page(struct pa if (p->swap_map[swp_offset(entry)] == 1) { /* Recheck the page count with the pagecache lock held.. */ write_lock(&swapper_space.page_lock); - if ((page_count(page) - !!page_has_buffers(page) == 2) && - !PageWriteback(page)) { + if ((page_count(page) == 2) && !PageWriteback(page)) { __delete_from_swap_cache(page); - /* - * NOTE: if/when swap gets buffer/page coherency - * like other mappings, we'll need to mark the buffers - * dirty here too. set_page_dirty(). - */ SetPageDirty(page); retval = 1; } @@ -329,8 +324,6 @@ int remove_exclusive_swap_page(struct pa swap_info_put(p); if (retval) { - if (page_has_buffers(page) && !try_to_free_buffers(page)) - BUG(); swap_free(entry); page_cache_release(page); } @@ -356,8 +349,9 @@ void free_swap_and_cache(swp_entry_t ent if (page) { int one_user; + BUG_ON(page_has_buffers(page)); page_cache_get(page); - one_user = (page_count(page) - !!page_has_buffers(page) == 2); + one_user = (page_count(page) == 2); /* Only cache user (+us), or swap space full? Free it! */ if (!PageWriteback(page) && (one_user || vm_swap_full())) { delete_from_swap_cache(page); @@ -691,7 +685,7 @@ static int try_to_unuse(unsigned int typ * Note shmem_unuse already deleted its from swap cache. */ if ((*swap_map > 1) && PageDirty(page) && PageSwapCache(page)) { - rw_swap_page(WRITE, page); + swap_writepage(page); lock_page(page); } if (PageSwapCache(page)) { @@ -725,6 +719,207 @@ static int try_to_unuse(unsigned int typ return retval; } +/* + * Use this swapdev's extent info to locate the (PAGE_SIZE) block which + * corresponds to page offset `offset'. + */ +sector_t map_swap_page(struct swap_info_struct *sis, pgoff_t offset) +{ + struct swap_extent *se = sis->curr_swap_extent; + struct swap_extent *start_se = se; + + for ( ; ; ) { + struct list_head *lh; + + if (se->start_page <= offset && + offset < (se->start_page + se->nr_pages)) { + return se->start_block + (offset - se->start_page); + } + lh = se->list.prev; + if (lh == &sis->extent_list) + lh = lh->prev; + se = list_entry(lh, struct swap_extent, list); + sis->curr_swap_extent = se; + BUG_ON(se == start_se); /* It *must* be present */ + } +} + +/* + * Free all of a swapdev's extent information + */ +static void destroy_swap_extents(struct swap_info_struct *sis) +{ + while (!list_empty(&sis->extent_list)) { + struct swap_extent *se; + + se = list_entry(sis->extent_list.next, + struct swap_extent, list); + list_del(&se->list); + kfree(se); + } + sis->nr_extents = 0; +} + +/* + * Add a block range (and the corresponding page range) into this swapdev's + * extent list. The extent list is kept sorted in block order. + * + * This function rather assumes that it is called in ascending sector_t order. + * It doesn't look for extent coalescing opportunities. + */ +static int +add_swap_extent(struct swap_info_struct *sis, unsigned long start_page, + unsigned long nr_pages, sector_t start_block) +{ + struct swap_extent *se; + struct swap_extent *new_se; + struct list_head *lh; + + lh = sis->extent_list.next; /* The highest-addressed block */ + while (lh != &sis->extent_list) { + se = list_entry(lh, struct swap_extent, list); + if (se->start_block + se->nr_pages == start_block) { + /* Merge it */ + se->nr_pages += nr_pages; + return 0; + } + lh = lh->next; + } + + /* + * No merge. Insert a new extent, preserving ordering. + */ + new_se = kmalloc(sizeof(*se), GFP_KERNEL); + if (new_se == NULL) + return -ENOMEM; + new_se->start_page = start_page; + new_se->nr_pages = nr_pages; + new_se->start_block = start_block; + + lh = sis->extent_list.prev; /* The lowest block */ + while (lh != &sis->extent_list) { + se = list_entry(lh, struct swap_extent, list); + if (se->start_block > start_block) + break; + lh = lh->prev; + } + list_add_tail(&new_se->list, lh); + sis->nr_extents++; + return 0; +} + +/* + * A `swap extent' is a simple thing which maps a contiguous range of pages + * onto a contiguous range of disk blocks. An ordered list of swap extents + * is built at swapon time and is then used at swap_writepage/swap_readpage + * time for locating where on disk a page belongs. + * + * If the swapfile is an S_ISBLK block device, a single extent is installed. + * This is done so that the main operating code can treat S_ISBLK and S_ISREG + * swap files identically. + * + * Whether the swapdev is an S_ISREG file or an S_ISBLK blockdev, the swap + * extent list operates in PAGE_SIZE disk blocks. Both S_ISREG and S_ISBLK + * swapfiles are handled *identically* after swapon time. + * + * For S_ISREG swapfiles, setup_swap_extents() will walk all the file's blocks + * and will parse them into an ordered extent list, in PAGE_SIZE chunks. If + * some stray blocks are found which do not fall within the PAGE_SIZE alignment + * requirements, they are simply tossed out - we will never use those blocks + * for swapping. + * + * The amount of disk space which a single swap extent represents varies. + * Typically it is in the 1-4 megabyte range. So we can have hundreds of + * extents in the list. To avoid much list walking, we cache the previous + * search location in `curr_swap_extent', and start new searches from there. + * This is extremely effective. The average number of iterations in + * map_swap_page() has been measured at about 0.3 per page. - akpm. + */ +static int setup_swap_extents(struct swap_info_struct *sis) +{ + struct inode *inode; + unsigned blocks_per_page; + unsigned long page_no; + unsigned blkbits; + sector_t probe_block; + sector_t last_block; + int ret; + + inode = sis->swap_file->f_dentry->d_inode; + if (S_ISBLK(inode->i_mode)) { + ret = add_swap_extent(sis, 0, sis->max, 0); + goto done; + } + + blkbits = inode->i_blkbits; + blocks_per_page = PAGE_SIZE >> blkbits; + + /* + * Map all the blocks into the extent list. This code doesn't try + * to be very smart. + */ + probe_block = 0; + page_no = 0; + last_block = inode->i_size >> blkbits; + while ((probe_block + blocks_per_page) <= last_block && + page_no < sis->max) { + unsigned block_in_page; + sector_t first_block; + + first_block = bmap(inode, probe_block); + if (first_block == 0) + goto bad_bmap; + + /* + * It must be PAGE_SIZE aligned on-disk + */ + if (first_block & (blocks_per_page - 1)) { + probe_block++; + goto reprobe; + } + + for (block_in_page = 1; block_in_page < blocks_per_page; + block_in_page++) { + sector_t block; + + block = bmap(inode, probe_block + block_in_page); + if (block == 0) + goto bad_bmap; + if (block != first_block + block_in_page) { + /* Discontiguity */ + probe_block++; + goto reprobe; + } + } + + /* + * We found a PAGE_SIZE-length, PAGE_SIZE-aligned run of blocks + */ + ret = add_swap_extent(sis, page_no, 1, + first_block >> (PAGE_SHIFT - blkbits)); + if (ret) + goto out; + page_no++; + probe_block += blocks_per_page; +reprobe: + continue; + } + ret = 0; + if (page_no == 0) + ret = -EINVAL; + sis->max = page_no; + sis->highest_bit = page_no - 1; +done: + sis->curr_swap_extent = list_entry(sis->extent_list.prev, + struct swap_extent, list); + goto out; +bad_bmap: + printk(KERN_ERR "swapon: swapfile has holes\n"); + ret = -EINVAL; +out: + return ret; +} + asmlinkage long sys_swapoff(const char * specialfile) { struct swap_info_struct * p = NULL; @@ -741,7 +936,6 @@ asmlinkage long sys_swapoff(const char * if (err) goto out; - lock_kernel(); prev = -1; swap_list_lock(); for (type = swap_list.head; type >= 0; type = swap_info[type].next) { @@ -771,9 +965,7 @@ asmlinkage long sys_swapoff(const char * total_swap_pages -= p->pages; p->flags &= ~SWP_WRITEOK; swap_list_unlock(); - unlock_kernel(); err = try_to_unuse(type); - lock_kernel(); if (err) { /* re-insert swap space back into swap_list */ swap_list_lock(); @@ -799,6 +991,7 @@ asmlinkage long sys_swapoff(const char * swap_map = p->swap_map; p->swap_map = NULL; p->flags = 0; + destroy_swap_extents(p); swap_device_unlock(p); swap_list_unlock(); vfree(swap_map); @@ -812,7 +1005,6 @@ asmlinkage long sys_swapoff(const char * err = 0; out_dput: - unlock_kernel(); path_release(&nd); out: return err; @@ -866,12 +1058,12 @@ int get_swaparea_info(char *buf) asmlinkage long sys_swapon(const char * specialfile, int swap_flags) { struct swap_info_struct * p; - char *name; + char *name = NULL; struct block_device *bdev = NULL; struct file *swap_file = NULL; struct address_space *mapping; unsigned int type; - int i, j, prev; + int i, prev; int error; static int least_priority = 0; union swap_header *swap_header = 0; @@ -880,10 +1072,10 @@ asmlinkage long sys_swapon(const char * unsigned long maxpages = 1; int swapfilesize; unsigned short *swap_map; - + struct page *page = NULL; + if (!capable(CAP_SYS_ADMIN)) return -EPERM; - lock_kernel(); swap_list_lock(); p = swap_info; for (type = 0 ; type < nr_swapfiles ; type++,p++) @@ -896,7 +1088,9 @@ asmlinkage long sys_swapon(const char * } if (type >= nr_swapfiles) nr_swapfiles = type+1; + INIT_LIST_HEAD(&p->extent_list); p->flags = SWP_USED; + p->nr_extents = 0; p->swap_file = NULL; p->old_block_size = 0; p->swap_map = NULL; @@ -917,7 +1111,6 @@ asmlinkage long sys_swapon(const char * if (IS_ERR(name)) goto bad_swap_2; swap_file = filp_open(name, O_RDWR, 0); - putname(name); error = PTR_ERR(swap_file); if (IS_ERR(swap_file)) { swap_file = NULL; @@ -939,8 +1132,12 @@ asmlinkage long sys_swapon(const char * PAGE_SIZE); if (error < 0) goto bad_swap; - } else if (!S_ISREG(swap_file->f_dentry->d_inode->i_mode)) + p->bdev = bdev; + } else if (S_ISREG(swap_file->f_dentry->d_inode->i_mode)) { + p->bdev = swap_file->f_dentry->d_inode->i_sb->s_bdev; + } else { goto bad_swap; + } mapping = swap_file->f_dentry->d_inode->i_mapping; swapfilesize = mapping->host->i_size >> PAGE_SHIFT; @@ -954,15 +1151,20 @@ asmlinkage long sys_swapon(const char * goto bad_swap; } - swap_header = (void *) __get_free_page(GFP_USER); - if (!swap_header) { - printk("Unable to start swapping: out of memory :-)\n"); - error = -ENOMEM; + /* + * Read the swap header. + */ + page = read_cache_page(mapping, 0, + (filler_t *)mapping->a_ops->readpage, swap_file); + if (IS_ERR(page)) { + error = PTR_ERR(page); goto bad_swap; } - - lock_page(virt_to_page(swap_header)); - rw_swap_page_nolock(READ, swp_entry(type,0), (char *) swap_header); + wait_on_page_locked(page); + if (!PageUptodate(page)) + goto bad_swap; + kmap(page); + swap_header = page_address(page); if (!memcmp("SWAP-SPACE",swap_header->magic.magic,10)) swap_header_version = 1; @@ -976,33 +1178,10 @@ asmlinkage long sys_swapon(const char * switch (swap_header_version) { case 1: - memset(((char *) swap_header)+PAGE_SIZE-10,0,10); - j = 0; - p->lowest_bit = 0; - p->highest_bit = 0; - for (i = 1 ; i < 8*PAGE_SIZE ; i++) { - if (test_bit(i,(unsigned long *) swap_header)) { - if (!p->lowest_bit) - p->lowest_bit = i; - p->highest_bit = i; - maxpages = i+1; - j++; - } - } - nr_good_pages = j; - p->swap_map = vmalloc(maxpages * sizeof(short)); - if (!p->swap_map) { - error = -ENOMEM; - goto bad_swap; - } - for (i = 1 ; i < maxpages ; i++) { - if (test_bit(i,(unsigned long *) swap_header)) - p->swap_map[i] = 0; - else - p->swap_map[i] = SWAP_MAP_BAD; - } - break; - + printk(KERN_ERR "version 0 swap is no longer supported. " + "Use mkswap -v1 %s\n", name); + error = -EINVAL; + goto bad_swap; case 2: /* Check the swap header's sub-version and the size of the swap file and bad block lists */ @@ -1058,15 +1237,20 @@ asmlinkage long sys_swapon(const char * goto bad_swap; } p->swap_map[0] = SWAP_MAP_BAD; + p->max = maxpages; + p->pages = nr_good_pages; + + if (setup_swap_extents(p)) + goto bad_swap; + swap_list_lock(); swap_device_lock(p); - p->max = maxpages; p->flags = SWP_ACTIVE; - p->pages = nr_good_pages; nr_swap_pages += nr_good_pages; total_swap_pages += nr_good_pages; - printk(KERN_INFO "Adding Swap: %dk swap-space (priority %d)\n", - nr_good_pages<<(PAGE_SHIFT-10), p->prio); + printk(KERN_INFO "Adding %dk swap on %s. Priority:%d extents:%d\n", + nr_good_pages<<(PAGE_SHIFT-10), name, + p->prio, p->nr_extents); /* insert swap space into swap_list: */ prev = -1; @@ -1100,14 +1284,18 @@ bad_swap_2: if (!(swap_flags & SWAP_FLAG_PREFER)) ++least_priority; swap_list_unlock(); + destroy_swap_extents(p); if (swap_map) vfree(swap_map); if (swap_file && !IS_ERR(swap_file)) filp_close(swap_file, NULL); out: - if (swap_header) - free_page((long) swap_header); - unlock_kernel(); + if (page && !IS_ERR(page)) { + kunmap(page); + page_cache_release(page); + } + if (name) + putname(name); return error; } @@ -1176,78 +1364,10 @@ bad_file: goto out; } -/* - * Page lock needs to be held in all cases to prevent races with - * swap file deletion. - */ -int swap_count(struct page *page) -{ - struct swap_info_struct * p; - unsigned long offset, type; - swp_entry_t entry; - int retval = 0; - - entry.val = page->index; - if (!entry.val) - goto bad_entry; - type = swp_type(entry); - if (type >= nr_swapfiles) - goto bad_file; - p = type + swap_info; - offset = swp_offset(entry); - if (offset >= p->max) - goto bad_offset; - if (!p->swap_map[offset]) - goto bad_unused; - retval = p->swap_map[offset]; -out: - return retval; - -bad_entry: - printk(KERN_ERR "swap_count: null entry!\n"); - goto out; -bad_file: - printk(KERN_ERR "swap_count: %s%08lx\n", Bad_file, entry.val); - goto out; -bad_offset: - printk(KERN_ERR "swap_count: %s%08lx\n", Bad_offset, entry.val); - goto out; -bad_unused: - printk(KERN_ERR "swap_count: %s%08lx\n", Unused_offset, entry.val); - goto out; -} - -/* - * Prior swap_duplicate protects against swap device deletion. - */ -void get_swaphandle_info(swp_entry_t entry, unsigned long *offset, - struct inode **swapf) +struct swap_info_struct * +get_swap_info_struct(unsigned type) { - unsigned long type; - struct swap_info_struct *p; - - type = swp_type(entry); - if (type >= nr_swapfiles) { - printk(KERN_ERR "rw_swap_page: %s%08lx\n", Bad_file, entry.val); - return; - } - - p = &swap_info[type]; - *offset = swp_offset(entry); - if (*offset >= p->max && *offset != 0) { - printk(KERN_ERR "rw_swap_page: %s%08lx\n", Bad_offset, entry.val); - return; - } - if (p->swap_map && !p->swap_map[*offset]) { - printk(KERN_ERR "rw_swap_page: %s%08lx\n", Unused_offset, entry.val); - return; - } - if (!(p->flags & SWP_USED)) { - printk(KERN_ERR "rw_swap_page: %s%08lx\n", Unused_file, entry.val); - return; - } - - *swapf = p->swap_file->f_dentry->d_inode; + return &swap_info[type]; } /* --- 2.5.22/mm/swap_state.c~swap-bio Sun Jun 16 22:50:18 2002 +++ 2.5.22-akpm/mm/swap_state.c Sun Jun 16 22:50:18 2002 @@ -14,54 +14,27 @@ #include <linux/init.h> #include <linux/pagemap.h> #include <linux/smp_lock.h> -#include <linux/buffer_head.h> /* block_sync_page()/try_to_free_buffers() */ +#include <linux/buffer_head.h> /* block_sync_page() */ #include <asm/pgtable.h> /* - * We may have stale swap cache pages in memory: notice - * them here and get rid of the unnecessary final write. - */ -static int swap_writepage(struct page *page) -{ - if (remove_exclusive_swap_page(page)) { - unlock_page(page); - return 0; - } - rw_swap_page(WRITE, page); - return 0; -} - -/* - * swapper_space doesn't have a real inode, so it gets a special vm_writeback() - * so we don't need swap special cases in generic_vm_writeback(). - * - * Swap pages are PageLocked and PageWriteback while under writeout so that - * memory allocators will throttle against them. - */ -static int swap_vm_writeback(struct page *page, int *nr_to_write) -{ - struct address_space *mapping = page->mapping; - - unlock_page(page); - return generic_writepages(mapping, nr_to_write); -} - -static struct address_space_operations swap_aops = { - vm_writeback: swap_vm_writeback, - writepage: swap_writepage, - sync_page: block_sync_page, - set_page_dirty: __set_page_dirty_nobuffers, -}; - -/* * swapper_inode doesn't do anything much. It is really only here to * avoid some special-casing in other parts of the kernel. + * + * We set i_size to "infinity" to keep the page I/O functions happy. The swap + * block allocator makes sure that allocations are in-range. A strange + * number is chosen to prevent various arith overflows elsewhere. For example, + * `lblock' in block_read_full_page(). */ static struct inode swapper_inode = { - i_mapping: &swapper_space, + i_mapping: &swapper_space, + i_size: PAGE_SIZE * 0xffffffffLL, + i_blkbits: PAGE_SHIFT, }; +extern struct address_space_operations swap_aops; + struct address_space swapper_space = { page_tree: RADIX_TREE_INIT(GFP_ATOMIC), page_lock: RW_LOCK_UNLOCKED, @@ -149,14 +122,9 @@ void delete_from_swap_cache(struct page { swp_entry_t entry; - /* - * I/O should have completed and nobody can have a ref against the - * page's buffers - */ BUG_ON(!PageLocked(page)); BUG_ON(PageWriteback(page)); - if (page_has_buffers(page) && !try_to_free_buffers(page)) - BUG(); + BUG_ON(page_has_buffers(page)); entry.val = page->index; @@ -222,16 +190,9 @@ int move_from_swap_cache(struct page *pa void **pslot; int err; - /* - * Drop the buffers now, before taking the page_lock. Because - * mapping->private_lock nests outside mapping->page_lock. - * This "must" succeed. The page is locked and all I/O has completed - * and nobody else has a ref against its buffers. - */ BUG_ON(!PageLocked(page)); BUG_ON(PageWriteback(page)); - if (page_has_buffers(page) && !try_to_free_buffers(page)) - BUG(); + BUG_ON(page_has_buffers(page)); write_lock(&swapper_space.page_lock); write_lock(&mapping->page_lock); @@ -361,7 +322,7 @@ struct page * read_swap_cache_async(swp_ /* * Initiate read into locked page and return. */ - rw_swap_page(READ, new_page); + swap_readpage(NULL, new_page); return new_page; } } while (err != -ENOENT && err != -ENOMEM); - - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |