lkml.org 
[lkml]   [2006]   [Jun]   [23]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    SubjectRe: [PATCH 1/5] mm: tracking shared dirty pages
    From
    Date

    From: Peter Zijlstra <a.p.zijlstra@chello.nl>

    Tracking of dirty pages in shared writeable mmap()s.

    The idea is simple: write protect clean shared writeable pages,
    catch the write-fault, make writeable and set dirty. On page write-back
    clean all the PTE dirty bits and write protect them once again.

    The implementation is a tad harder, mainly because the default
    backing_dev_info capabilities were too loosely maintained. Hence it is
    not enough to test the backing_dev_info for cap_account_dirty.

    The current heuristic is as follows, a VMA is eligible when:
    - its shared writeable
    (vm_flags & (VM_WRITE|VM_SHARED)) == (VM_WRITE|VM_SHARED)
    - it is not a 'special' mapping
    (vm_flags & (VM_PFNMAP|VM_INSERTPAGE)) == 0
    - the backing_dev_info is cap_account_dirty
    mapping_cap_account_dirty(vma->vm_file->f_mapping)
    - f_op->mmap() didn't change the default page protection

    Page from remap_pfn_range() are explicitly excluded because their
    COW semantics are already horrid enough (see vm_normal_page() in
    do_wp_page()) and because they don't have a backing store anyway.

    mprotect() is taught about the new behaviour as well. However it fudges
    the last.

    Cleaning the pages on write-back is done with page_mkclean() a new
    rmap call. It can be called on any page, but is currently only
    implemented for mapped pages, if the page is found the be of a
    trackable VMA it will also wrprotect the PTE.



    Bah Bah Bah, why didn't the page_mkwrite() patch re-protect clean pages?
    And is it a Bad-Thing (tm) that that can happen now?



    Finally, in fs/buffers.c:try_to_free_buffers(); remove clear_page_dirty()
    from under ->private_lock. This seems to be safe, since ->private_lock
    is used to serialize access to the buffers, not the page itself.
    This is needed because clear_page_dirty() will call into page_mkclean()
    and would thereby violate locking order.

    Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
    ---
    Changes in -v12

    - Linus suggested a nice replacement for the beast condition

    Changes in -v11

    - small cleanups
    - pulled page_mkclean back under mapping_cap_account_dirty()

    Changes in -v10

    - 2.6.17-mm1
    - Drop the ugly duckling pgprotting, Hugh suggested resetting the
    vma->vm_page_prot when f_op->mmap() didn't modify it. If it were
    modified we're not interested anyway.
    - abandon is_shared_writable() because its actually spelled writeable
    and it didn't actually mean that any more.
    - Comments all round.

    Changes in -v9

    - respin against latest -mm.

    Changes in -v8

    - access_process_vm() and other force users of get_user_pages() can
    induce COW of read-only shared mappings.

    Changes in -v7

    - changed is_shared_writable() to exclude VM_PFNMAP'ed regions.
    - Hugh's tiresome problem wasn't fully solved, now using the ugly duckling
    method.

    Changes in -v6

    - make page_mkclean_one() modify the pte more like change_pte_range()
    (suggested by Christoph Lameter)
    - made is_shared_writable() take vm_flags, it now resembles is_cow_mapping().
    - fixed the mprotect() bug (spotted by Hugh Dickins)
    - hopefully fixed the tiresome issue of do_mmap_pgoff() trampling on
    driver specific vm_page_prot settings (spotted by Hugh Dickins)
    - made a new version of the page_mkwrite() patch to go on top of all this.
    This so that Linus could merge this very early on in 2.6.18.

    Changes in -v5

    - rename page_wrprotect() to page_mkclean() (suggested by Nick Piggin)
    - added comment to test_clear_page_dirty() (Andrew Morton)
    - cleanup page_wrprotect() (Andrew Morton)
    - renamed VM_SharedWritable() to is_shared_writable()
    - fs/buffers.c try_to_free_buffers(): remove clear_page_dirty() from under
    ->private_lock. This seems to be save, since ->private_lock is used to
    serialize access to the buffers, not the page itself.
    - rebased on top of David Howells' page_mkwrite() patch.

    Changes in -v4:

    - small cleanup as suggested by Christoph Lameter.

    Changes in -v3:

    - move set_page_dirty() outside pte lock (suggested by Christoph Lameter)

    Changes in -v2:

    - only wrprotect pages from dirty capable mappings. (Nick Piggin)
    - move the writefault handling from do_wp_page() into handle_pte_fault().
    (Nick Piggin)
    - revert to the old install_page interface. (Nick Piggin)
    - also clear the pte dirty bit when we make pages read-only again.
    (spotted by Rik van Riel)
    - make page_wrprotect() return the number of reprotected ptes.

    fs/buffer.c | 2 -
    include/linux/mm.h | 34 ++++++++++++++++++++++++
    include/linux/rmap.h | 8 +++++
    mm/memory.c | 29 ++++++++++++++++-----
    mm/mmap.c | 10 ++-----
    mm/mprotect.c | 22 ++++++----------
    mm/page-writeback.c | 15 ++++++++--
    mm/rmap.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++
    8 files changed, 161 insertions(+), 29 deletions(-)

    Index: 2.6-mm/mm/memory.c
    ===================================================================
    --- 2.6-mm.orig/mm/memory.c 2006-06-24 01:39:29.000000000 +0200
    +++ 2.6-mm/mm/memory.c 2006-06-24 01:40:20.000000000 +0200
    @@ -1458,14 +1458,19 @@ static int do_wp_page(struct mm_struct *
    {
    struct page *old_page, *new_page;
    pte_t entry;
    - int reuse, ret = VM_FAULT_MINOR;
    + int reuse = 0, ret = VM_FAULT_MINOR;
    + struct page *dirty_page = NULL;

    old_page = vm_normal_page(vma, address, orig_pte);
    if (!old_page)
    goto gotten;

    - if (unlikely((vma->vm_flags & (VM_SHARED|VM_WRITE)) ==
    - (VM_SHARED|VM_WRITE))) {
    + /*
    + * Only catch write-faults on shared writable pages, read-only
    + * shared pages can get COWed by get_user_pages(.write=1, .force=1).
    + */
    + if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
    + (VM_WRITE|VM_SHARED))) {
    if (vma->vm_ops && vma->vm_ops->page_mkwrite) {
    /*
    * Notify the address space that the page is about to
    @@ -1494,13 +1499,12 @@ static int do_wp_page(struct mm_struct *
    if (!pte_same(*page_table, orig_pte))
    goto unlock;
    }
    -
    + dirty_page = old_page;
    + get_page(dirty_page);
    reuse = 1;
    } else if (PageAnon(old_page) && !TestSetPageLocked(old_page)) {
    reuse = can_share_swap_page(old_page);
    unlock_page(old_page);
    - } else {
    - reuse = 0;
    }

    if (reuse) {
    @@ -1566,6 +1570,10 @@ gotten:
    page_cache_release(old_page);
    unlock:
    pte_unmap_unlock(page_table, ptl);
    + if (dirty_page) {
    + set_page_dirty(dirty_page);
    + put_page(dirty_page);
    + }
    return ret;
    oom:
    if (old_page)
    @@ -2098,6 +2106,7 @@ static int do_no_page(struct mm_struct *
    unsigned int sequence = 0;
    int ret = VM_FAULT_MINOR;
    int anon = 0;
    + struct page *dirty_page = NULL;

    pte_unmap(page_table);
    BUG_ON(vma->vm_flags & VM_PFNMAP);
    @@ -2192,6 +2201,10 @@ retry:
    } else {
    inc_mm_counter(mm, file_rss);
    page_add_file_rmap(new_page);
    + if (write_access) {
    + dirty_page = new_page;
    + get_page(dirty_page);
    + }
    }
    } else {
    /* One of our sibling threads was faster, back out. */
    @@ -2204,6 +2217,10 @@ retry:
    lazy_mmu_prot_update(entry);
    unlock:
    pte_unmap_unlock(page_table, ptl);
    + if (dirty_page) {
    + set_page_dirty(dirty_page);
    + put_page(dirty_page);
    + }
    return ret;
    oom:
    page_cache_release(new_page);
    Index: 2.6-mm/mm/mmap.c
    ===================================================================
    --- 2.6-mm.orig/mm/mmap.c 2006-06-24 01:39:29.000000000 +0200
    +++ 2.6-mm/mm/mmap.c 2006-06-24 01:40:20.000000000 +0200
    @@ -1090,12 +1090,6 @@ munmap_back:
    goto free_vma;
    }

    - /* Don't make the VMA automatically writable if it's shared, but the
    - * backer wishes to know when pages are first written to */
    - if (vma->vm_ops && vma->vm_ops->page_mkwrite)
    - vma->vm_page_prot =
    - protection_map[vm_flags & (VM_READ|VM_WRITE|VM_EXEC)];
    -
    /* We set VM_ACCOUNT in a shared mapping's vm_flags, to inform
    * shmem_zero_setup (perhaps called through /dev/zero's ->mmap)
    * that memory reservation must be checked; but that reservation
    @@ -1113,6 +1107,10 @@ munmap_back:
    pgoff = vma->vm_pgoff;
    vm_flags = vma->vm_flags;

    + if (vma_wants_writenotify(vma))
    + vma->vm_page_prot =
    + protection_map[vm_flags & (VM_READ|VM_WRITE|VM_EXEC)];
    +
    if (!file || !vma_merge(mm, prev, addr, vma->vm_end,
    vma->vm_flags, NULL, file, pgoff, vma_policy(vma))) {
    file = vma->vm_file;
    Index: 2.6-mm/mm/mprotect.c
    ===================================================================
    --- 2.6-mm.orig/mm/mprotect.c 2006-06-24 01:39:29.000000000 +0200
    +++ 2.6-mm/mm/mprotect.c 2006-06-24 01:40:20.000000000 +0200
    @@ -123,8 +123,7 @@ mprotect_fixup(struct vm_area_struct *vm
    unsigned long oldflags = vma->vm_flags;
    long nrpages = (end - start) >> PAGE_SHIFT;
    unsigned long charged = 0;
    - unsigned int mask;
    - pgprot_t newprot;
    + unsigned int mask = VM_READ|VM_WRITE|VM_EXEC|VM_SHARED;
    pgoff_t pgoff;
    int error;

    @@ -176,24 +175,21 @@ mprotect_fixup(struct vm_area_struct *vm
    }

    success:
    - /* Don't make the VMA automatically writable if it's shared, but the
    - * backer wishes to know when pages are first written to */
    - mask = VM_READ|VM_WRITE|VM_EXEC|VM_SHARED;
    - if (vma->vm_ops && vma->vm_ops->page_mkwrite)
    - mask &= ~VM_SHARED;
    -
    - newprot = protection_map[newflags & mask];
    -
    /*
    * vm_flags and vm_page_prot are protected by the mmap_sem
    * held in write mode.
    */
    vma->vm_flags = newflags;
    - vma->vm_page_prot = newprot;
    + vma->vm_page_prot = protection_map[newflags & mask];
    + if (vma_wants_writenotify(vma)) {
    + mask &= ~VM_SHARED;
    + vma->vm_page_prot = protection_map[newflags & mask];
    + }
    +
    if (is_vm_hugetlb_page(vma))
    - hugetlb_change_protection(vma, start, end, newprot);
    + hugetlb_change_protection(vma, start, end, vma->vm_page_prot);
    else
    - change_protection(vma, start, end, newprot);
    + change_protection(vma, start, end, vma->vm_page_prot);
    vm_stat_account(mm, oldflags, vma->vm_file, -nrpages);
    vm_stat_account(mm, newflags, vma->vm_file, nrpages);
    return 0;
    Index: 2.6-mm/mm/page-writeback.c
    ===================================================================
    --- 2.6-mm.orig/mm/page-writeback.c 2006-06-24 01:39:29.000000000 +0200
    +++ 2.6-mm/mm/page-writeback.c 2006-06-24 01:40:20.000000000 +0200
    @@ -29,6 +29,7 @@
    #include <linux/sysctl.h>
    #include <linux/cpu.h>
    #include <linux/syscalls.h>
    +#include <linux/rmap.h>

    /*
    * The maximum number of pages to writeout in a single bdflush/kupdate
    @@ -566,7 +567,7 @@ int do_writepages(struct address_space *
    return 0;
    wbc->for_writepages = 1;
    if (mapping->a_ops->writepages)
    - ret = mapping->a_ops->writepages(mapping, wbc);
    + ret = mapping->a_ops->writepages(mapping, wbc);
    else
    ret = generic_writepages(mapping, wbc);
    wbc->for_writepages = 0;
    @@ -728,8 +729,14 @@ int test_clear_page_dirty(struct page *p
    page_index(page),
    PAGECACHE_TAG_DIRTY);
    write_unlock_irqrestore(&mapping->tree_lock, flags);
    - if (mapping_cap_account_dirty(mapping))
    + /*
    + * We can continue to use `mapping' here because the
    + * page is locked, which pins the address_space
    + */
    + if (mapping_cap_account_dirty(mapping)) {
    + page_mkclean(page);
    dec_page_state(nr_dirty);
    + }
    return 1;
    }
    write_unlock_irqrestore(&mapping->tree_lock, flags);
    @@ -759,8 +766,10 @@ int clear_page_dirty_for_io(struct page

    if (mapping) {
    if (TestClearPageDirty(page)) {
    - if (mapping_cap_account_dirty(mapping))
    + if (mapping_cap_account_dirty(mapping)) {
    + page_mkclean(page);
    dec_page_state(nr_dirty);
    + }
    return 1;
    }
    return 0;
    Index: 2.6-mm/mm/rmap.c
    ===================================================================
    --- 2.6-mm.orig/mm/rmap.c 2006-06-24 01:39:29.000000000 +0200
    +++ 2.6-mm/mm/rmap.c 2006-06-24 01:40:20.000000000 +0200
    @@ -53,6 +53,7 @@
    #include <linux/rmap.h>
    #include <linux/rcupdate.h>
    #include <linux/module.h>
    +#include <linux/backing-dev.h>

    #include <asm/tlbflush.h>

    @@ -434,6 +435,75 @@ int page_referenced(struct page *page, i
    return referenced;
    }

    +static int page_mkclean_one(struct page *page, struct vm_area_struct *vma, int protect)
    +{
    + struct mm_struct *mm = vma->vm_mm;
    + unsigned long address;
    + pte_t *pte, entry;
    + spinlock_t *ptl;
    + int ret = 0;
    +
    + address = vma_address(page, vma);
    + if (address == -EFAULT)
    + goto out;
    +
    + pte = page_check_address(page, mm, address, &ptl);
    + if (!pte)
    + goto out;
    +
    + if (!(pte_dirty(*pte) || (protect && pte_write(*pte))))
    + goto unlock;
    +
    + entry = ptep_get_and_clear(mm, address, pte);
    + entry = pte_mkclean(entry);
    + if (protect)
    + entry = pte_wrprotect(entry);
    + ptep_establish(vma, address, pte, entry);
    + lazy_mmu_prot_update(entry);
    + ret = 1;
    +
    +unlock:
    + pte_unmap_unlock(pte, ptl);
    +out:
    + return ret;
    +}
    +
    +static int page_mkclean_file(struct address_space *mapping, struct page *page)
    +{
    + pgoff_t pgoff = page->index << (PAGE_CACHE_SHIFT - PAGE_SHIFT);
    + struct vm_area_struct *vma;
    + struct prio_tree_iter iter;
    + int ret = 0;
    +
    + BUG_ON(PageAnon(page));
    +
    + spin_lock(&mapping->i_mmap_lock);
    + vma_prio_tree_foreach(vma, &iter, &mapping->i_mmap, pgoff, pgoff) {
    + int protect = mapping_cap_account_dirty(mapping) &&
    + ((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
    + (VM_WRITE|VM_SHARED));
    + BUG_ON(vma->vm_flags & (VM_PFNMAP|VM_INSERTPAGE));
    + ret += page_mkclean_one(page, vma, protect);
    + }
    + spin_unlock(&mapping->i_mmap_lock);
    + return ret;
    +}
    +
    +int page_mkclean(struct page *page)
    +{
    + int ret = 0;
    +
    + BUG_ON(!PageLocked(page));
    +
    + if (page_mapped(page)) {
    + struct address_space *mapping = page_mapping(page);
    + if (mapping)
    + ret = page_mkclean_file(mapping, page);
    + }
    +
    + return ret;
    +}
    +
    /**
    * page_set_anon_rmap - setup new anonymous rmap
    * @page: the page to add the mapping to
    Index: 2.6-mm/include/linux/rmap.h
    ===================================================================
    --- 2.6-mm.orig/include/linux/rmap.h 2006-06-24 01:39:29.000000000 +0200
    +++ 2.6-mm/include/linux/rmap.h 2006-06-24 01:40:20.000000000 +0200
    @@ -103,6 +103,14 @@ pte_t *page_check_address(struct page *,
    */
    unsigned long page_address_in_vma(struct page *, struct vm_area_struct *);

    +/*
    + * Cleans the PTEs of shared mappings.
    + * (and since clean PTEs should also be readonly, write protects them too)
    + *
    + * returns the number of cleaned PTEs.
    + */
    +int page_mkclean(struct page *);
    +
    #else /* !CONFIG_MMU */

    #define anon_vma_init() do {} while (0)
    Index: 2.6-mm/fs/buffer.c
    ===================================================================
    --- 2.6-mm.orig/fs/buffer.c 2006-06-24 01:39:29.000000000 +0200
    +++ 2.6-mm/fs/buffer.c 2006-06-24 01:40:20.000000000 +0200
    @@ -2983,6 +2983,7 @@ int try_to_free_buffers(struct page *pag

    spin_lock(&mapping->private_lock);
    ret = drop_buffers(page, &buffers_to_free);
    + spin_unlock(&mapping->private_lock);
    if (ret) {
    /*
    * If the filesystem writes its buffers by hand (eg ext3)
    @@ -2994,7 +2995,6 @@ int try_to_free_buffers(struct page *pag
    */
    clear_page_dirty(page);
    }
    - spin_unlock(&mapping->private_lock);
    out:
    if (buffers_to_free) {
    struct buffer_head *bh = buffers_to_free;
    Index: 2.6-mm/include/linux/mm.h
    ===================================================================
    --- 2.6-mm.orig/include/linux/mm.h 2006-06-24 01:39:29.000000000 +0200
    +++ 2.6-mm/include/linux/mm.h 2006-06-24 01:40:20.000000000 +0200
    @@ -15,6 +15,7 @@
    #include <linux/fs.h>
    #include <linux/mutex.h>
    #include <linux/debug_locks.h>
    +#include <linux/backing-dev.h>

    struct mempolicy;
    struct anon_vma;
    @@ -847,6 +848,39 @@ struct shrinker {
    extern struct shrinker *set_shrinker(int, shrinker_t);
    extern void remove_shrinker(struct shrinker *shrinker);

    +/*
    + * Some shared mappigns will want the pages marked read-only
    + * to track write events. If so, we'll downgrade vm_page_prot
    + * to the private version (using protection_map[] without the
    + * VM_SHARED bit).
    + */
    +static inline int vma_wants_writenotify(struct vm_area_struct *vma)
    +{
    + unsigned int vm_flags = vma->vm_flags;
    +
    + /* If it was private or non-writable, the write bit is already clear */
    + if ((vm_flags & (VM_WRITE|VM_SHARED)) != ((VM_WRITE|VM_SHARED)))
    + return 0;
    +
    + /* The backer wishes to know when pages are first written to? */
    + if (vma->vm_ops && vma->vm_ops->page_mkwrite)
    + return 1;
    +
    + /* The open routine did something to the protections already? */
    + if (pgprot_val(vma->vm_page_prot) !=
    + pgprot_val(protection_map[vm_flags &
    + (VM_READ|VM_WRITE|VM_EXEC|VM_SHARED)]))
    + return 0;
    +
    + /* Specialty mapping? */
    + if (vm_flags & (VM_PFNMAP|VM_INSERTPAGE))
    + return 0;
    +
    + /* Can the mapping track the dirty pages? */
    + return vma->vm_file && vma->vm_file->f_mapping &&
    + mapping_cap_account_dirty(vma->vm_file->f_mapping);
    +}
    +
    extern pte_t *FASTCALL(get_locked_pte(struct mm_struct *mm, unsigned long addr, spinlock_t **ptl));

    int __pud_alloc(struct mm_struct *mm, pgd_t *pgd, unsigned long address);

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2006-06-24 01:57    [W:0.045 / U:0.468 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site