[PATCH] SMP iowait stats

[PATCH] SMP iowait stats

Patch from William Lee Irwin III <wli@holomorphy.com>

Idle time accounting is disturbed by the iowait statistics, for several
reasons:

(1) iowait time is not subdivided among cpus.
        The only way the distinction between idle time subtracted from
        cpus (in order to be accounted as iowait) can be made is by
        summing counters for a total and dividing the individual tick
        counters by the proportions. Any tick type resolution which is
        not properly per-cpu breaks this, meaning that cpus which are
        entirely idle, when any iowait is present on the system, will
        have all idle ticks accounted to iowait instead of true idle time.

(2) kstat_read_proc() misreports iowait time
        The idle tick counter is passed twice to the sprintf(), once
        in the idle tick position, and once in the iowait tick position.

(3) performance enhancement
        The O(1) scheduler was very carefully constructed to perform
        accesses only to localized cachelines whenever possible. The
        global counter violates one of its core design principles,
        and the localization of "most" accesses is in greater harmony
        with its overall design and provides (at the very least) a
        qualitative performance improvement wrt. cache.

The method of correcting this is simple: embed an atomic iowait counter
in the runqueues, find the runqueue being manipulated in io_schedule(),
increment its atomic counter prior to schedule(), and decrement it
after returning from schedule(), which is guaranteed to be the same one,
as the counter incremented is tracked as a variable local to the procedure.
Then simply sum to obtain a global iowait statistic.

(Atomicity is required as the post-wait decrement may occur on a different
cpu from the one owning the counter.)

io_schedule() and io_schedule_timeout() are moved to sched.c as they must
access the runqueues, which are private to sched.c, and nr_iowait() is
created in order to export the sum of all runqueues' nr_iowait().