Messages in this thread |  | | | Date | Mon, 26 Oct 2009 15:07:00 -0700 | | From | Mike Travis <> | | Subject | Re: [PATCH 1/8] SGI x86_64 UV: Add limit console output function |
| |
Andi Kleen wrote: > On Mon, Oct 26, 2009 at 11:03:59AM -0700, Mike Travis wrote: >> >> Andi Kleen wrote: >>> Mike Travis <travis@sgi.com> writes: >>> >>>> With a large number of processors in a system there is an excessive amount >>>> of messages sent to the system console. It's estimated that with 4096 >>>> processors in a system, and the console baudrate set to 56K, the startup >>>> messages will take about 84 minutes to clear the serial port. >>>> >>>> This patch adds (for SGI UV only) a kernel start option "limit_console_ >>>> output" (or 'lco' for short), which when set provides the ability to >>>> temporarily reduce the console loglevel during system startup. This allows >>>> informative messages to still be seen on the console without producing >>>> excessive amounts of repetious messages. >>>> >>>> Note that all the messages are still available in the kernel log buffer. >>> I've run into the same problem (kernel log being flooded on large number of CPU thread >>> systems). It's definitely not a UV only problem. Making such a option UV only >>> is definitely not the right approach, if anything it needs to be for everyone. >> I could use something like the MAXSMP config option to enable it...? > > No, it's a problem long before MAXSMP sizes. > >>> Frankly a lot of these messages made sense for debugging at some point, >>> but really don't anymore and should just be removed. >> That they still go to the kernel log buffer means the messages are still >> available for debugging system problems. KDB has a kernel print option if >> you end up there before being able to use 'dmesg'. > > Again they should be just reevaluated and pr_debug()ed or completely > removed. > >>> Also I don't like the defaults of on. It would be better to evaluate if >>> these various messages are really useful and if they are not just remove them. >> I believe most distros already do that by setting the loglevel argument >> (but I could be wrong since I haven't looked at too many of them.) > > Even spamming dmesg is a problem. loglevel doesn't fix that. > >>> For example do we really need the scheduler debug messages by default? >> This was the most painful message at Nasa (which has a 2k cpu system). It took >> well over an hour for these scheduler messages to print, just because we wanted >> to get some other DEBUG prints. > > They should be just removed.
I had changed this to CONFIG_DEBUG_SCHED at one time. Perhaps this would be acceptible?
> >>> Or do we really need to print the caches for each CPU at boot? The information >>> is in sysfs anyways and rarely changes (I added this originally on 64bit, >>> but in hindsight it was a bad idea) >> I was attempting not to decide whether each message was pertinent, only if it >> was redundant. > > You should decide or at least ask whoever added it > > ("How many bugs did you fix with that message last year?" If the answer > is < 10 or so, remove it)
Ok.
>>> I don't think it makes much sense to print more than 2-3 lines for each CPU boot >>> for example. >> That would still be 4 to 12 thousand lines of information which, as you say is >> available by other means. > > A simple checkpoint for debugging is not available by other means. > > The cache, mce etc. information is. > > For the checkpoint problem on CPU boot it might be reasonable > to write them into a special buffer and only print it when the other > CPU does not come up (BP detects a time out) > > With that a single line of per CPU output should be feasible without > losing any debuggability. > > In fact debuggability could be improved by putting the output > at better strategic points instead of the ad-hoc way it is currently. > > -Andi >
Ok, thanks for the feedback. I'll see about reducing the output more intelligently for CPU's (as per Ingo's suggestions as well.)
Mike
|  |