 | | | high context switching and high load averages slowing down sy stem | high context switching and high load averages slowing down sy stem 2005-08-01 - By Magnus Andersen
Back
Performance today, all day, has been great and everyone has been
happy. I believe that implementing the hugetlb is what took care of
it.
Thanks all, especially Tom Sightler and Rafael Ferreira.
Magnus
On 8/1/05, Tom Sightler <ttsig@(protected)> wrote:
> Those numbers look reasonably good. I don't know what type of storage
> array you have on the back end but for a mid-tier array I would so that
> sustaining 90MB/sec is pretty good. I have an EMC CX400 that can
> sustain 70-75MB/sec assuming it's setup well, and testing on my new
> CX700 would indicate that I may be able to do twice that much, but
> that's with a lot a manual tweaking and tuning. Out of the box I get
> roughly what you're seeing, around 95 MB/sec.
>
> The IOzone numbers also look decent, once again without knowing your
> exact hardware and storage array configuration. Based on those numbers,
> and the amount of IO's that your system normally seems to generate based
> on your previously posted logs, I would think that your IO system would
> be able to handle the load easily. Admittedly that's only a very basic
> analysis and there could be other things to consider, but I would still
> say we can safely look at places other than the IO subsystem if your
> problem continues.
>
> I'll be interested to hear how the switch to hugetlb works for you.
> Hopefully that will make a big difference. There are certain loads
> where it's affect can be huge, perhaps your seeing one.
>
> Later,
> Tom
>
>
> On Sun, 2005-07-31 at 20:15 -0400, Magnus Andersen wrote:
> > Here is the information from hdparm -tT
> > DISKGROUP 1
> > /dev/sda:
> > Timing buffer-cache reads: 5024 MB in 2.00 seconds = 2512.00 MB/sec
> > Timing buffered disk reads: 282 MB in 3.01 seconds = 93.69 MB/sec
> >
> > /dev/sda:
> > Timing buffer-cache reads: 3940 MB in 2.00 seconds = 1970.00 MB/sec
> > Timing buffered disk reads: 284 MB in 3.01 seconds = 94.35 MB/sec
> >
> > /dev/sda:
> > Timing buffer-cache reads: 4396 MB in 2.00 seconds = 2198.00 MB/sec
> > Timing buffered disk reads: 284 MB in 3.01 seconds = 94.35 MB/sec
> >
> > /dev/sda:
> > Timing buffer-cache reads: 4424 MB in 2.00 seconds = 2212.00 MB/sec
> > Timing buffered disk reads: 284 MB in 3.01 seconds = 94.35 MB/sec
> >
> > /dev/sda:
> > Timing buffer-cache reads: 4380 MB in 2.00 seconds = 2190.00 MB/sec
> > Timing buffered disk reads: 278 MB in 3.01 seconds = 92.36 MB/sec
> >
> > DISKGROUP 2
> > /dev/sdb:
> > Timing buffer-cache reads: 4440 MB in 2.00 seconds = 2220.00 MB/sec
> > Timing buffered disk reads: 274 MB in 3.01 seconds = 91.03 MB/sec
> >
> > /dev/sdb:
> > Timing buffer-cache reads: 4636 MB in 2.00 seconds = 2318.00 MB/sec
> > Timing buffered disk reads: 280 MB in 3.00 seconds = 93.33 MB/sec
> >
> > /dev/sdb:
> > Timing buffer-cache reads: 4356 MB in 2.00 seconds = 2178.00 MB/sec
> > Timing buffered disk reads: 278 MB in 3.00 seconds = 92.67 MB/sec
> >
> > /dev/sdb:
> > Timing buffer-cache reads: 4448 MB in 2.00 seconds = 2224.00 MB/sec
> > Timing buffered disk reads: 272 MB in 3.00 seconds = 90.67 MB/sec
> >
> > /dev/sdb:
> > Timing buffer-cache reads: 4256 MB in 2.00 seconds = 2128.00 MB/sec
> > Timing buffered disk reads: 266 MB in 3.02 seconds = 88.08 MB/sec
> >
> >
> > I've also attached a IOzone report made on the oradata location. I'm
> > not sure if the values are good or bad. I've configured hugetlb over
> > the weekend and I am going to gather stats on SYS tonight. We'll see
> > how things work in the morning.
> >
> > Thanks,
> > Magnus
> >
> > On 7/29/05, Tom Sightler <ttsig@(protected)> wrote:
> > > On Fri, 2005-07-29 at 12:56 -0400, Magnus Andersen wrote:
> > > > There are NO silly questions...
> > > >
> > > > 1. I have statistics for the two shemas in the database. I've used
> > > > execute dbms_stats.flush_database_monitoring_info;
> > > > execute dbms_stats.alter_schema_tab_monitoring ('$1', TRUE);
> > > > execute dbms_stats.gather_schema_stats(ownname=>'$1',options=>'GATHER
AUTO');
> > > > to gather them. $1 is a parameter provided when I run my shell script.
> > >
> > > Since 9.2.0.5 Oracle seems to recommend gathering statistics on the SYS
> > > schema. This is something many DBA's don't do since before this version
> > > Oracle always recommended the opposite. It might be worth looking into.
> > > We had at least one case where queries went from 2 hours to a few
> > > minutes after gather statistics on SYS.
> > >
> > > Of course, what would be interesting is analyzing the queries during the
> > >
> > > > 2. My SGA is 1.7 GB. I can increase it by lowering the memory map
> > > > base on RH and start if back up. I don't know if that will help.
> > > > When we ran this database on the previous server (HP-UX) it had the
> > > > same size SGA and ran fine.
> > >
> > > Well, SGA is really based on the size of the typical dataset, the number
> > > of users, etc. Since I don't know your environment it's hard to guess
> > > at that. Are you using 9i tuning options (db_cache_size) or classic
> > > options (db_block_buffers, etc).
> > >
> > > > 3. I think I've done all the settings. Here are what I am running
> > > > from sysctl.conf...
> > >
> > > Those certainly look OK.
> > >
> > > > Two things that RedHat has asked me to consider is hugetlb and to
> > > > mount the Oracle shares with rw,async,noatime,data=writeback options.
> > > > I currenlty mount it without the data=writeback options. I've been
> > > > told that the Oracle parameter use_indirect_buffer (which I have to
> > > > set to use hugtlb) is bad for performance. What are you thoughts?
> > >
> > > I can't say that these suggestions are bad, I already run my database
> > > mounts with noatime and data=writeback option (the async option should
> > > be the default as far as I know). That being said I would guess that
> > > wouldn't help much since I don't really see any significant IO on your
> > > system based on output from your previous posts and these parameters are
> > > more likely to help writes while you disk waits are showing on reads.
> > >
> > > Actually, have you tested the performance of your IO system with a
> > > benchmarking tool? Something like IOzone would be useful, but even a
> > > quick hdparm -tT would give you some idea.
> > >
> > > Also, the fact that the processes are in 'D' state, there is a lot of
> > > system time, and yet not much IO wait time, I would also be very
> > > suspicious of networking. I never heard any answer as to whether there
> > > might be a lot of network activity.
> > >
> > > Also, Redhat is incorrect in stating that use_indirect_buffer is a
> > > requirement to use hugetlb. The use_indirect_buffer option is a
> > > requirement to use VLM for a large SGA, and this can indeed have a
> > > negative performance impact, however, hugetlb support does not depend on
> > > this option.
> > >
> > > Simply mount ramfs on /dev/shm and set vm.hugetlb_pool to the
> > > appropriate value (you'll probably have to play to come up with this but
> > > Oracle documents a procedure to calculate this reasonably close) and you
> > > should be good.
> > >
> > > Also, make sure that async IO is enabled, it's not by default on Linux
> > > like other Unix systems.
> > >
> > > My system has an ~2.7GB SGA (using a lowered memory-map rather than the
> > > performance zapping use_indirect_buffer option) and I have the following
> > > setting for hugetlb and Async IO support:
> > >
> > > ########
> > > # Async I/O
> > > # Parameters for enabling Asyn I/O
> > > ########
> > >
> > > disk_asynch_io=true
> > > filesystemio_options=asynch
> > >
> > > ########
> > > # Large SGA Support
> > > # Parameters for enabling a large SGA
> > > ########
> > >
> > > #use_indirect_data_buffers=true
> > >
> > > Notice that use_indirect_data_buffers is disabled. Also, note that you
> > > will need to relink Oracle to actually enable async IO support before
> > > using the above parameters.
> > >
> > > Here is a snippet of my /proc/meminfo showing that hugetlb is being
> > > used:
> > >
> > > HugePages_Total: 1296
> > > HugePages_Free: 2
> > > Hugepagesize: 2048 kB
> > >
> > > So my system is using 1294 HugePages of the 1296 I allocated. Also, you
> > > can monitor /proc/slabinfo to determine if Async IO is actually working.
> > > My output looks like:
> > >
> > > kioctx 1647 1650 128 55 55 1 : 1008 252
> > > kiocb 72426 72930 128 2415 2431 1 : 1008 252
> > >
> > > On a system where Async IO is not working or is disabled it looks more
> > > like this:
> > >
> > > kioctx 0 0 256 0 0 1 : 252 63
> > > kiocb 0 0 192 0 0 1 : 252 63
> > >
> > > Hope this helps.
> > >
> > > Later,
> > > Tom
> > >
> > >
> > >
> > >
> >
> >
>
>
--
Magnus Andersen
Systems Administrator / Oracle DBA
Walker & Associates, Inc.
--
Taroon-list mailing list
Taroon-list@(protected)
http://www.redhat.com/mailman/listinfo/taroon-list
Earn $52 per hosting referral at Lunarpages.
|
|
|