Mailing List
Home
Forum Home
Linux - General Red Hat Linux discussion list
Installation - Getting started with Red Hat Linux
Enterprise Linux 3 - Discussion of Red Hat Enterprise Linux 3 (Taroon)
Red Hat Linux 9 - Discussion of Red Hat Linux 9 (Shrike)
Red Hat Linux 7.2 - Discussion of Red Hat Linux 7.2 (Enigma)
Red Hat Linux 7.3 - Discussion of Red Hat Linux 7.3 (Valhalla)
Apache Web Server
Oracle database, Microsoft SQL server ...
Subjects
• Subject: application/x mplayer2 plugin
• RPM error: db4 error(16) from dbenv >remove: Device or resource
   busy
• Command stream end of file while reading
• Subject: X Windows problem (xauth)
• Subject: Upgrading openoffice 1 1 rpm
• Subject: FTP: connection refused
• Subject: FTP: connection refused
• mount: /dev/cdrom: is not a valid block device
• Dell Precision 650, RedHat 9, no sound
• how to trace the cause resulting in the crash of bind server
• Virus on the list
• UNINSTALL RPM MYSQL
• usb pen drives: mounting as a user
• Subject: broadcom network interface
• make mrproper
• Couldn 't open PID file /var/run/named/named pid Permission denied
• sendmail configuration on redhat
• kernel 2 6 and /dev/sound/mixer not found
• Subject: Promise 378 controller
• Subject: Problem using up2date
• mrtg step by step howto/configuration for a newbie?
• Compiling and Installing Kernel 2 6
• Can 't locate module ppp0, can 't locate module ppp compress 21
• Subject: Lotus Notes under Wine
• HOW I CAN MAKE BOOTABLE FLOPPY DISKET
• /etc/security/limits conf question
• Intel E/1000 driver
• rpm database corrupt
• Command stream end of file while reading
• qla2300 modules
 
"Out of Memory: Killed process " errors on server running Oracle

"Out of Memory: Killed process " errors on server running Oracle

2007-07-12       - By Gene

 Back
Reply:     1     2     3     4     5     6     7     8     9     10     >>  

Hola,

 Every 5-6 days I'm getting "Out of Memory: Killed process" messages on a
Redhat AS 3 server running Oracle 9.20.  These errors will occur for a
couple hours and will eventually lead to Oracle shutting down.  The load on
the server did increase recently, but there's still 1.8GB memory free on the
system during these errors.

 The box is running the smp kernel.  Should I do something like increase
the stack size or use the hugemem kernel?

$ uname -a
Linux prddb02 2.4.21-15.0.3.ELsmp #1 SMP Tue Jun 29 18:04:47 EDT 2004 i686
i686 i386 GNU/Linux

$ ulimit -a
core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) 4
max memory size       (kbytes, -m) unlimited
open files                    (-n) 65536
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) 10240
cpu time             (seconds, -t) unlimited
max user processes            (-u) 16384
virtual memory        (kbytes, -v) unlimited

>From /var/log/messages:

Jul 11 19:06:31 prddb02 kernel: Mem-info:
Jul 11 19:06:31 prddb02 kernel: Zone:DMA freepages:  2549 min:     0
low:     0 high:     0
Jul 11 19:06:31 prddb02 kernel: Zone:Normal freepages:  1254 min:  1279
low:  4544 high:  6304
Jul 11 19:06:31 prddb02 kernel: Zone:HighMem freepages:501745 min:   255
low: 20992 high: 31488
Jul 11 19:06:31 prddb02 kernel: Free pages:      505548 (501745 HighMem)
Jul 11 19:06:31 prddb02 kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:2549
Jul 11 19:06:31 prddb02 kernel:   aa:0 ac:40 id:82 il:73 ic:0 fr:1254
Jul 11 19:06:31 prddb02 kernel:   aa:530276 ac:82904 id:17203 il:6249
ic:8419 fr:501745
Jul 11 19:06:31 prddb02 kernel: 1*4kB 2*8kB 2*16kB 1*32kB 0*64kB 1*128kB
1*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 10196kB)
Jul 11 19:06:31 prddb02 kernel: 137*4kB 121*8kB 45*16kB 19*32kB 6*64kB
2*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 5020kB)
Jul 11 19:06:31 prddb02 kernel: 35001*4kB 11806*8kB 4209*16kB 1479*32kB
1490*64kB 513*128kB 291*256kB 48*512kB 15*1024kB 5*2048kB 335*4096kB =
2006980kB)
Jul 11 19:06:31 prddb02 kernel: Swap cache: add 0, delete 0, find 0/0, race
0+0
Jul 11 19:06:31 prddb02 kernel: 192549 pages of slabcache
Jul 11 19:06:31 prddb02 kernel: 1192 pages of kernel stacks
Jul 11 19:06:31 prddb02 kernel: 0 lowmem pagetables, 127372 highmem
pagetables
Jul 11 19:06:31 prddb02 kernel: Free swap:       8192980kB
Jul 11 19:06:31 prddb02 kernel: 1572864 pages of RAM
Jul 11 19:06:31 prddb02 kernel: 1277920 pages of HIGHMEM
Jul 11 19:06:31 prddb02 kernel: 89860 reserved pages
Jul 11 19:06:31 prddb02 kernel: 24863915 pages shared
Jul 11 19:06:31 prddb02 kernel: 0 pages swap cached
Jul 11 19:06:31 prddb02 kernel: Out of Memory: Killed process 8302 (oracle).

$ sar -r -f /var/log/sa/sa11
10:00:00 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree
kbswpused  %swpused  kbswpcad
05:00:00 PM   1901836   4030180     67.94      4228   1976412
8192980         0      0.00         0
05:10:00 PM   1755672   4176344     70.40      1456   2115832
8192980         0      0.00         0
05:20:00 PM   1901120   4030896     67.95      2952   1976168
8192980         0      0.00         0
05:30:00 PM   1896664   4035352     68.03      2392   1975920
8192980         0      0.00         0
05:39:59 PM   1892232   4039784     68.10      2200   1975900
8192980         0      0.00         0
05:50:00 PM   1889808   4042208     68.14      2408   1976076
8192980         0      0.00         0
06:00:00 PM   1897984   4034032     68.00      2336   1976244
8192980         0      0.00         0
06:10:00 PM   1831708   4100308     69.12      1192   2040788
8192980         0      0.00         0
06:20:01 PM   1832808   4099208     69.10      1372   2040912
8192980         0      0.00         0
06:30:03 PM   1844548   4087468     68.91      1284   2040988
8192980         0      0.00         0
06:40:01 PM   1895092   4036924     68.05      1304   2041120
8192980         0      0.00         0
06:50:00 PM   1909852   4022164     67.80      1300   2041248
8192980         0      0.00         0
07:00:01 PM   1906980   4025036     67.85      1104   2041300
8192980         0      0.00         0
07:10:01 PM   4363412   1568604     26.44     13816    726432
8192980         0      0.00         0

-Gene
gene@(protected)

Hola,<br><br>&nbsp; Every 5-6 days I&#39;m getting &quot;Out of Memory: Killed
process&quot; messages on a Redhat AS 3 server running Oracle 9.20.&nbsp; These
errors will occur for a couple hours and will eventually lead to Oracle
shutting down.&nbsp; The load on the server did increase recently, but there&
#39;s still
1.8GB memory free on the system during these errors.<br><br>&nbsp; The box is
running the smp kernel.&nbsp; Should I do something like increase the stack
size or use the hugemem kernel?<br><br><span style="font-weight: bold;">$ uname
-a
</span><br>Linux prddb02 2.4.21-15.0.3.ELsmp #1 SMP Tue Jun 29 18:04:47 EDT
2004 i686 i686 i386 GNU/Linux<br><br><span style="font-weight: bold;">$ ulimit
-a</span><br>core file size&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (blocks,
-c) 0<br>data seg size&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (kbytes,
-d) unlimited
<br>file size&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&nbsp; (blocks, -f) unlimited<br>max locked memory&nbsp;&nbsp;&nbsp;&nbsp;
(kbytes, -l) 4<br>max memory size&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (kbytes,
-m) unlimited<br>open files&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (-n) 65536<br>pipe
size&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (512 bytes, -p) 8
<br>
stack size&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
(kbytes, -s) 10240<br>cpu time&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp; (seconds, -t) unlimited<br>max user processes&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (-u) 16384<br
>virtual memory&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (kbytes, -v) unlimited
<br><br><span style="font-weight: bold;">

>From /var/log/messages:</span><br><br>Jul 11 19:06:31 prddb02 kernel: Mem-info
:<br>Jul 11 19:06:31 prddb02 kernel: Zone:DMA freepages:&nbsp; 2549 min:&nbsp;
&nbsp;&nbsp;&nbsp; 0 low:&nbsp;&nbsp;&nbsp;&nbsp; 0 high:&nbsp;&nbsp;&nbsp;&nbsp
; 0<br>Jul 11 19:06:31 prddb02 kernel: Zone:Normal freepages:&nbsp; 1254 min:
&nbsp; 1279 low:&nbsp; 4544 high:&nbsp; 6304
<br>Jul 11 19:06:31 prddb02 kernel: Zone:HighMem freepages:501745 min:&nbsp;
&nbsp; 255 low: 20992 high: 31488<br>Jul 11 19:06:31 prddb02 kernel: Free pages:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 505548 (501745 HighMem)<br>Jul 11 19:06:31
prddb02 kernel:&nbsp;&nbsp; aa:0 ac:0 id:0 il:0 ic:0 fr:2549
<br>Jul 11 19:06:31 prddb02 kernel:&nbsp;&nbsp; aa:0 ac:40 id:82 il:73 ic:0 fr
:1254<br>Jul 11 19:06:31 prddb02 kernel:&nbsp;&nbsp; aa:530276 ac:82904 id:17203
il:6249 ic:8419 fr:501745<br>Jul 11 19:06:31 prddb02 kernel: 1*4kB 2*8kB 2*16kB
1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 10196kB)
<br>Jul 11 19:06:31 prddb02 kernel: 137*4kB 121*8kB 45*16kB 19*32kB 6*64kB 2
*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 5020kB)<br>Jul 11 19:06:31
prddb02 kernel: 35001*4kB 11806*8kB 4209*16kB 1479*32kB 1490*64kB 513*128kB 291
*256kB 48*512kB 15*1024kB 5*2048kB 335*4096kB = 2006980kB)
<br>Jul 11 19:06:31 prddb02 kernel: Swap cache: add 0, delete 0, find 0/0, race
0+0<br>Jul 11 19:06:31 prddb02 kernel: 192549 pages of slabcache<br>Jul 11 19
:06:31 prddb02 kernel: 1192 pages of kernel stacks<br>Jul 11 19:06:31 prddb02
kernel: 0 lowmem pagetables, 127372 highmem pagetables
<br>Jul 11 19:06:31 prddb02 kernel: Free swap:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&nbsp; 8192980kB<br>Jul 11 19:06:31 prddb02 kernel: 1572864 pages of RAM<br>Jul
11 19:06:31 prddb02 kernel: 1277920 pages of HIGHMEM<br>Jul 11 19:06:31 prddb02
kernel: 89860 reserved pages
<br>Jul 11 19:06:31 prddb02 kernel: 24863915 pages shared<br>Jul 11 19:06:31
prddb02 kernel: 0 pages swap cached<br>Jul 11 19:06:31 prddb02 kernel: Out of
Memory: Killed process 8302 (oracle).<br><br><span style="font-weight: bold;">

$ sar -r -f /var/log/sa/sa11</span><br><font size="2"><span style="font-family:
courier new,monospace;">10:00:00 AM kbmemfree kbmemused&nbsp; %memused
kbbuffers&nbsp; kbcached kbswpfree kbswpused&nbsp; %swpused&nbsp; kbswpcad<
/span><br style="font-family: courier new,monospace;">

<span style="font-family: courier new,monospace;">05:00:00 PM&nbsp;&nbsp;
1901836&nbsp;&nbsp; 4030180&nbsp;&nbsp;&nbsp;&nbsp; 67.94&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp; 4228&nbsp;&nbsp; 1976412&nbsp;&nbsp; 8192980&nbsp;&nbsp;&nbsp;&nbsp
;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0.00&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0</span><br style="font-family: courier new
,monospace;"><span style="font-family: courier new,monospace;">

05:10:00 PM&nbsp;&nbsp; 1755672&nbsp;&nbsp; 4176344&nbsp;&nbsp;&nbsp;&nbsp; 70
.40&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1456&nbsp;&nbsp; 2115832&nbsp;&nbsp; 8192980
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
0.00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0</span><br style="font
-family: courier new,monospace;"><span style="font-family: courier new,monospace
;">05:20:00 PM&nbsp;&nbsp; 1901120&nbsp;&nbsp; 4030896&nbsp;&nbsp;&nbsp;&nbsp;
67.95&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2952&nbsp;&nbsp; 1976168&nbsp;&nbsp;
8192980&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp; 0.00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0</span><br
style="font-family: courier new,monospace;"><span style="font-family: courier
new,monospace;">05:30:00 PM&nbsp;&nbsp; 1896664&nbsp;&nbsp; 4035352&nbsp;&nbsp;
&nbsp;&nbsp; 68.03&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2392&nbsp;&nbsp; 1975920&nbsp;
&nbsp; 8192980&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;
0.00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0</span><br style="font
-family: courier new,monospace;"><span style="font-family: courier new,monospace
;">05:39:59 PM&nbsp;&nbsp; 1892232&nbsp;&nbsp; 4039784&nbsp;&nbsp;&nbsp;&nbsp;
68.10&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2200&nbsp;&nbsp; 1975900&nbsp;&nbsp;
8192980&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp; 0.00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0
</span>
<br style="font-family: courier new,monospace;"><span style="font-family:
courier new,monospace;">05:50:00 PM&nbsp;&nbsp; 1889808&nbsp;&nbsp; 4042208
&nbsp;&nbsp;&nbsp;&nbsp; 68.14&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2408&nbsp;&nbsp;
1976076&nbsp;&nbsp; 8192980&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0.00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&nbsp; 0</span><br style="font-family: courier new,monospace;">

<span style="font-family: courier new,monospace;">06:00:00 PM&nbsp;&nbsp;
1897984&nbsp;&nbsp; 4034032&nbsp;&nbsp;&nbsp;&nbsp; 68.00&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp; 2336&nbsp;&nbsp; 1976244&nbsp;&nbsp; 8192980&nbsp;&nbsp;&nbsp;&nbsp
;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0.00&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0</span><br style="font-family: courier new
,monospace;"><span style="font-family: courier new,monospace;">

06:10:00 PM&nbsp;&nbsp; 1831708&nbsp;&nbsp; 4100308&nbsp;&nbsp;&nbsp;&nbsp; 69
.12&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1192&nbsp;&nbsp; 2040788&nbsp;&nbsp; 8192980
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
0.00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0</span><br style="font
-family: courier new,monospace;"><span style="font-family: courier new,monospace
;">06:20:01 PM&nbsp;&nbsp; 1832808&nbsp;&nbsp; 4099208&nbsp;&nbsp;&nbsp;&nbsp;
69.10&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1372&nbsp;&nbsp; 2040912&nbsp;&nbsp;
8192980&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp; 0.00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0</span><br
style="font-family: courier new,monospace;"><span style="font-family: courier
new,monospace;">06:30:03 PM&nbsp;&nbsp; 1844548&nbsp;&nbsp; 4087468&nbsp;&nbsp;
&nbsp;&nbsp; 68.91&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1284&nbsp;&nbsp; 2040988&nbsp;
&nbsp; 8192980&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;
0.00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0</span><br style="font
-family: courier new,monospace;"><span style="font-family: courier new,monospace
;">06:40:01 PM&nbsp;&nbsp; 1895092&nbsp;&nbsp; 4036924&nbsp;&nbsp;&nbsp;&nbsp;
68.05&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1304&nbsp;&nbsp; 2041120&nbsp;&nbsp;
8192980&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp; 0.00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0
</span>
<br style="font-family: courier new,monospace;"><span style="font-family:
courier new,monospace;">06:50:00 PM&nbsp;&nbsp; 1909852&nbsp;&nbsp; 4022164
&nbsp;&nbsp;&nbsp;&nbsp; 67.80&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1300&nbsp;&nbsp;
2041248&nbsp;&nbsp; 8192980&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0.00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&nbsp; 0</span><br style="font-family: courier new,monospace;">

<span style="font-family: courier new,monospace;"><span style="font-weight:
bold;">07:00:01 PM&nbsp;&nbsp; 1906980&nbsp;&nbsp; 4025036</span>&nbsp;&nbsp;
&nbsp;&nbsp; 67.85&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1104&nbsp;&nbsp; 2041300&nbsp;
&nbsp; 8192980&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; 0.00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0</span>
<br style="font-family: courier new,monospace;">

<span style="font-family: courier new,monospace;"><span style="font-weight:
bold;">07:10:01 PM&nbsp;&nbsp; 4363412&nbsp;&nbsp; 1568604</span>&nbsp;&nbsp;
&nbsp;&nbsp; 26.44&nbsp;&nbsp;&nbsp;&nbsp; 13816&nbsp;&nbsp;&nbsp; 726432&nbsp;
&nbsp; 8192980&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; 0.00&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0</span>
</font><br><br>-Gene<br>

<a href="mailto:gene@(protected)" target="_blank" onclick="return top.js
.OpenExtLink(window,event,this)">gene@(protected)</a><br>

--
Taroon-list mailing list
Taroon-list@(protected)
https://www.redhat.com/mailman/listinfo/taroon-list