Taroon + numa question ? 2006-11-28 - By Paul Krizak
Back
Unfortunately RHEL3's NUMA implementation is extraordinarly broken.
We've been working with them for many months now trying to fix it, and
have finally given up and are waiting for RHEL5 to implement a correct
NUMA architecture for our CPUs.
As far as we can tell, here's what happens for various OS/NUMA combinations:
RHEL3, U<7 + NUMA:
* Processes that grow larger than one "NUMA node", i.e. the memory
attached to one node, will dip into swap instead of using memory from
other nodes.
* The kernel has no clue where the memory is mapped in relation to the
CPU cores, and so makes dumb decisions on where to put processes. Using
cpuset helps, but is impractical for a batch compute node.
RHEL3, U>=7 + NUMA:
* Processes that grow larger than one "NUMA node" will use memory from
other nodes, but at a SEVERE performance penalty (though not as great as
using swap)
* The kernel still has no clue where the memory is mapped to CPU cores.
RHEL4, U>=2 + NUMA:
* Memory allocation works fine. No significant performance penalty as a
process grows beyond one node.
* The kernel is still clueless about where to put processes.
Given the poor luck we've had with NUMA, our compute nodes run with the
following configuration options, and perform (on average) better than
with NUMA enabled:
* ACPI SRAT table disabled
* Node Interleaving enabled
* Bank Interleaving enabled
"numa=off" on kernel command line
Various hardware and software vendors balk at the disabling of NUMA, but
our internal benchmarks don't lie -- NUMA is way slower (in general) in
RHEL3 and RHEL4 than non-NUMA in RHEL3 and RHEL4 on the Opteron platform.
Paul Krizak 5900 E. Ben White Blvd. MS 625
Advanced Micro Devices Austin, TX 78741
Linux/Unix Systems Engineering Phone: (512) 602-8775
Silicon Design Division Cell: (512) 791-0686
Stanley, Jon wrote:
>> Hmm, somehow I think NUMA doesn't really come in to play with a single
>> socket whatevercore cpu's.
>>
>
> This is correct - a single physical socket is one NUMA node. However,
> in the event of a process needing 6GB of RAM, it will get the memory
> from another node prior to going to swap for it. There is a performance
> penalty with inter-node memory allocation, but it's not really something
> that can be avoided, I don't think.
>
> Again, I could be totally wrong.
>
> --
> Taroon-list mailing list
> Taroon-list@(protected)
> https://www.redhat.com/mailman/listinfo/taroon-list
>
>
--
Taroon-list mailing list
Taroon-list@(protected)
https://www.redhat.com/mailman/listinfo/taroon-list
|
|
