Blog dedicated to Oracle Applications (E-Business Suite) Technology; covers Apps Architecture, Administration and third party bolt-ons to Apps

Showing posts with label Solaris. Show all posts
Showing posts with label Solaris. Show all posts

Friday, March 7, 2008

Top command CPU column, sum of all rows exceeds 100%

Recently during a performance call, someone asked me why the cpu column in a top command output doesn't total to 100%:

load averages: 16.9, 17.9, 19.3; up 63+02:24:38 18:19:12
2616 processes: 2587 sleeping, 10 zombie, 4 stopped, 15 on cpu
CPU states: 36.6% idle, 42.2% user, 21.2% kernel, 0.0% iowait, 0.0% swap
Memory: 96G phys mem, 29G free mem, 16G swap, 16G free swap

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
1686 oraerp01 1 0 4 2820M 1532M cpu 539:23 99.65% oracle
5784 oraerp01 1 38 4 2955M 1473M cpu 674:42 99.64% oracle
20496 oraerp01 1 0 4 4353M 1345M cpu 50.8H 98.98% oracle
28909 oraerp01 1 0 4 5056M 1420M cpu 88.3H 98.29% oracle
27438 oraerp01 1 0 4 1222M 1201M cpu 72:33 97.17% oracle
23596 oraerp02 1 0 0 3242M 2236M cpu 10:29 95.91% oracle
2768 oraerp02 1 0 0 3240M 2231M cpu 52:19 95.72% oracle
22037 oraerp02 1 60 0 3239M 2233M sleep 0:38 57.39% oracle
18232 oraerp02 1 0 0 3239M 2234M cpu 1:29 51.90% oracle
18261 oraerp03 1 0 0 3280M 2271M sleep 1:24 51.62% oracle
23563 oraerp02 1 1 0 3256M 2251M cpu 0:21 36.48% oracle
20855 oraerp02 1 21 0 3245M 2239M cpu 0:21 34.23% oracle
24205 oraerp02 1 10 0 3239M 2234M cpu 0:06 23.47% oracle
23536 oraerp02 1 52 0 3275M 2260M sleep 11:21 16.36% oracle
20283 oraerp02 1 53 0 3238M 2228M sleep 14:58 13.82% oracle

The box from which the above is run is a Sun E2900 with 24 cores and 96 GB RAM. I told that person that, top is actually showing the percentage of a single CPU being used by that process. I have tried to verify from various sources whether my answer was correct and haven't come up with anything conclusive. Will keep looking for the real answer, which may involve going into the source code of top command.

Monday, December 3, 2007

noexec_user_stack, Solaris and Oracle 10g

Metalink Note 395334.1 explains Why do we need to set NOEXEC_USER_STACK to 1 before Oracle installation as a mandatory step:

Sometimes programs allocate buffers but do not always check the size of the data coming into these buffers. An unwanted program can overflow such a buffer by placing a pointer to its own code and take control of the application with the privileges assigned to that application. Solaris 9 introduced the option to disable execution of user programs on the stack and assist in preventing this type of issue. This feature is only available on SPARC and AMD64 systems because it requires hardware support that is not available in current Intel 32bit CPUs. The protection is setup by employing the system wide parameter 'set noexec_user_stack=1' in the /etc/system file.

I got additional information from Solaris manuals:

A number of security bugs are related to default executable stacks when permissions are set to read, write, and execute. Although the SPARC and Intel application binary interface (ABI) mandates that stacks have execute permissions, most programs can function correctly without using executable stacks.

The Solaris 2.6 release provides a noexec_user_stack variable, which enables you to specify whether stack mappings are executable or not. By default, the value for the variable is zero, which provides ABI-compliant behavior. If the variable is set to non-zero, the system marks the stack of every process in the system as readable and writable but not executable.

Disabling Programs from Using Executable StacksUse the following steps to disable programs from using executable stacks:
1. Become superuser.
2. Add the line set noexec_user_stack=1 to the /etc/system file.
3. Type init 6 to reboot the system.

When the variable is set to a non-zero value, programs that execute code on their stack are sent a SIGSEGV signal, which usually terminates the program with a core dump. Such programs also generate a warning message, which includes the name of the program, the PID, and the UID of the user who ran the program, as shown in the following example:
a.out[347] attempt to execute code on stack by uid 555

The message is logged by the syslogd(1M) daemon when the syslog kern facility is set to notice level. This logging is set by default in the syslog.conf file, which means that the message is sent to both the console and to the /var/adm/messages file.

When you have set the noexec_user_stack variable, you can monitor these messages to observe potential security problems. You can also monitor the messages to identify valid programs that depend on executable stacks and have been prevented from correct operation. You can explicitly mark program stacks as executable by using the mprotect function.

Because of hardware limitations, executable stack problems can be caught and reported only on sun4m, sun4d, and sun4u platforms.

Developers sometimes allocate buffers in applications but do not always check the size of the data coming into these buffers. An attacker can overflow such a buffer and by placing a pointer to a piece of code under the attacker's control can take control of the application with the privileges assigned to that application. Buffer overflow exploits are most common on the stack and not very common on the heap (since heap memory allocation is different and hence harder to exploit). Solaris 9 introduced the option to disable execution of user programs on stack and assist in lowering the likely hood of exploit from stack based buffer overflows, this feature is only available on SPARC and AMD64 systems because it requires hardware support that is not available in current Intel 32bit CPUs. This can be done by employing a system wide parameter for example. Introducing the line 'set noexec_user_stack=1' in the /etc/system file. Another option is to have the application's stack be defined non-executable using the new mapfile segment descriptor.

The ability to prevent stack buffer overflow exploits along with Process Rights Management introduced in Solaris 10 provides a powerful mechanism to strengthen the security environment for applications. It relieves the IT administrator from the burden of worrying whether an application on one of the systems can compromise the rest of the system and the network due to buffer overflow exploits. It provides the IT administrator with a global switch to disallow even poorly written applications to be susceptible to stack buffer overflow attacks.

The Process Rights Management feature combined with the noexec_user_stack switch introduced in Solaris 7, provides some of the most powerful mechanisms of preventing and limiting the damage caused by hackers when they exploit application vulnerabilities. It is important to note that N1 Grid Containers have lower maximum privilege sets and for example, don't have access to devices.