noexec_user_stack, Solaris and Oracle 10g

Metalink Note 395334.1 explains Why do we need to set NOEXEC_USER_STACK to 1 before Oracle installation as a mandatory step:

Sometimes programs allocate buffers but do not always check the size of the data coming into these buffers. An unwanted program can overflow such a buffer by placing a pointer to its own code and take control of the application with the privileges assigned to that application. Solaris 9 introduced the option to disable execution of user programs on the stack and assist in preventing this type of issue. This feature is only available on SPARC and AMD64 systems because it requires hardware support that is not available in current Intel 32bit CPUs. The protection is setup by employing the system wide parameter 'set noexec_user_stack=1' in the /etc/system file.

I got additional information from Solaris manuals:

A number of security bugs are related to default executable stacks when permissions are set to read, write, and execute. Although the SPARC and Intel application binary interface (ABI) mandates that stacks have execute permissions, most programs can function correctly without using executable stacks.

The Solaris 2.6 release provides a noexec_user_stack variable, which enables you to specify whether stack mappings are executable or not. By default, the value for the variable is zero, which provides ABI-compliant behavior. If the variable is set to non-zero, the system marks the stack of every process in the system as readable and writable but not executable.

Disabling Programs from Using Executable StacksUse the following steps to disable programs from using executable stacks:
1. Become superuser.
2. Add the line set noexec_user_stack=1 to the /etc/system file.
3. Type init 6 to reboot the system.

When the variable is set to a non-zero value, programs that execute code on their stack are sent a SIGSEGV signal, which usually terminates the program with a core dump. Such programs also generate a warning message, which includes the name of the program, the PID, and the UID of the user who ran the program, as shown in the following example:
a.out[347] attempt to execute code on stack by uid 555

The message is logged by the syslogd(1M) daemon when the syslog kern facility is set to notice level. This logging is set by default in the syslog.conf file, which means that the message is sent to both the console and to the /var/adm/messages file.

When you have set the noexec_user_stack variable, you can monitor these messages to observe potential security problems. You can also monitor the messages to identify valid programs that depend on executable stacks and have been prevented from correct operation. You can explicitly mark program stacks as executable by using the mprotect function.

Because of hardware limitations, executable stack problems can be caught and reported only on sun4m, sun4d, and sun4u platforms.

Developers sometimes allocate buffers in applications but do not always check the size of the data coming into these buffers. An attacker can overflow such a buffer and by placing a pointer to a piece of code under the attacker's control can take control of the application with the privileges assigned to that application. Buffer overflow exploits are most common on the stack and not very common on the heap (since heap memory allocation is different and hence harder to exploit). Solaris 9 introduced the option to disable execution of user programs on stack and assist in lowering the likely hood of exploit from stack based buffer overflows, this feature is only available on SPARC and AMD64 systems because it requires hardware support that is not available in current Intel 32bit CPUs. This can be done by employing a system wide parameter for example. Introducing the line 'set noexec_user_stack=1' in the /etc/system file. Another option is to have the application's stack be defined non-executable using the new mapfile segment descriptor.

The ability to prevent stack buffer overflow exploits along with Process Rights Management introduced in Solaris 10 provides a powerful mechanism to strengthen the security environment for applications. It relieves the IT administrator from the burden of worrying whether an application on one of the systems can compromise the rest of the system and the network due to buffer overflow exploits. It provides the IT administrator with a global switch to disallow even poorly written applications to be susceptible to stack buffer overflow attacks.

The Process Rights Management feature combined with the noexec_user_stack switch introduced in Solaris 7, provides some of the most powerful mechanisms of preventing and limiting the damage caused by hackers when they exploit application vulnerabilities. It is important to note that N1 Grid Containers have lower maximum privilege sets and for example, don't have access to devices.