Browse Prior Art Database

Enabling Crashed Stack Analysis Through a Virtualized Single Instruction Multiple Data Stack

IP.com Disclosure Number: IPCOM000021499D
Publication Date: 2004-Jan-21
Document File: 4 page(s) / 572K

Publishing Venue

The IP.com Prior Art Database

Abstract

Disclosed is a method that leverages a Single Instruction Multiple Data (SIMD) register to operate as a minimal rotating call stack, providing data about the call sequence prior to a smashed stack. Benefits include a solution that works with any calling convention.

This text was extracted from a Microsoft Word document.
At least one non-text object (such as an image or picture) has been suppressed.
This is the abbreviated version, containing approximately 58% of the total text.

Enabling Crashed Stack Analysis Through a Virtualized Single Instruction Multiple Data Stack

Disclosed is a method that leverages a Single Instruction Multiple Data (SIMD) register to operate as a minimal rotating call stack, providing data about the call sequence prior to a smashed stack. Benefits include a solution that works with any calling convention.

Background

Currently, developers analyzing a program crash or bug with a “smashed stack” are unable to determine which procedures called the code that actually crashed. This occurs because the x86 programming architecture uses the same call stack for local variables, stack frame pointers, and return addresses.

In the case of a buffer overflow of a local variable, it is possible to overwrite the stack frame pointer, the return address, or both. This prevents a debugger from being able to show the call trace or “back trace” (i.e. the set of procedures leading up to the crash). Often, the data that caused the crash (and likely also smashed the stack) happened in a procedure that is different from where the exception or error is actually detected.

General Description

The disclosed method leverages an SIMD register to operate as a minimal rotating call stack, providing data about the call sequence leading up to the smashed stack. The disclosed method will provide invaluable data to developers in frequent smashed stack situations.

One implementation of the disclosed method uses 128 bit SSE2 to reserve one of the SIMD registers, and modify the predicate of each function to shift the register four bytes to the left and place the return address in the beginning of the SIMD register. Prior to returning from the function, the register is shifted four bytes to the right. A sample function “foo” is show below with the changes compiled in gcc ATT Syntax Format:

void foo()

{unsigned int x[2] = {0x11111111,0x22222222};

         bar();}

Originally Compiled to:

With SIMD Stack Tracking Support (Changes in Red):

0x08048318 <foo+0>: push %ebp

0x08048319 <foo+1>: mov %esp,%ebp

0x0804831b <foo+3>: sub $0x8,%esp

0x0804831e <foo+6>: movl $0x11111111,0xfffffff8(%ebp)

0x08048325 <foo+13>: movl $0x22222222,0xfffffffc(%ebp)

0x0804832c <foo+20>: call 0x80482f4 <bar>

0x08048331 <foo+25>: leave

0x08048332 <foo+26>: ret

0x0804832b <foo+0>: movd (%esp,1),%xmm6

0x08048330 <foo+5>: pslldq $0x4,%xmm7

0x08048335 <foo+10>: por %xmm6,%xmm7

0x08048339 <foo+14>: push %ebp

0x0804833a <foo+15>: mov %esp,%ebp

0x0804833c <foo+17>: sub $0x8,%esp

0x0804833f <foo+20>: movl $0x11111111,0xff...