Browse Prior Art Database

Secure execution environment for untrusted programs using debugger technology Disclosure Number: IPCOM000009145D
Publication Date: 2002-Aug-09

Publishing Venue

The Prior Art Database

Related People

Wayne Gramlich


[ IPCOM000000002S originally published 2001-05-01 16:55 UTC ] This disclosure describes a way to execute untrusted programs safely, including programs from an unknown source that may contain a computer virus, worm, trojan horse, or other malicious code. Using a debugger, any program's interaction with the computer may be monitored and limited, eliminating the need to scan the program for malicious content such as a computer virus. A running program must interact with the system it is running on. A malicious program may use these interactions to modify the system in undesired ways. Java, Python, and Tcl, among others, use a technology called "sandbox", where an untrusted program is run as a subset of another program, with the executing program limiting the interactions this program can perform. However, the tight coupling of the trusted and untrusted code, and the complexity of the trusted code, allow numerous opportunities for errors that the untrusted code can exploit. Any system where trusted and untrusted code share a process space (memory resources) is likely to be vulnerable. This invention, which we may call "padded cell", uses debugger technology to run untrusted programs safely. Debuggers are designed to run and control programs (which frequently contain errors that lead to bad effects) without letting the programs affect the rest of the system. If the operating system supports it, a debugger will set up a memory space that the untrusted code can run in safely, and can be programmed to intercept all attempts by the program to communicate with the rest of the system. This disclosure will describe how to implement this technology on the Unix operating system. Programmers skilled in other operating systems and debuggers will easily be able to adapt this invention to their system, although some systems (e.g. Windows 95) cannot provide the necessary separation between memory spaces. On Unix, this could be implemented through the "/proc" interface which is used by debuggers. Depending on the debugger implementation, it may be desirable to modify the debugger program, write a separate program using much of the debugger source code, or configure the existing debugger. None of these options requires exceptional skill; the last option will be assumed in this file. Common forms of attack include: 1) Creating files on the system that will later be used maliciously. 2) Reading files containing data that should be secret. 3) Opening network connections (e.g. to report to the owner, or carry out an attack on another system). 4) Interacting with other programs in a way that exploits their weaknesses (e.g. corrupting their memory with malicious instructions). 5) Misleading the user (e.g. putting up a window to encourage them to supply their password). All of these attacks require a program to invoke "system calls". For example, the first three require an "open" call. The debugger can be configured to intercept or trap calls such as "open" and prevent them from executing, or make more complex decisions as to whether to allow the call to succeed. If the call should fail, the debugger can make it look to the program like the call failed naturally (e.g. the desired file didn't exist). If the program is non-malicious and well-written it can continue running without the desired resource. It may be more efficient not to trap certain system calls. Unix implementations vary as to whether they support this; Solaris does. For example, if the program is allowed to open a file at all, "read" calls to that file need not be trapped. A "wrapper" can invoke and configure the debugger and run the code with a single command, or be invoked automatically. This may be part of a system-wide security strategy. The padded cell may provide various levels of permission according to access type or target, such as which directory of the file system. It may prevent or restrict X Window attacks, for example by using an X protocol proxy such as xfwp or Xnest. It can restrict the use of terminal emulators to avoid CURSES attacks. It may provide the user with various levels of options when the program does something suspicious. It is more flexible than chroot and more widely applicable than access control lists (ACL) (e.g. used by Multics). It may also provide extra security for trusted programs that handle untrusted data, such as web browsers, mail readers, network port daemons (mitigating the risks of buffer overrun attacks), and for interpreters for any language including Java, JavaScript, Visual Basic, and VBScript. The disclosure is continued, and these points are developed more fully, in the attached text. [ 000000002S 02S 2S ]