Exploring SEH prologues a little

One of the very common reasons that IDA generates the sp-analysis failed messages when disassembling code is due to the existence of SEH and the SEH prologues and epilogues that compilers use to set up the SEH frame. Because the prologue directly modifies the stack pointer, IDA can get a bit confused when doing it’s stack pointer tracing. Sometimes it gets it right, and sometimes it doesn’t.

The following describes what happens in a generic SEH prologue, using the C++ EH_prolog3 as an example. There are various such prologues and epilogues, as the following table shows:

Name                                     Type      EH Cookie   GS Cookie  Catch Handlers</strong>
SEH_prolog/SEH_epilog                    SEH3      -           -
SEH_prolog4/SEH_epilog4                  SEH4      +           -
SEH_prolog4_GS/SEH_epilog4_GS            SEH4      +           +
EH_prolog                                C++ EH    -           -          +/-
EH_prolog3/EH_epilog3                    C++ EH    +           -          -
EH_prolog3_catch/EH_epilog3              C++ EH    +           -          +
EH_prolog3_GS/EH_epilog3_GS              C++ EH    +           +          -
EH_prolog3_catch_GS/EH_epilog3_catch_GS  C++ EH    +           +          +

Source: http://www.openrce.org/articles/full_view/21

Let’s examine some code generated by IDA:

Caller:

.text:01371D53     var_14         = dword ptr -14h  ; Real local variable
.text:01371D53     var_4          = dword ptr -4    ; SEH scope table ptr that 
                                                    ; IDA thinks is local vars
                                                    ; (see later)

.text:01378673 000  push    8                       ; No. of bytes of 
                                                    ; local vars

.text:01378675 004  mov     eax, offset Handler     ; Store ptr to handler
.text:0137867A 004  call    __EH_prolog             ; CALL Prologue

.text:013B2501
.text:013B2501     arg_0           = byte ptr  4
.text:013B2501

.text:013B2501 000  push    eax                     ; Push "SEH Handler"
.text:013B2502 004  push    large dword ptr fs:0    ; Push ptr to "SEH Head"
.text:013B2509 008  lea     eax, [esp+8+arg_0]      ; EAX = Address of "push 8"
.text:013B250D 008  sub     esp, [esp+12]           ; Grow stack by "push 8" 
                                                    ; (i.e. 8)
.text:013B2511 00C  push    ebx                     ; Non-volatile
.text:013B2512 010  push    esi                     ; Non-volatile
.text:013B2513 014  push    edi                     ; Non-volatile
.text:013B2514 018  mov     [eax], ebp              ; Change "push 8" to 
                                                    ; "parent EBP"
.text:013B2516 018  mov     ebp, eax                ; Set EBP = "push 8"
.text:013B2518 018  mov     eax, ___security_cookie
.text:013B251D 018  xor     eax, ebp
.text:013B251F 018  push    eax                     ; Push SEH canary
.text:013B2520 01C  push    dword ptr [ebp-4]       ; Push return address (temp)
.text:013B2523 020  mov     dword ptr [ebp-4], 0FFFFFFFFh  ; Ptr to SEH scope table
.text:013B252A 020  lea     eax, [ebp-12]
.text:013B252D 020  mov     large fs:0, eax         ; Make FS:[0] point to THIS
                                                    ; handler
.text:013B2533 020  retn

Before prologue:

Stack contains the size of the local variables needed by the caller.

Stack: {8}

After CALLing prologue:

As part of CALL, return address of caller is pushed onto stack.

Stack: {8, RetAddr}

push eax:

Prior to this prologue, eax holds the pointer to the SEH handler (or wrapper). This is pushed onto the stack.

Stack: {8, RetAddr, SEH_Handler}

push large dword ptr fs:0

We set the “next” pointer of the SEH record to point to the first item on the SEH chain (prior to us being inserted). This is generic linked-list behaviour.

Stack: {8, RetAddr, SEH_Handler, SEH_Next}

lea eax, [esp+8+arg_0]:

This is simply the address of the “8” on the stack. We are going to make this EBP later.

Stack: {8, RetAddr, SEH_Handler, SEH_Next}

sub esp, [esp+12]

This is exactly the same address – of the “8” on the stack. We are allocating space for the caller’s locals. At this point, we can see that a standard SEH record is 8 bytes, and because of the original “8”, it is 12 bytes. In other words, the first local is at the 16th byte, or 0x10. Once the “8” becomes the location of EBP (later), our locals then start at [EBP+10h], or var_10.

Stack: {8, RetAddr, SEH_Handler, SEH_Next, DWORD_Local, DWORD_Local}

push ebx, esi, edi:

This is just the standard storage of non-volatile registers.

mov [eax], ebp:

We want to create the real EBP now (as mentioned earlier). [EBP] needs to store the caller’s parent’s EBP, which we do here.

Stack: {OldEBP, RetAddr, SEH_Handler, SEH_Next, DWORD_Local, DWORD_Local, ebx, esi, edi}

mov ebp, eax:

Make EBP point to the “8”. This is becoming just like a standard EBP-based stack frame.

mov eax, ___security_cookie xor eax, ebp push eax:

We create the canary and put it on the stack.

Stack: {OldEBP, RetAddr, SEH_Handler, SEH_Next, DWORD_Local, DWORD_Local, ebx, esi, edi, CANARY}

push dword ptr [ebp-4]:

[ebp-4] points to the RetAddr, which we haven’t touch since. However, we want to use this to return correctly out of the prologue, but the problem is that retn = pop eip, taking the top-most value on the stack to return to. Our stack has grown quite a bit, and esp is not where retn expects it to be. We create a temporary return address at the top of the stack, just for this.

Stack: {OldEBP, RetAddr, SEH_Handler, SEH_Next, DWORD_Local, DWORD_Local, ebx, esi, edi, CANARY, RetAddr}

mov dword ptr [ebp-4], 0FFFFFFFFh:

This is the pointer to the SEH scope table, and it’s supposed to be below the SEH handler. The scope table stores the specific code for __finally and __except. Hence, we make it so. The caller may overwrite this, so we just fill this with 0xFFFFFFFF first.

Stack: {OldEBP, 0xFFFFFFFF, SEH_Handler, SEH_Next, DWORD_Local, DWORD_Local, ebx, esi, edi, CANARY, RetAddr}

lea eax, [ebp-12]

mov large fs:0, eax:

Finally, we need to update the FS:[0] SEH head pointer. We make it point to our handler since we inserted ourselves at the top of the SEH linked list.

retn:

We return. The top of the stack nicely contains the return address

Stack: {OldEBP, RetAddr, SEH_Handler, SEH_Next, DWORD_Local, DWORD_Local, ebx, esi, edi, CANARY}

Also,

EBP > OldEBP ESP > CANARY

Final comments

To complete the picture, IDA often disassembles the caller and generates var_4 as a local variable (as can be seen above).

Note that since var_4 is [ebp+4], which is the SEH scope table pointer (0xFFFFFFFF), it is not a local variable. It is used when the caller wishes to modify the scope table pointer, and IDA cannot recognize that generically and hence errs on the side of it being a local variable. var_10 and above, up to the size of the local variables (8 in the above example), are the real local variables.