Use of syscall and sysenter in VMProtect 3.1


Few days ago Xjun briefly mentioned a new feature of VMProtect 3.1 - it uses direct syscalls to check if software is running under debugger. I decided to take a quick look myself and figure out how exactly it works.

I'll give you 2 targets for the research:

  • oppo_flash_tool.exe (!ZgJzjQxR!cNEHMwM-jKnLVgPXf4OUupyk1DNt69FYB2rEfY-5AlA) - it was given as an example by Xjun;
  • asshurt.dll ( - some sort of a cheat for Roblox. I don't care about the cheat itself, it just happened to have the syscall feature enabled. And it uses different syscalls than Oppo.

In addition to that, I'll provide a very simple demo executable which replicates part of the VMProtect protection, so that you don't waste time looking at obfuscated code.

As a debugger in 32-bit OS you can use anything you like. On 64-bit OS you will really need to use WinDbg - as far as I know, it's the only debugger that can handle those tricks..

32bit OS

Let's start by debugging oppo_flash_tool.exe. First, we need to get past the usual tricks like IsDebuggerPresent and CheckRemoteDebuggerPresent. If you're reading this, I'm sure you know how to do that.

Few moments later we'll arrive here:

00dcfd8e f744250000800000 test    dword ptr [ebp],8000h
00dcfd96 e9f6bdfdff      jmp     00dabb91
00dabb91 0f84841a0b00    je      00e5d61b

Remember this conditional jump. It's taken on 32bit OSes and not taken on 64-bit OS.

Let's look at 32bit OS version first. Now VMProtect prepares to call sysenter.

00e5d61b 56              push    esi
00e5d61c 85f8            test    eax,edi
00e5d61e 57              push    edi
00e5d61f 0facd074        shrd    eax,edx,74h
00e5d623 6633d3          xor     dx,bx
00e5d626 53              push    ebx
00e5d627 8bd9            mov     ebx,ecx
00e5d629 81fcfe567054    cmp     esp,547056FEh
00e5d62f 8bd3            mov     edx,ebx
00e5d631 35fb5cc775      xor     eax,75c75cfbh
00e5d636 c1e202          shl     edx,2
00e5d639 0fbae028        bt      eax,28h
00e5d63d 8bc5            mov     eax,ebp
00e5d63f 6681fbda38      cmp     bx,38DAh
00e5d644 8d0410          lea     eax,[eax+edx]
00e5d647 f7c615443214    test    esi,14324415h

Since the code is obfuscated, here comes a cleaned-up version. Please note that in other applications, different registers can be used.

    ;   ECX = number of parameters for the syscall
    ;   [EBP] = syscall id. See
    ;   [EBP+4] .. [EBP+X] = params for the syscall
    ;   [EBP-4] .. [EBP-8] = free space to save registers
    push    esi
    push    edi
    push    ebx

    ; save register values for later
    lea     eax, [ebp+ecx*4]
    mov     dword ptr [ebp-4], eax
    mov     dword ptr [ebp-8], esp

    ; set up stack frame for syscall
    mov     eax, dword ptr [ebp+ecx*4]
    push    eax
    sub     ecx, 1
    jnz     setupParams

    ; put syscall number in EAX
    mov     eax, dword ptr [ebp]

    ; the actual call
    call    trampoline1

    ; restore stack and frame pointers
    mov     esp, dword ptr [ebp-8]
    mov     ebp, dword ptr [ebp-4]

    ; save result
    mov     dword ptr [ebp], eax

    ; restore registers
    pop     ebx
    pop     edi
    pop     esi
    jmp     00de21de

    call    trampoline2

    mov     edx, esp

// -- continue VM execution as usual --
00de21de 8b06            mov     eax,dword ptr [esi]
00de21e2 8db604000000    lea     esi,[esi+4]
00de21ea 33c3            xor     eax,ebx
00de21ec 8d807cc2efb1    lea     eax,[eax-4E103D84h]
00de21f6 f7d0            not     eax
00de21fd 35ee613a76      xor     eax,763a61ee
00df9a28 48              dec     eax
00df9a29 f8              clc
00df9a2a c1c002          rol     eax,2
00df9a2e 33d8            xor     ebx,eax
00df9a32 03f8            add     edi,eax
00df9a34 e9a0eafbff      jmp     00db84d9
00db84d9 ffe7            jmp     edi
00ddaeba 8b542500        mov     edx,dword ptr [ebp]

64-bit OS

Here it is getting interesting! 🙂 You cannot use sysenter instruction from 32-bit code in 64-bit Windows. But, as ReWolf described few years ago, one can mix x86 code with x64 code in the same process. And that's exactly what VMProtect 3.1 is doing.

Let's go back to that conditional jump and see what happens in 64-bit OS. The jump will not be taken:

00dabb91 0f84841a0b00    je      00e5d61b
00dabb97 9adf4ce8003300  call    0033:00E84CDF
00dabb9e 668cd1          mov     cx,ss
00dabba1 668ed1          mov     ss,cx

Far call?! Last time I saw that was in 16-bit Windows era..

As explained in ReWolf's article:

Summing things up, for every process (x86 & x64) running on 64-bits Windows there are allocated two code segments:

  • cs = 0x23 -> x86 mode
  • cs = 0x33 -> x64 mode

So, as soon as you execute that call, you'll switch to a 64-bit world. WinDbg happily recognizes that, all other debuggers just go astray..

00dabb97 9adf4ce8003300  call    0033:00E84CDF

00000000`00e84cdf 56              push    rsi
00000000`00e84ce0 57              push    rdi
00000000`00e84ce1 53              push    rbx
00000000`00e84ce2 8bd9            mov     ebx,ecx
00000000`00e84ce4 8bd3            mov     edx,ebx
00000000`00e84ce6 33c9            xor     ecx,ecx
00000000`00e84ce8 81fb04000000    cmp     ebx,4
00000000`00e84cee 0f8606000000    jbe     00000000`00e84cfa
00000000`00e84cf4 8d8bfcffffff    lea     ecx,[rbx-4]
00000000`00e84cfa c1e103          shl     ecx,3

x64 code does pretty much the same thing as x86 code - sets up a stack frame, sets up registers and then executes syscall instruction. Cleaned-up and shortened version follows:

    ;   ECX = number of parameters for the syscall
    ;   [EBP] = encoded syscall id. 
    ;           High order byte = special handling info
    ;           Lowest 15 bits = syscall id
    ;   [EBP+4] .. [EBP+X] = params for the syscall
    ;   [EBP-8] .. [EBP-10] = free space to save registers
    ;   [EBP-..] = free space to use in specific syscalls

    push    rsi
    push    rdi
    push    rbx
    mov     ebx,ecx
    mov     edx,ebx
    xor     ecx,ecx

    ; calculate new stack frame pointer
    cmp     ebx,4
    jbe     @F
    lea     ecx,[rbx-4]
    shl     ecx,3
    shl     edx,2
    mov     rax,rbp
    add     rax,rdx

    ; save registers
    mov     qword ptr [rbp-8],rax
    mov     qword ptr [rbp-10h],rsp

    ; adjust RSP
    sub     rsp,rcx
    and     rsp,0FFFFFFFFFFFFFFF0h
    add     rsp,rcx

    ; useless?
    mov     r10d,dword ptr [rbp]
    shr     r10d,9

    ; set up params for syscall
    test    ebx,ebx
    je      doneSettingParams

    mov     eax,dword ptr [rbp+rbx*4]
    cmp     ebx,1
    jne     @F
    mov     rcx,rax
    jmp     nextParam
    cmp     ebx,2
    jne     @F
    mov     rdx,rax
    jmp     nextParam
    cmp     ebx,3
    jne     @F
    mov     r8,rax
    jmp     nextParam
    cmp     ebx,4
    jne     @F
    mov     r9,rax
    jmp     nextParam
    push    rax

    sub     ebx,1
    jne     loopSetParams

    ; check if syscall needs special handling
    mov     rax,qword ptr [rbp]
    mov     r10d,eax
    shr     r10d,18h

    ; 3 = NtQueryInformationProcess
    cmp     r10b,3
    jne     doSyscall

    ; fix current process pseudo-handle, if it's there
    cmp     ecx,0FFFFFFFFh
    jne     @F
    movsx   rcx,cl
    ; is this ProcessDebugObjectHandle request?
    cmp     edx,1Eh
    jne     @F

    ; if so, fix buffer and size for ProcessDebugObjectHandle request. 
    ; It should be 8-bytes long.
    lea     r10,[rbp-18h]
    mov     r8,r10
    mov     r9d,8
    jmp     doSyscall

    and     eax,7FFFh
    sub     rsp,20h
    call    trampoline
    jmp     processResult

    mov     r10,rcx

    ; check for special handling again
    mov     r10d,dword ptr [rbp]
    shr     r10d,18h

    ; 3 = NtQueryInformationProcess
    cmp     r10b,3
    jne     returnToX86

    ; is this ProcessDebugObjectHandle ?
    cmp     dword ptr [rbp+8],1Eh
    jne     returnToX86

    ; were 2 buffers the same in original call?
    mov     ecx,dword ptr [rbp+0Ch]
    cmp     ecx,dword ptr [rbp+14h]
    je      @F

    ; if not, copy returned DebugObjectHandle back to original buffer
    mov     r10d,dword ptr [rbp-18h]
    mov     dword ptr [rcx],r10d
    jmp     returnToX86

    mov     rsp,qword ptr [rbp-10h]
    mov     rbp,qword ptr [rbp-8]
    mov     dword ptr [rbp],eax
    pop     rbx
    pop     rdi
    pop     rsi

You'll notice that x64 version is slightly more complex due to the way parameters are passed (registers vs. stack). It also includes a special treatment for 8 special edge cases - it will modify syscall parameters to adjust buffers and pointer sizes to satisfy requirements for 64-bit code.

NOTE - to keep code simple, I only showed the part which deals with NtQueryInformationProcess but other cases are similar.

As you can see, return back from x64 to the x86 world is a simple retf instruction. x86 code continues right where it left off:

00000000`00e7edf1 cb              retf
00dabb9e 668cd1          mov     cx,ss
00dabba1 668ed1          mov     ss,cx
00dabba4 8b16            mov     edx,dword ptr [esi]
00dabba6 3bf8            cmp     edi,eax
00dabba8 80fbf2          cmp     bl,0F2h
00dabbab 81c604000000    add     esi,4
00dabbb1 6685c8          test    ax,cx
00dabbb4 33d3            xor     edx,ebx
00dabbb6 f5              cmc
00dabbb7 8d927cc2efb1    lea     edx,[edx-4E103D84h]
00dabbbd 6681ff8e5b      cmp     di,5B8Eh
00dabbc2 f7d2            not     edx
00dabbc4 80fbf5          cmp     bl,0F5h
00dabbc7 81f2ee613a76    xor     edx,763a61ee
00dabbcd e9479f0500      jmp     00e05b19
00e05b19 4a              dec     edx
00e05b1a f8              clc
00e05b1b c1c202          rol     edx,2
00e05b1e 33da            xor     ebx,edx
00e05b20 6681fff273      cmp     di,73F2h
00e05b25 03fa            add     edi,edx
00e05b27 ffe7            jmp     edi
00ddaeba 8b542500        mov     edx,dword ptr [ebp]

Instruction at address 0x00ddaeba is the same for both x86 and x64 OS-es and VM continues as usual.

Different protection modes and syscalls

I provided you with 2 real-world test executables. Oppo seems to be simpler and use just 3 syscalls:

  • NtQueryInformationProcess with ProcessDebugObjectHandle class
  • NtSetInformationThread with ThreadHideFromDebugger class
  • NtProtectVirtualMemory to set protection attributes for each section in original executable

Asshurt doesn't have antidebug trick with NtQueryInformationProcess but it uses additional syscalls for some purposes:

  • NtOpenFile
  • NtCreateSection
  • NtMapViewOfSection
  • NtQueryVirtualMemory
  • NtUnmapViewOfSection
  • NtClose

Suggested workaround

Since VMProtect is using undocumented Windows features, it somehow needs to ensure that the protection will work on each and every Windows version. That's VMProtect's biggest strength and also the biggest weakness.

Windows' syscall numbers change in each version and also between major builds. Use the wrong syscall number and you're guaranteed to receive unexpected results. So, VMProtect developers had to hardcode a table with Windows build numbers and corresponding syscall id's in the executable.

You can see the syscall numbers in the j00ru's page (slightly out of date) or in tinysec's windows kernel syscall table

To obtain Windows build number, VMProtect uses information from PEB (Process Environment Block). The method is already described in The MASM Forum, so I'll just reproduce the (ugly) code from their page:

	print	"Read From Process Environment Block:",13,10
    ASSUME	FS:Nothing
    mov		edx,fs:[30h]	;PEB.InheritedAddressSpace
    mov     eax,[edx+0A4h]	;eax = Major Version
    push	eax
    push    edx
    print   ustr$(eax),'.'
    pop     edx
    push	edx
    mov     eax,[edx+0A8h]	;eax = Minor Version
    print   ustr$(eax),'.'
    pop edx
    mov eax,[edx+0ACh]		;eax = build
    and eax,0FFFFh			;because win 7 collapses
    print ustr$(eax),13,10,13,10	
	pop		eax

VMProtect checks only the build number and picks the corresponding syscall number. However, if the build number is not in the internal database, it will not use direct syscall and fall back to standard protection. Bingo, problem solved - no need for ugly hacks like Xjun's SharpOD plugin!

Hint: VMProtect 3.1 doesn't support Windows 10 Creators Update (build number 15063).

Demo time

As promised, here is a download link for the test application:
Note: it should support most common builds of Windows XP/7/8.1/10. Windows 2003/Vista and other rare systems are not supported!

If it shows "OK" message, you've hidden your debugger well. If it shows "Debugger detected", you have a problem. 🙂

Have fun!

EDIT: Updated download link for Oppo. Mediafire's antivirus tends to have plenty of False Positives..