19 Aug 2017

Use of syscall and sysenter in VMProtect 3.1

Few days ago Xjun briefly mentioned a new feature of VMProtect 3.1 - it uses direct syscalls to check if software is running under debugger. I decided to take a quick look myself and figure out how exactly it works.

I'll give you 2 targets for the research:

  • oppo_flash_tool.exe (https://mega.nz/#!ZgJzjQxR!cNEHMwM-jKnLVgPXf4OUupyk1DNt69FYB2rEfY-5AlA) - it was given as an example by Xjun;
  • asshurt.dll (https://mediafire.com/?3xyc0ugc2hxervn) - some sort of a cheat for Roblox. I don't care about the cheat itself, it just happened to have the syscall feature enabled. And it uses different syscalls than Oppo.

In addition to that, I'll provide a very simple demo executable which replicates part of the VMProtect protection, so that you don't waste time looking at obfuscated code.

As a debugger in 32-bit OS you can use anything you like. On 64-bit OS you will really need to use WinDbg - as far as I know, it's the only debugger that can handle those tricks..

32bit OS

Let's start by debugging oppo_flash_tool.exe. First, we need to get past the usual tricks like IsDebuggerPresent and CheckRemoteDebuggerPresent. If you're reading this, I'm sure you know how to do that.

Few moments later we'll arrive here:

Remember this conditional jump. It's taken on 32bit OSes and not taken on 64-bit OS.

Let's look at 32bit OS version first. Now VMProtect prepares to call sysenter.

Since the code is obfuscated, here comes a cleaned-up version. Please note that in other applications, different registers can be used.

64-bit OS

Here it is getting interesting! smile You cannot use sysenter instruction from 32-bit code in 64-bit Windows. But, as ReWolf described few years ago, one can mix x86 code with x64 code in the same process. And that's exactly what VMProtect 3.1 is doing.

Let's go back to that conditional jump and see what happens in 64-bit OS. The jump will not be taken:

Far call?! Last time I saw that was in 16-bit Windows era..

As explained in ReWolf's article:

Summing things up, for every process (x86 & x64) running on 64-bits Windows there are allocated two code segments:

  • cs = 0x23 -> x86 mode
  • cs = 0x33 -> x64 mode

So, as soon as you execute that call, you'll switch to a 64-bit world. WinDbg happily recognizes that, all other debuggers just go astray..

x64 code does pretty much the same thing as x86 code - sets up a stack frame, sets up registers and then executes syscall instruction. Cleaned-up and shortened version follows:

You'll notice that x64 version is slightly more complex due to the way parameters are passed (registers vs. stack). It also includes a special treatment for 8 special edge cases - it will modify syscall parameters to adjust buffers and pointer sizes to satisfy requirements for 64-bit code.

NOTE - to keep code simple, I only showed the part which deals with NtQueryInformationProcess but other cases are similar.

As you can see, return back from x64 to the x86 world is a simple retf instruction. x86 code continues right where it left off:

Instruction at address 0x00ddaeba is the same for both x86 and x64 OS-es and VM continues as usual.

Different protection modes and syscalls

I provided you with 2 real-world test executables. Oppo seems to be simpler and use just 3 syscalls:

  • NtQueryInformationProcess with ProcessDebugObjectHandle class
  • NtSetInformationThread with ThreadHideFromDebugger class
  • NtProtectVirtualMemory to set protection attributes for each section in original executable

Asshurt doesn't have antidebug trick with NtQueryInformationProcess but it uses additional syscalls for some purposes:

  • NtOpenFile
  • NtCreateSection
  • NtMapViewOfSection
  • NtQueryVirtualMemory
  • NtUnmapViewOfSection
  • NtClose

Suggested workaround

Since VMProtect is using undocumented Windows features, it somehow needs to ensure that the protection will work on each and every Windows version. That's VMProtect's biggest strength and also the biggest weakness.

Windows' syscall numbers change in each version and also between major builds. Use the wrong syscall number and you're guaranteed to receive unexpected results. So, VMProtect developers had to hardcode a table with Windows build numbers and corresponding syscall id's in the executable.

You can see the syscall numbers in the j00ru's page (slightly out of date) or in tinysec's windows kernel syscall table

To obtain Windows build number, VMProtect uses information from PEB (Process Environment Block). The method is already described in The MASM Forum, so I'll just reproduce the (ugly) code from their page:

VMProtect checks only the build number and picks the corresponding syscall number. However, if the build number is not in the internal database, it will not use direct syscall and fall back to standard protection. Bingo, problem solved - no need for ugly hacks like Xjun's SharpOD plugin!

Hint: VMProtect 3.1 doesn't support Windows 10 Creators Update (build number 15063).

Demo time

As promised, here is a download link for the test application: https://mediafire.com/?niqqbs0fqcq8n23
Note: it should support most common builds of Windows XP/7/8.1/10. Windows 2003/Vista and other rare systems are not supported!

If it shows "OK" message, you've hidden your debugger well. If it shows "Debugger detected", you have a problem. smile

Have fun!
kao.

EDIT: Updated download link for Oppo. Mediafire's antivirus tends to have plenty of False Positives..