19 Aug 2017

Use of syscall and sysenter in VMProtect 3.1

Few days ago Xjun briefly mentioned a new feature of VMProtect 3.1 - it uses direct syscalls to check if software is running under debugger. I decided to take a quick look myself and figure out how exactly it works.

I'll give you 2 targets for the research:

  • oppo_flash_tool.exe (https://mega.nz/#!ZgJzjQxR!cNEHMwM-jKnLVgPXf4OUupyk1DNt69FYB2rEfY-5AlA) - it was given as an example by Xjun;
  • asshurt.dll (https://mediafire.com/?3xyc0ugc2hxervn) - some sort of a cheat for Roblox. I don't care about the cheat itself, it just happened to have the syscall feature enabled. And it uses different syscalls than Oppo.

In addition to that, I'll provide a very simple demo executable which replicates part of the VMProtect protection, so that you don't waste time looking at obfuscated code.

As a debugger in 32-bit OS you can use anything you like. On 64-bit OS you will really need to use WinDbg - as far as I know, it's the only debugger that can handle those tricks..

32bit OS

Let's start by debugging oppo_flash_tool.exe. First, we need to get past the usual tricks like IsDebuggerPresent and CheckRemoteDebuggerPresent. If you're reading this, I'm sure you know how to do that.

Few moments later we'll arrive here:

Remember this conditional jump. It's taken on 32bit OSes and not taken on 64-bit OS.

Let's look at 32bit OS version first. Now VMProtect prepares to call sysenter.

Since the code is obfuscated, here comes a cleaned-up version. Please note that in other applications, different registers can be used.

64-bit OS

Here it is getting interesting! smile You cannot use sysenter instruction from 32-bit code in 64-bit Windows. But, as ReWolf described few years ago, one can mix x86 code with x64 code in the same process. And that's exactly what VMProtect 3.1 is doing.

Let's go back to that conditional jump and see what happens in 64-bit OS. The jump will not be taken:

Far call?! Last time I saw that was in 16-bit Windows era..

As explained in ReWolf's article:

Summing things up, for every process (x86 & x64) running on 64-bits Windows there are allocated two code segments:

  • cs = 0x23 -> x86 mode
  • cs = 0x33 -> x64 mode

So, as soon as you execute that call, you'll switch to a 64-bit world. WinDbg happily recognizes that, all other debuggers just go astray..

x64 code does pretty much the same thing as x86 code - sets up a stack frame, sets up registers and then executes syscall instruction. Cleaned-up and shortened version follows:

You'll notice that x64 version is slightly more complex due to the way parameters are passed (registers vs. stack). It also includes a special treatment for 8 special edge cases - it will modify syscall parameters to adjust buffers and pointer sizes to satisfy requirements for 64-bit code.

NOTE - to keep code simple, I only showed the part which deals with NtQueryInformationProcess but other cases are similar.

As you can see, return back from x64 to the x86 world is a simple retf instruction. x86 code continues right where it left off:

Instruction at address 0x00ddaeba is the same for both x86 and x64 OS-es and VM continues as usual.

Different protection modes and syscalls

I provided you with 2 real-world test executables. Oppo seems to be simpler and use just 3 syscalls:

  • NtQueryInformationProcess with ProcessDebugObjectHandle class
  • NtSetInformationThread with ThreadHideFromDebugger class
  • NtProtectVirtualMemory to set protection attributes for each section in original executable

Asshurt doesn't have antidebug trick with NtQueryInformationProcess but it uses additional syscalls for some purposes:

  • NtOpenFile
  • NtCreateSection
  • NtMapViewOfSection
  • NtQueryVirtualMemory
  • NtUnmapViewOfSection
  • NtClose

Suggested workaround

Since VMProtect is using undocumented Windows features, it somehow needs to ensure that the protection will work on each and every Windows version. That's VMProtect's biggest strength and also the biggest weakness.

Windows' syscall numbers change in each version and also between major builds. Use the wrong syscall number and you're guaranteed to receive unexpected results. So, VMProtect developers had to hardcode a table with Windows build numbers and corresponding syscall id's in the executable.

You can see the syscall numbers in the j00ru's page (slightly out of date) or in tinysec's windows kernel syscall table

To obtain Windows build number, VMProtect uses information from PEB (Process Environment Block). The method is already described in The MASM Forum, so I'll just reproduce the (ugly) code from their page:

VMProtect checks only the build number and picks the corresponding syscall number. However, if the build number is not in the internal database, it will not use direct syscall and fall back to standard protection. Bingo, problem solved - no need for ugly hacks like Xjun's SharpOD plugin!

Hint: VMProtect 3.1 doesn't support Windows 10 Creators Update (build number 15063).

Demo time

As promised, here is a download link for the test application: https://mediafire.com/?niqqbs0fqcq8n23
Note: it should support most common builds of Windows XP/7/8.1/10. Windows 2003/Vista and other rare systems are not supported!

If it shows "OK" message, you've hidden your debugger well. If it shows "Debugger detected", you have a problem. smile

Have fun!
kao.

EDIT: Updated download link for Oppo. Mediafire's antivirus tends to have plenty of False Positives..

20 thoughts on “Use of syscall and sysenter in VMProtect 3.1

  1. Great article and knowledge kao! Thank you so much for sharing.

    I'll need to read your great blog-post (this one ;)) a number of times just to start understanding what you're explaining ... I'm a noob, sorry.

    The link for the "oppo_flash_tool" has been already blocked by mediafire ... can someone reupload it, please?

    Thanks and Best Regards,
    Tony

    • Glad you like it. If you have specific questions, please don't hesitate to ask - I'll try to explain it better. :)

      Mediafire relies on certain (ahem, ahem..) antivirus to detect malware, and it's prone to False Positives. Link updated.

  2. You are a life saver!

    I could not for the life of me get SharpOD to work with the other plugins I required to hide from my target application, then stumbled upon this post, you have no idea how much easier you've made my life :D Thanks so much!

  3. hello kao,
    your demo application run without error fired from desktop. Opened with debugger @ring3 level (all hiding tools checked), i get ERROR "Debugger detected" :)
    It is necessary to patch all involved system-files or can it only be debugged @ring0 level?

    • It can be done in ring-3 without patching any system files. I used Olly+modified ScyllaHide plugin to make that last screenshot in the blog post.

      (if you do not see a short movie there, you need to enable animated GIFs in your browser)

    • AFAIK, they changed something in v3.11. I haven't had time to look at it myself.

      But if you search for XJun's posts in tuts4you or Chinese forums, you should be able to find more information.

  4. The below code can help you to detect if the host OS is x32 or x64? Appreciate your explanation, how it's work?

    00dcfd8e f744250000800000 test dword ptr [ebp],8000h
    00dcfd96 e9f6bdfdff jmp 00dabb91
    ...
    00dabb91 0f84841a0b00 je 00e5d61b

    • Those 3 lines alone cannot detect anything. You need to find what is the value of "EBP", "dword ptr [EBP]" and where it's coming from. That's the important part. ;)

Leave a Reply to kao Cancel reply

  • Be nice to me and everyone else.
  • If you are reporting a problem in my tool, please upload the file which causes the problem.
    I can`t help you without seeing the file.
  • Links in comments are visible only to me. Other visitors cannot see them.

Your email address will not be published.

four  +  6  =