11 Mar 2016

About .NET, googling and lazy programmers.

Delphi fail. .NET win.

Recently, several people sent me bug reports where my EnigmaVB unpacker failed to extract files. In all cases, the problem was caused by really large files, like 3.5GB in size. So, what's causing the problem?

EnigmaVB unpacker is a 32bit application written in Delphi. And Delphi streams are retarded. They look like they've been written in 1990s and were never updated. TMemoryStream uses a continuous memory range, so it can never support anything larger than 2GB. TFileStream internally uses longint, so it doesn't properly support files larger than 2GB. WTF?

So, I have two choices. I can either make a custom stream class in Delphi, or I can pick another framework and rewrite my tool to use that.

I'm not a programmer, I'm a reverser. I don't want to spend my time developing custom stream classes. I'd very much rather use this time breaking some code. So, say hello to .NET framework - my next version of EnigmaVB unpacker will be made in C#.. smile

Am I a programmer or a googler?

While researching all the Delphi limitations and possible workarounds, I ran into this great article by Scott Hanselman. Reading both the post and the comments made me think a lot.

Does using Google to solve your programming tasks makes you less of a programmer? I don't think so.

In fact, I'm just lazy. Most people are. Why should I spend 30 minutes remembering basic algorithms for converting string to hex, if Google query can solve it in 10 seconds? Why reinvent the wheel and write CRC calculation from scratch? I'll just open StackOverflow and have a solution that's already tried and tested. It doesn't mean I can't do those boring tasks - I just don't want to.

How about you? Would you be able to write some tools without using Google and StackOverflow?

29 Jun 2015

Linking OMF files with Delphi

Continuing the discussion about Delphi compiler and the object files.

Here is the OMF file template I made for 010 Editor: https://www.mediafire.com/?bkpbkjvgen7ubz1
omf_parser

Please note, it is not a full-featured implementation of OMF specification. I only implemented all OMF file records that are processed by Delphi 2007 compiler. So, next time you have a cryptic compiler error while trying to link OMF file in Delphi, you can take a look into your OBJ file and make an educated guess what's causing the problem.

TL;DR version

In 95+% of cases you will encounter OBJ file that has unsupported segment name in SEGDEF record. And it's a simple fix, too - you just need to use objconv.exe by Agner Fog and use -nr option to rename the offending segment. Something like this:

Next possible issue is exceeding the number of EXTDEF or LNAMES records - this can happen if you're trying to convert a really large DLL file into OBJ file.

Finally, your OBJ file might contain some record type which is not supported by Delphi compiler at all. I'm not aware of a simple way to fix it, I would try using 010Editor and OMF template to remove the entire record.

If your problem is not caused from any of the above issues, please feel free to drop me a note - I'll be happy to look into it.

Known limitations of Delphi compiler

This is a list of limitations I was able to compile and/or confirm. Some of them come from Embarcadero official notes and the rest I obtained by analyzing dcc32.exe.

SEGDEF (98H, 99H)

  • Not more than 10 segments - if number of segments exceeds 10, buffer overrun will probably happen.
  • Segments must be 32bits. Will cause "E2215 16-Bit segment encountered in object file '%s'"
  • Segment name must be one of (case insensitive):
    • Code segments: "CODE", "CSEG", "_TEXT"
    • Constant data segments: "CONST", "_DATA"
    • Read-write data segments: "DATA", "DSEG", "_BSS"

    Segment with any other name will be ignored.

LNAMES (96H)

Not more than 50 local names in LNAMES records - will cause "E2045 Bad object file format: '%s'" error.

EXTDEF (8CH)

Not more than 255 external symbols - will cause "E2045 Bad object file format: '%s'"
Certain EXTDEF records can also cause "E2068 Illegal reference to symbol '%s' in object file '%s'" and "E2045 Bad object file format: '%s'"

PUBDEF (90H, 91H)

Can cause "E2045 Bad object file format: '%s'" and "F2084 Internal Error: %s%d"

LEDATA (A0H, A1H)

Embarcadero says that "LEDATA and LIDATA records must be in offset order" - I am not really sure what that means. Can cause "E2045 Bad object file format: '%s'"

LIDATA (A2H, A3H)

Embarcadero says that "LEDATA and LIDATA records must be in offset order" - I am not really sure what that means. Can cause "E2045 Bad object file format: '%s'"

FIXUPP (9CH)

This type of record is unsupported, will cause immediate error "E2103 16-Bit fixup encountered in object file '%s'"

FIXUPP (9DH)

Embarcadero documentation says:

  • No THREAD subrecords are supported in FIXU32 records
  • Only segment and self relative fixups
  • Target of a fixup must be a segment, a group or an EXTDEF

Again I'm not sure what they mean. But there are lots of checks that can cause "E2045 Bad object file format: '%s'"

THEADR (80H)

Accepted by compiler, but no real checks are performed.

LINNUM (94H, 95H)

Accepted by compiler, but no real checks are performed.

MODEND (8AH, 8BH)

Accepted by compiler, but no real checks are performed.

COMMENT (88H) and GRPDEF (9AH)

Ignored by compiler.

That's the end of the list. Any other entry type will cause immediate error "E2045 Bad object file format: '%s'" smile

Useful links

My OMF file template for 010Editor: https://www.mediafire.com/?bkpbkjvgen7ubz1
OMF file format specification.
The Borland Developer's Technical Guide
Objconv.exe by Agner Fog
Manual for objconv.exe

19 May 2015

Static linking of Bassmod in Delphi

BASS and BASSMOD are very well known freeware libraries for playing XM, IT, WAV and many more sound file formats. They are widely used in keygens and other apps. However, authors only distribute them in a DLL form, there is no LIB file or any other option for linking them statically.

Last week someone resurrected an old thread at Tuts4You and asked how to convert DLL to LIB and link it statically with Delphi. I gave poster the standard answer but he was still running in all sorts of issues. So, can it be done?

Then answer is - yes. But it's not easy.

Note - This article is written for good old Delphi 6/7/2007. Since Delphi XE2 the process should be easier as the linker was improved to use COFF OBJ files directly. However, I don't have those new Delphi versions, so I can't test the claims.

Steps to be taken

I'll make a list of all necessary steps first and then I'll discuss them in details.

  1. Unpack the DLL properly
  2. Convert DLL to LIB
  3. Extract OBJ files from LIB
  4. Convert COFF OBJ files to OMF OBJ files
  5. Make those OMF files usable by Delphi
  6. Write a wrapper unit that works around Delphi limitations

Unpack the DLL properly

First step sounds easy, right? It isn't.

The LIB provided in Tuts4you thread is badly unpacked. Sure, it can work when compiled with MASM. It can be made work with Delphi, but you'll need to hex-edit compiled EXE file first. The reason for this is extremely primitive Delphi compiler/linker. You have no control over PE section names or attributes. It relies on specific section names and always makes code section read-only. But the LIB from tuts4you uses one segment for both code and data and it must have read-write-execute characteristics. Ooops.

So, we need to unpack DLL ourselves using all the standard steps. PE packer is a very simple one, so you can easily find OEP, dump the file, load DLL at different imagebase, find the OEP and make a 2nd dump, use 2 dumps to fix the relocations using Relox and finally restore Import Table using Scylla or ImpRec. Nothing new here.

Once you've unpacked the DLL, you will have to detect section boundaries and create new PE section table. When you're at OEP of bass.dll, check the memory map in some process exploring tool. You'll see the sections and their characteristics nicely:
BASS memory map in PETools

Now use any PE editor to create appropriate PE section headers:
Section headers in CFF
To make Delphi happy, code section should be named _TEXT and data section should be named _DATA. All the sections you don't need in final OBJ file, should be named ".reloc", ".edata" or similar - Dll2lib will remove them automatically.

Convert DLL to LIB

Well, this step is easy. Use DLL2LIB (google "DLL.To.Lib.v1.42.Full.Retail-DLL2Lib" or get trial version from official site), leave all the default settings and press "Start convert".

Dll2lib

Extract OBJ files from LIB

For next few steps you'll need objconv.exe by Agner Fog. It's better to download the latest version, as earlier versions didn't support extracting LIB files.

It's a simple command objconv.exe -lx bass.lib

Convert COFF OBJ files to OMF OBJ files

That's also simple. Just run objconv.exe -fomf bass.obj bass-omf.obj

Make those OMF files usable by Delphi

Delphi imposes quite a few limitations to OBJ file format. Some of them are documented, some of them aren't. So, it's better to rely on special tools made for this purpose, like omf2d.exe by EliCZ.

I'm sure that objconv.exe can do the same, but I'm too lazy to try to figure the right command line parameters. So, just run omd2f.exe bassmod-omf.obj bassmod-omf-d.obj

Note - omd2d.exe will mess up some decorated names from msvcrt.dll, like "??2@YAPAXI@Z". That's not a problem, we'll fix that in the wrapper unit.

Write a wrapper unit that works around Delphi limitations

This is also tough. And again the problems are caused by the primitive Delphi compiler/linker.

Delphi doesn't support direct API calls, all API calls will go through the thunk table. When you try to reference any external API from Delphi code, in reality you'll get address of the thunk code.

For the same reason in Delphi you can't access exported global variables from another DLL.

Unfortunately BASS/BASSMOD uses both direct API calls and global variables from msvcrt.dll. Little bit of clever hacking is required to work around that - you'll have to load msvcrt and other DLLs from unit initialization code and use GetProcAddress to get the required addresses.

So, the implementation part of the unit will look like this:

In addition to that we need to call the original DllMain function to make sure that BASS is initialized properly:

As a final touch, in the finalization part of unit we'll have to call DllMain again to make sure all resources are freed properly.

Putting it all together

I already outlined all the steps needed. Anyone with proper skills should be able to replicate them and make his/her own BASS unit.

For those who are lazy - here is the package with Delphi units+obj files + all the intermediate files + compiled projects from BASS/BASSMOD examples to show that it really works.

Have fun!

Useful links

Unpacking DLLs #1: Tutorial by Mr. eXodia
Unpacking DLLs #2: How to use Relox in few simple steps
Omf2d: https://www.mediafire.com/?hsksyjwnwlaw3zb