On Software Reverse Engineering

truth, April 7, 2004

This article discusses the methods of software reverse engineering and the case study of FLEXlm system.

Problem Description

I downloaded IMSL CNL (C Numerical Library) 5.5 from ftp.vni.com but it’s protected by FLEXlm. There are different
binary downloads for different OS and compilers, but the general cracking techniques apply across platforms (in our
case the FLEXlm license file is certainly platform independent). Here we address CNL for Microsoft Windows and
Visual Studio. The product mainly includes libraries (static and dynamic) of mathematical and statistical subroutines
under FLEXlm feature names “CMATH” and “CSTAT”. The x86 distribution also has an optimized version of CMATH
called “CMPERF” that utilizes the bundled Intel MKL 6.1 to achieve high performance1. No additional licensing is
needed for CMPERF. Since CMATH and CSTAT use the same licensing mechanism, we will focus on CMATH from now
on, the procedure for CSTAT is completely similar.

The setup provides a simple program cmath.c for validation purpose. Without loss of generality, we link its object
file with cmath_s.lib (the static library) to get the executable. It calls imsl_f_lin_sol_gen() and checks the
license file, if the license file is not right, error is reported. Searching the web we found the following license file for
IMSL CNL 5.0

SERVER hostname hostid 27000

DAEMON VNI "<vni_dir>\license\bin\bin.i386nt\vni.exe"
FEATURE CMATH VNI 5.0 permanent uncounted 3F23BE3056E4 HOSTID=ANY
FEATURE CSTAT VNI 5.0 permanent uncounted 2C60CD4570B0 HOSTID=ANY

We tried it and expectedly got “version incorrect” error, replacing 5.0 by 5.5 we got “incorrect softkey code” error,
so obviously naïve approach does not work. In fact it takes quite sophisticated work to beat FLEXlm. There are
different levels of software cracking, and the associated complexity ranges from relatively simple to dauntingly
difficult – we will see that later. We now list our task and tools used in below.

Target: Visual Numerics IMSL CNL 5.5

Protection: Macrovision FLEXlm 9.2
Tools: Microsoft Visual Studio 7.1 (CL, NMAKE, DUMPBIN, LIB, …)
RedHat Cygwin 1.3.5
IDM UltraEdit 9.0
Datarescue IDA Pro 4.3 (FLAIR, …)
URSoft W32Dasm 8.9
Sysinternals File Monitor 6.0
Resources: Macrovision FLEXlm 9.2 SDK source code
Macrovision FLEXlm 8.1 SDK binary release

Preliminary Attempts

All hackings start from gathering information. For example, it’s very easy to find out the target employs FLEXlm 9.2
for license management by searching cmath_s.lib in UltraEdit. Reading relevant literature also very helpful. There
are some excellent essays in [2] describing previous attacks against earlier versions of FLEXlm, which contain a lot
of precious knowledge.

In reality I loaded cmath.exe into W32Dasm debugger and started tracing pretty soon. It is horrendous experience
tracing through jungles of assembly code, you can spend hours jumping back and forth without getting any clues
about what is really going on. But I managed to figure out something anyway (with the help of File Monitor).

004133DE E89D2F0100 call 00426380 ;read license.dat

... ...
00413411 – 0041345B ... ... ;a loop read out “CMATH”

1
Note the difference between IMSL and MKL. MKL functions are low-level, fundamental subroutines like BLAS, LAPACK,
FFT, etc. IMSL, on the other hand, contains much more higher-level functionalities such as differential equation solvers and
statistical regressions. MKL serves as good building blocks for many IMSL functions.
... ...
004138CD E83E670000 call 0041A010 ;return must be 0 to pass check
004138D2 83C41C add esp, 0000001C
004138D5 8945F0 mov dword ptr [ebp-10], eax ;return value stored in EAX
004138D8 837DF000 cmp dword ptr [ebp-10], 00000000
004138DC 0F848C070000 je 0041406E ;proceed to imsl_f_lin_sol_gen()
... ...
00413907 8B8D18F7FFFF mov ecx, dword ptr [ebp+FFFFF718]
0041390D 33C0 xor eax, eax
0041390F 8A81A7444100 mov al, byte ptr [ecx+004144A7]
00413915 FF24857F444100 jmp dword ptr [4*eax+0041447F] ;jump to error message

Apparently subroutine 0041A010 is the key. Stepping into it reveals more intricate structure – there are complicated
call chains inside. In practice the procedure returns FFFFFFF8 (or –8), which should be the error code. So we set a
breakpoint at 004138D5, using W32Dasm’s “Modify Data” button we change the value of EAX register to 0, and let
it go. Bang! The program is fooled and yields the correct result as if we had the right license data.

Now things become clear, we can patch the code section 004138D2 – 004138DC to set the return value to always
be 0. To do that we consult [4] for detailed x86 instruction format on MOV and JMP, and the following is the modified
code. Note that NOP is added to patch extra bytes left by the change so that the modified executable has the same
length as the old one. In fact they differ only on these three lines that we have changed.

004138CD E83E670000 call 0041A010 ;return must be 0 to pass check

004138D2 83C41C add esp, 0000001C ;stack pointer adjustment
004138D5 C745F000000000 mov [ebp-10], 00000000 ;pretend the return value is 0
004138DC 90 nop ;patch the extra byte to maintain code alignment
004138DD E98C070000 jmp 0041406E ;unconditional jump to imsl_f_lin_sol_gen()

Of course patching cmath.exe is just proof of concept, the real thing is to patch the library itself. cmath.c per se is
embarrassingly short, virtually all contents of cmath.exe come from cmath_s.lib, including both IMSL functions
and FLEXlm code. To locate the position of the above code, we search the binary string 837DF0000F848C070000
(code for CMP and JE lines) in UltraEdit2. This leads us to the unique location in cmath_s.lib (file offset 00106F70
– 00106F80) where those code lies. Change the bytes accordingly and we get a patched library. Test the program
again by linking it to the new patched library, everything works fine even though our license file is invalid.

So we’ve had our first success. Next we can do the same thing to DLL version of the library and ideally, develop a
utility to do that automatically rather than manually, but we’ll omit it for now. The point is, patching is usually the
easiest and first step of cracking, it requires little insight into the protection scheme. Moreover, patching works only
for a particular binary target, the patching utility cannot function for different versions or on different platforms.
Though powerful and effective as it is, patching is far from full reverse engineering.

More Analysis

We fooled FLEXlm by passing a fake “OK” to it, but we still don’t know the true license code. Most software protection
works in this way: Based on the user profile (name, organization, purchased feature, etc.) and certain algorithm,
the program calculates some hash/checksum/license code/signature and compares it to the one user provides. If
they match, the user is authenticated. This mechanism is almost universal to all software we have seen, and FLEXlm
is no different. The code 3F23BE3056E4 in our license file is the “SIGN=” signature, just it’s a wrong one.

One more word about FLEXlm: it’s called FLEXlm because it claims to provide a flexible solution to commercial
software license management. It does put a lot of efforts in treating various situations – counted/uncounted,
feature/incremental, server/local, borrowing, trial, mobile, … – but we are not interested in those things. What we
want is the right signature that enables us to run the specified target anytime, anywhere.

Since we have the source code of FLEXlm SDK 9.2, the logical thing to do is to read it. FLEXlm SDK is what
Macrovision gives to their client, the so-called “vendor”, and helps them to ship their own “vendor software” to the
“end user”. In our case Visual Numerics is the vendor, IMSL CNL 5.5 is the vendor software, and we are the end users.

2
Remember x86 architecture is little-endian. In some applications searching binary string can be tricky because we need
to reverse the byte order. Fortunately UltraEdit handles it well and we can do it as is.
Each vendor has a unique vendor name or vendor ID. As seen in the license, here it is “VNI”. Usually vendors only
get binary release or partial source of FLEXlm SDK, but we have the luck to obtain the whole source for that, which
is what FLEXlm is all about.

However, it turns out that the all–C source (no C++) are not very readable. The project evolves over more than a
decade (version 1.0 release in 1988); the old and new functions overlap/intertwine like spaghettis, often with
unnecessary redundancies; it is poorly commented and some old style coding conventions are very bad; the
overuse of macros and preprocessing directives are very annoying. Developed originally on UNIX platforms, it is
ported to Windows environment by NMAKE utility. To efficiently build and debug such a large application we need a
good IDE, but there is no IDE under Windows that can take in the Makefile directly. As Visual Studio is possibly the
best IDE on Windows and has “Makefile Project” capability, we set out to create a VS7 project for FLEXlm SDK. It
took me some time to do that – need to fix some errors in makefiles – but when it’s done it’s really convenient.

The core component of FLEXlm is lmgr.lib (or lmgr9a.dll), on which all others heavily depend. Vendors and end
users are more familiar with the tools like lmgrd.exe, lmtools.exe, lmnewgen.exe, makekey.exe, etc. After
successful building, we try to generate VNI license file but failed because we don’t have their vendor keys and seeds.
According to [2], [5], [6], each vendor receives 5 vendor keys (VENDOR_KEY1, … VENDOR_KEY5) from Macrovision
and they themselves choose 3 random seeds (LM_SEED1, LM_SEED2, LM_SEED3). These eight numbers are placed
in lm_code.h and then encrypted, obfuscated, and finally built into the target as well as the tools that generate the
license. Our job is of course to recover these numbers, but how?

Or we can try a less difficult way: since the target will calculate the real signature and compare it against the one
it reads from the license file, we can go catch the real signature when the comparison takes place without knowing
vendor keys and seeds. This is not as easy as it may seem, it requires us to set the breakpoint in the right place at
the right time. After hours of tracing we determined that simply following the instruction flow was a dead end, it’s
hopeless to locate the comparison code in this way unless FLEXlm uses some standard APIs like ANSI strncmp() or
Win32 CompareString() (actually FLEXlm defines its own macro STRNCMP in l_privat.h).

This brings us to a fundamental question on reverse engineering: how can we make sense out of the chaotic,
high-entropy assembly code? There are no complete answers to this question and we refer the interested reader to
[1] for some serious theoretic discussions. But here we want to focus on a practical technology on this issue: FLAIR
(a.k.a. FLIRT, c.f. [3]) introduced by IDA Pro.

We all know that debugging can be made much easier if the application is built as debug version and symbols files
(.PDB, .DBG files) are available. Symbols are information about the program including identifier (variable, function)
names and memory offsets, source code line number, etc. In a debug build compiler/linker saves these info either
into the application binary or to separate symbol files. They enable debuggers to present the users code that closely
resembles the original source (or even better, like source debugging in VS). Naturally symbols are stripped off from
software releases delivered to end users, as in our case.

But this is not end of the story. Although without symbols, we can still get something useful from the binary files,
especially libraries. On Windows .EXE and .DLL are PE format while .OBJ and .LIB (LIB is no more than a pile of OBJs
stacked together) are COFF format; in both formats library calls are made via function names and arguments, which
have to be publicly visible3. For that purpose PE has imports & exports sections and COFF has symbol table (note PE
kind of contains COFF as a subdivision). Visual Studio offers a couple of commands to explore them4:

F:\>dumpbin /disasm %vni_dir%\cnl55\cmath.exe

F:\>dumpbin /rawdata %vni_dir%\cnl55\cmath.exe
F:\>dumpbin /imports %vni_dir%\cnl55\cmath.exe
F:\>dumpbin /exports %vni_dir%\cnl55\bin\cmath.dll
F:\>dumpbin /imports %vni_dir%\cnl55\bin\cmath.dll
F:\>dumpbin /exports %vni_dir%\cnl55\lib\cmath.lib
F:\>dumpbin /symbols %vni_dir%\cnl55\lib\cmath_s.lib
F:\>dumpbin /linkermember %vni_dir%\cnl55\lib\cmath.lib
F:\>dumpbin /linkermember %vni_dir%\cnl55\lib\cmath_s.lib
F:\>dumpbin /archivemembers %vni_dir%\cnl55\lib\cmath.lib
F:\>dumpbin /archivemembers %vni_dir%\cnl55\lib\cmath_s.lib

3
API calls in DLL could be based on ordinals (indexes). Since ordinals have bijective mapping to function names, it’s an
indirect way of calling by names. However COM DLL APIs may not be outside visible, see footnote 5.
4
Here cmath_s.lib and cmath.dll are the “real thing” while cmath.lib is just the import library for cmath.dll.
F:\>lib /list %vni_dir%\cnl55\lib\cmath.lib
F:\>lib /list %vni_dir%\cnl55\lib\cmath_s.lib
F:\>lib /extract:vc++\flexlm.obj %vni_dir%\cnl55\lib\cmath_s.lib
F:\>dumpbin /symbols /disasm flexlm.obj

We must stress the difference between LIB and DLL here, it’s more than merely static linking vs. dynamic linking.
There are generally three stages in developing a program:
Original: Source File (C, H, C++…), ASCII format
Intermediate: Object File (OBJ, LIB…), COFF format
Final: Image File (EXE, DLL, SYS…), PE format
Compiler processes source file to produce object file, linker takes in object file to output image file, and loader loads
image file from disk into memory. Note DLL has already been processed by linker (after-linking), turning identifiers
into memory addresses or “replaced letters with numbers”. In contrast object files are “before-linking” and have to
retain the original symbols; otherwise linker cannot resolve them. Hence DLL is much closer to EXE than to LIB in
spite of its name as “library”. The “dynamic linking” part is actually done by system loader at runtime (some
fixup/relocations), not by the linker pass. You can say this is an M$ trick that uses misleading terms to confuse
people and conceal technical gist, which they are very good at.

From a hacker’s point of view, this means the later the stage, the higher the entropy, the less the information. In
particular, LIB provides more clues to us than DLL does. DLL exports only public APIs and hides the private ones (e.g.
in cmath.dll only IMSL APIs are exported while FLEXlm functions are kept internal), but LIB symbols include both.
In fact, to build a program that calls DLL APIs we need to pass its import library to the linker5. Import library is a
COFF LIB file that servers as a symbol reference (by pointing to the DLL) and does not contain function bodies.

Now we can find out what functions are there in a library file, locate our concerned function, and even extract the
corresponding object file (only for LIB). The following is an example with imsl_f_lin_sol_gen(). As we said before,
debug info, which reside in PE debug section or separate symbols files, are as best as we can get only next to source
code (debug info lies roughly between the first and second stage). Nevertheless, what we obtain here is still very
important in reverse engineering (debug-build symbols is a superset of COFF symbols).

F:\>dumpbin /archivemembers /symbols %vni_dir%\cnl55\lib\cmath_s.lib|egrep "member|imsl_f_lin_sol_gen”

... ...
Archive member name at 1C8CEE: vc++\gmres.obj/
00B 00000000 SECT3 notype () External | _imsl_f_lin_sol_gen_min_residual
... ...
Archive member name at 1DAAAA: vc++\fspgen.obj/
00B 00000000 SECT3 notype () External | _imsl_f_lin_sol_gen_coordinate
05A 00000000 UNDEF notype () External | _imsl_f_lin_sol_gen
... ...
Archive member name at 1FC10E: /3128 vc++\fdmbndg.obj
007 00000000 SECT2 notype () External | _imsl_f_lin_sol_gen_band
... ...
Archive member name at 58AC3E: /6132 vc++\flinslg.obj
007 00000000 SECT2 notype () External | _imsl_f_lin_sol_gen
... ...

F:\>dumpbin /exports %vni_dir%\cnl55\bin\cmath.dll | grep -i imsl_f_lin_sol_gen

438 1B5 000432C0 imsl_f_lin_sol_gen
439 1B6 0023B130 imsl_f_lin_sol_gen_band
440 1B7 0024AE30 imsl_f_lin_sol_gen_coordinate
441 1B8 00259C30 imsl_f_lin_sol_gen_min_residual

F:\>lib /extract:vc++\flinslg.obj %vni_dir%\cnl55\lib\cmath_s.lib

IDA FLAIR moves one step further. Albeit in source code API calls look like “x = imsl_f_lin_sol_gen(n, a, b, 0);”,
it appears in disassemblies as “call 004033B0” (static linking) or “call [00402054]” (dynamic linking). Appropriate
labeling of function names along side of their memory addresses can drastically ease the assembly analysis, and
that’s exactly what FLAIR does. Most debuggers have such commenting functionality, but usually restricted to
exported APIs. IDA tries to extend it to include as many functions as possible.

5
Refers to traditional SDK DLL. Unlike SDK DLL where PE exports section is indispensable, the newer COM DLL employs
a totally different calling mechanism called automation. Member methods are invoked through interface pointers rather
than being exported directly. Thus the tightly encapsulated COM DLL gives us even less info and more challenge.
The idea of FLAIR is to create a “signature” for every identifiable library function so that when IDA analyzes the
assembly code it can recognize and label it. It is essentially a pattern recognition problem as its name indicates.
Again it works only for LIB, not for release-version DLL, due to their content differences. We must say that DLL does
have advantages such as code sharing and main program simplification. For instance, the size of statically linked
cmath.exe is about 700KB but that of dynamically linked cmath.exe is less than 4KB. But as far as cracking is
concerned, LIB is way better than DLL (when tracing DLL linked application, most of the time we are in 10000000+
or 80000000+ area instead of the familiar 00400000+ region; in DLL-version cmath.exe the instruction “102D12BA:
call 102D7980” returns FFFFFFF8).

IDA FLAIR is not perfect – it can’t handle DLL, some functions can’t be identified, false recognition could happen…
– yet it is very practical. Its original goal is to isolate boilerplate APIs (such as Win32, MFC, ATL, etc.6) so that people
can focus on main program algorithm instead of those standard library functions. In our case we are more interested
in getting those FLEXlm functions highlighted so we don’t need to step into every calls to get a big picture of the
whole code maze. In reality we created signature files of cmath_s.lib and lmgr.lib, applied them to cmath.exe in
IDA, and FLAIR did very well – it recognized most FLEXlm functions. As an outstanding static analysis toolbox, IDA
also offers a WinGraph32 feature called “Display Flow Chart”. I found it especially useful to facilitate understanding
of the code when contrasted side by side to the source.

FLEXlm Architecture

Equipped with VS + FLEXlm source, W32Dasm + cmath.exe, and IDA + cmath.exe (with signature), now we should
be able to unveil the FLEXlm kernel. The followings are some of our findings, where lm_ckout.c!lc_checkout()
means “function lc_checkout() in the module/file lm_ckout.c” and the arrow symbol denotes function call. Notice
that due to branching only portions of the code are traced through and presented, but in general we are only
interested in those branches anyway.

0047D0C6: push 00000018 ;program entry point

0047D22D: call 00401000
0047D240: call 0047F20D
sub_401000: cmath.exe!main()
0040101D: call 004033B0
00401039: call 00401050
sub_004033B0: vc++\flexlm.obj!imsl_f_lin_sol_gen()
004033C8: call 00408F24
0040342C: call 004034A0
sub_00408F24: vc++\error.obj!imsl_e1psh()
00408F3A: call 0040A850
00408F76: call 00414AFD
sub_00413290: vc++\flexlm.obj!imsl_flexlm()
004132EF: call 004294A0
004133A4: call 00426380
004133DE: call 00426380
00413486: call 00426380
004134C0: call 00426380
004134FB: call 00426380
004135A7: call 00426380
004136A6: call 0042420C
004138CD: call 0041A010
00414099: call 0047FA9B
004141C6: call 0042563D
0041434E: call 0047F8F0
sub_0041A010: lm_ckout.c!lc_checkout()
0041A093: call 0041A14B
;call cmath.exe!main()
;call chain to ntdll.dll!NtTerminateProcess()
;call vc++\flexlm.obj!imsl_f_lin_sol_gen()
;call vc++\fwrimat.obj!imsl_f_write_matrix()
;call vc++\error.obj!imsl_e1psh()
;call vc++\flinslg.obj!l_lin_sol_gen() to do the real work
;call chain vc++\single.obj!imsl_once() -> vc++\error.obj!l_error_init()
;-> vc++\flexlm.obj!imsl_flexlm()
;call imsl_highland_check() -> l_check.c!lc_timer() -> l_timer_heart()
;-> l_check() -> l_reconnect() -> lm_ckout.c!l_checkout() as heartbeat
;call lm_njob.c!lc_new_job()
;set LM_A_DISABLE_ENV to 1
;set LM_A_LICENSE_FILE_PTR to license file location
;set LM_A_CHECK_INTERVAL to -1
;set LM_A_RETRY_INTERVAL to -1
;set LM_A_RETRY_COUNT to -1
;set LM_A_LINGER to 0
;call l_check.c!lc_status(), returns LM_NEVERCHECKOUT
;call lm_ckout.c!lc_checkout()
;returns current date and time
;call lm_config.c!lc_get_config()
;check if license is expired, returns 0 if not
;call lm_ckout.c!l_checkout(), returns FFFFFFF8

6
They claim that modern real life applications contain 50+% of such standard API calls, see [3].
sub_0041A14B: lm_ckout.c!l_checkout()
0041A2E8: call [004AA01C] ;call lm_ckout.c!lm_start_real(), returns FFFFFFF8
sub_0041A875: lm_ckout.c!lm_start_real()
0041AA47: call 0041B4A5 ;call lm_ckout.c!l_local_verify_conf(), returns 1=success
0041AC01: call 0041BD89 ;call lm_ckout.c!l_good_lic_key(), returns 0=failure
sub_0041BD89: lm_ckout.c!l_good_lic_key()
0041BE30: call 00433D15 ;call l_getattr.c!l_xorname()
0041BE4D: call 0041DB9E ;call lm_ckout.c!l_sg()
0041C202: call 0041EBE3 ;call vc++\lm_ckout.obj!l_crypt_private(), returns 0
sub_0041DB9E: lm_ckout.c!l_sg()
0041DBF7: call [004AD064] ;call lm_new.c!l_n36_buff()
0041DC16: call 00443283 ;call l_key.c!l_key()
sub_0041EBE3: vc++\lm_ckout.obj!l_crypt_private()
0041EC07: call 0041EE44 ;call vc++\lm_ckout.obj!real_crypt(), returns 0
sub_0041EE44: vc++\lm_ckout.obj!real_crypt()
0041F9B6: call 00420AF6 ;call vc++\lm_ckout.obj!l_string_key(), returns 0
sub_00420AF6: vc++\lm_ckout.obj!l_string_key()
00420E94 - 00421156 ;invoke macro XOR_SEEDS_INIT_ARRAY(xor_arr)
00421247: call 0047F250 ;call strcpy(lkey, license_key)
0042145E: call 004803A0 ;call memcpy(newinput, input, inputlen)
0042191D: call 00421D66 ;call l_strkey.c!our_encrypt()
00421A13 - 00421B27 ;for{} loop license key matching
00421B34: call 00421C22 ;call l_strkey.c!atox() to convert binary string to ASCII text
sub_00426380: lm_set_attr.c!lc_set_attr()
004263E4: call 0042641E ;call lm_set_attr.c!l_set_attr() to set attributes in config structure
sub_0042641E: lm_set_attr.c!l_set_attr()
00427045: call 00427DBC ;if setting LM_A_LICENSE_FILE_PTR, call lm_set_attr.c!l_set_license_path()
;-> lm_config.c!l_flush_config() -> l_init_file.c!l_init_file()
;-> l_allfeat.c!l_allfeat() -> l_allfeat.c!l_parse_feature_line()
;-> l_allfeat.c!oldkey() -> vc++\l_allfeat.obj!l_crypt_private()
sub_004294A0: lm_njob.c!lc_new_job()
004294C0: call [004A5A98] ;call lm_new.c!l_n36_buf(), returns 1
004294D2: call [004A5A98] ;call lm_new.c!l_n36_buf() with all 0 arguments, returns 0
004294E8: call 0044357F ;call chain lm_init.c!lc_init() -> lm_init.c!l_init()
004294FD – 00429516 ;turn on LM_OPTFLAG_CUSTOM_KEY5 flag
sub_0043873B: vc++\l_allfeat.obj!l_crypt_private()
0043875F: call 00438771 ;call vc++\l_allfeat.obj!real_crypt(), returns 21D5B6E8572E
sub_00438771: vc++\l_allfeat.obj!real_crypt()
004392DC: call 0043A41C ;call vc++\l_allfeat.obj!l_string_key(), returns 21D5B6E8572E
sub_0043A41C: vc++\l_allfeat.obj!l_string_key()
sub_0044359E: lm_init.c!l_init() ;take in VENDORCODE and VENDORNAME to initialize job structure
00443E8F – 00444354 ;some validity test, could report error
004441F9: call 0041DB9E ;call lm_ckout.c!l_sg()
sub_0044A110: lm_new.c!l_n36_buf() ;initialize VENDORCODE structure and VENDORNAME
0044B503: push 00450BE0 ;push in lm_new.c!l_n36_buff() address
0044B508: call 00444B11 ;call lm_init.c!l_x77_buf() to set L_UNIQ_KEY5_FUNC
sub_00450BE0: lm_new.c!l_n36_buff()
00450C34 – 00450EB0 ;obfuscate mem_ptr2_bytes[] in job
00450EB5 – 00450FF8 ;obfuscate data[0] and data[1] in VENDORCODE
sub_77F8DD80: ntdll.dll!NtTerminateProcess()
77F8DD8B: ret 00000008 ;program exit

As we can see, the subroutine 0041A010 we mentioned earlier is actually lm_ckout.c!lc_checkout(), indeed a
crucial function. It returns 0 if the license is successfully checked out, otherwise it returns an error code (FFFFFFF8
is defined as LM_BADCODE in lmclient.h). l_checkout() may be invoked several times due to heartbeat but we
don’t care. Remember our aim is to position the checksum comparison code and retrieve the real signature.
A quick search tells us that STRNCMP (a macro defined in l_privat.h, sets result=0 if strings match) appears only
in lm_ckout.c!l_good_lic_key() and l_crypt.c!l_crypt_private(). Note l_crypt.c and l_strkey.c are not
directly compiled to objects, instead they are included in modules lm_ckout.obj and l_allfeat.obj. In addition to
those two, lm_crypt.obj also includes l_crypt.c and exposes its functions to the outside as APIs, many under
different aliases such as lc_crypt(). However the copies in lm_crypt are not used in cmath.exe: the address of
lc_crypt() is 00469960, at which we set breakpoint, but nothing happened.

lm_ckout.c
... ...
#define l_crypt_private l_ckout_crypt
#define l_string_key l_ckout_string_key
... ...
/* Include l_crypt.c, so that these functions won’t be global. */
#define LM_CKOUT
#include "l_crypt.c"
l_allfeat.c
... ...
#define LM_CRYPT_HASH
#include "l_crypt.c"
l_crypt.c
#include "l_strkey.c"
l_strkey.c
#include "l_strkey.h"
lm_crypt.c
#define l_crypt_private lc_crypt
... ...
#define LM_CRYPT
... ...
#include "l_crypt.c"

Included more than once, the multiple copies of l_crypt/l_strkey functions are not all the same due to compiling
directives. The following l_crypt_private() code illustrates that (STRNCMP here doesn’t really matter). Depending
on whether LM_CKOUT is defined, subroutines 0041EBE3 (long version) and 0043873B (short version) differ in the
two modules. IDA FLAIR recognizes the latter but not the former – in practice we have to manually identify those
functions in lm_ckout.obj.

ret = real_crypt(job, conf, sdate, code);

#ifdef LM_CKOUT
if (!(job->user_crypt_filter) && !(job->lc_this_keylist) && valid_code(conf->code))
{
if (job->flags & LM_FLAG_MAKE_OLD_KEY)
{
STRNCMP(conf->code, ret, MAX_CRYPT_LEN, not_eq);
}
else
{
STRNCMP(conf->lc_sign, ret, MAX_CRYPT_LEN, not_eq);
}
if (not_eq && !(job->options->flags & LM_OPTFLAG_STRINGS_CASE_SENSITIVE))
{
job->options->flags |= LM_OPTFLAG_STRINGS_CASE_SENSITIVE;
ret = real_crypt(job, conf, sdate, code);
job->options->flags &= ~LM_OPTFLAG_STRINGS_CASE_SENSITIVE;
}
}
#endif /* LM_CKOUT */
return ret;

In the above code comments, we see that l_crypt_private() is the key function that calculates the SIGN hash.
There are two call chains involved, initiated from lc_set_attr() when setting LM_A_LICENSE_FILE_PTR and
lc_checkout() respectively. The tracing result of l_crypt_private() was 21D5B6E8572E in the first case and 0 in
the second. Clearly this looks like the signature code and we immediately tried it in the license file. Unfortunately it
did not work.
Well, that’s not very surprising; the surprising thing is, why would lc_set_attr() compute the checksum? Setting
attributes should have nothing to do with authentication. We must point out, however, that during the call chain
every line of license file is parsed, literally! In our case there are 3 lines in license.dat but only 1 feature line, so
l_parse_feature_line() is called three times but oldkey() is called only once. The most probable explanation,
we think, is that it just fills a hash code to the config structure as initialization, as the name oldkey() suggests. Or,
for conspiracy theory fans, you can say this is a trick that lures crackers away from the real thing to some
camouflaged petty codes in order to waste their time.

OK, we are not that stupid, we know it is the checkout call chain that counts. Because later half of the chain returns
0 rather than the true hash, l_good_lic_key() reports failure (1=success, 0=failure), then lm_start_real() sets
error number accordingly and it’s relayed back to lc_checkout(). Note the STRNCMP code in l_good_lic_key()
does not behave as we expected. If the signature in license file is incorrect, then l_crypt_private() returns 0 and
STRNCMP is bypassed; if it is correct, then l_crypt_private() returns the same string and STRNCMP is totally
meaningless. Either way STRNCMP does not compare the wrong checksum with the right one, as we speculated.
Again you may have two perspectives on why FLEXlm did this.

code = l_crypt_private(job, conf, sdate, &vc);

... ...
if (job->user_crypt_filter)
{
if (!code || !*code)
str_res = 1;
}
else
{
if (conf->lc_keylist && job->L_SIGN_LEVEL)
{
if (!code || !*code || !*conf->code) /*P5552 */
str_res = 1;
else
STRNCMP(code, conf->lc_sign, MAX_CRYPT_LEN, str_res);
}
else
{
if (!code || !*code || !*conf->code) /*P5552 */
str_res = 1;
else
STRNCMP(code, conf->code, MAX_CRYPT_LEN, str_res);
}
}
if (str_res)
{
... ...
}
else
ok = 1;

Ignoring STRNCMP, we realized that we had to trace all the way down to the bottom of the call chain to dig out the
genuine signature – it has to be computed and compared to license file input somewhere! As it turns out, the place
is l_string_key(). The user file signature is passed in as argument, the correct license key is calculated, and then
the two are matched bit by bit. So this is where the right and wrong are revealed, not those phony STRNCMPs.

#ifdef LM_CKOUT
static unsigned char * l_string_key(job, input, inputlen, code, len, license_key)
#else
static unsigned char * l_string_key(job, input, inputlen, code, len)
#endif
{
... ...
strcpy(lkey, license_key);
... ... /* calculate y, the real checksum */
#ifdef LM_CKOUT
 for (i = 0; i < j; i++)
{
/* convert user checksum from ASCII to hex */
c = lkey[i * 2];
if (isdigit(c))
x = (c - '0') << 4;
else
x = ((c - 'A') + 10) << 4;

c = lkey[(i * 2) + 1];
if (isdigit(c))
x += (c - '0');
else
x += ((c - 'A') + 10);

/* compare user and real checksums */

if (x != y[i])
return 0;
}
... ...
#endif
ret = (atox(job, y, len));
return ret;
}

The rest of job is easy: we just trace in and grab y, the correct license key. In reality we modified the matching result
at runtime, evading the “return 0;” branch, and let the function return y in ASCII. Repeat the process for CSTAT,
and we obtain the following new license file.

SERVER hostname hostid 27000

DAEMON VNI "<vni_dir>\license\bin\bin.i386nt\vni.exe"
FEATURE CMATH VNI 5.5 permanent uncounted 6D5C01FD71C9 HOSTID=ANY
FEATURE CSTAT VNI 5.5 permanent uncounted 369B56AC8B35 HOSTID=ANY

Test it with all validation scripts provided by VNI and all pass with flying colors7. This accomplishment confirmed our
hypothesis that every protection scheme has to compare the right and the wrong in order to differentiate legitimate
and non-legitimate users. The key is to pinpoint that matching precisely in time and space. To do that hackers must
have some deep understandings of the code. In our example we had some luxuries such as source code and LIB files
that commonly people cannot afford. I don’t know if I could make it without them. Therefore solving license code is
absolutely at higher level than patching.

Uncover Keys and Seeds

From a pragmatic point of view we may stop now, for we can use the target freely as licensed copies. However, the
ultimate goal of software cracking is to reverse engineer all relevant algorithms and recreate them as if we were the
original authors. In our case, that is minimally to uncover VNI’s 5 vendor keys and 3 random seeds. Needless to say,
this requires more careful perusing of the FLEXlm SDK source code, so we briefly review how it is organized. The list
below shows the folders that contain the most crucial files.

src source files for lmgr.lib or lmgr9a.dll

app source files for lmgras.lib
server source files for lmgrs.lib
master source files for lmgrd.exe
utils source files for utilities
machind machine independent files (source & header)
i86_n3 final binary files for x86 platform
h header files
certicom library and header files from Certicom Corp.
ulite ultralite version of FLEXlm

7
We tried changing version to 6.5 or 7.0, but scripts did not pass. Evidently the SIGN hash relies on version.
Moreover, files and functions are assigned to different prefixes according on their roles. For instance, files in
directories app, server and master all start with ls_ while in utils it’s the lm prefix.

l_ license, for internal functions

lc_ license client, for client APIs
ls_ license server, for server APIs
lm_ license manager, for utilities and general stuff

We now go back to documentations [5] and [6] for directions on how vendors use FLEXlm SDK. Basically vendors
need to integrate much of the FLEXlm binaries with their own products and create a number of utilities – some for
internal use, some for end user distribution. There are very close ties among these vendor-generated files and here
is the dependency table for them:

File Generated by #include

lmrand1.exe lmrand1.c + lmgr.lib
lmcode.c, lsrvend.c lmrand1.exe + lsvendor.c lm_code.h
lmappfil.c, lmkeyfil.c lmrand1.exe
lmnewgen.exe lmnewgen.c + lmcode.c + lmgr.lib lm_code.h
lm_new.c lmnewgen.exe seeds & pubkey
lmseeds.h lmnewgen.exe
lmprikey.h, lmpubkey.h lmnewgen.exe
lmcrypt.exe lmcrypt.c + lmgr.lib lm_code.h,
lmseeds.h,
lmprikey.h
makekey.exe makekey.c + lmgr.lib lm_code.h,
lmseeds.h,
lmprikey.h
vendor application vendor code + lm_new.obj + lmgr.lib + seeds & pubkey
libsb.lib + libcrvs.lib
vendor daemon lsvendor.obj + lm_new.obj + lmgr.lib + lm_code.h,
lmgras.lib + lmgrs.lib + libsb.lib + lsserver.h,
libcrvs.lib seeds & pubkey

Notice that not all above files are important to us at this time. lmappfil.c and lmkeyfil.c are vendor specific
filters that are additional security measures but normally left unused. Similarly lmprikey.h and lmpubkey.h are for
CRO keys only, which we can safely ignore for our case. Vendors can also customize their daemons by editing
lsrvend.c or lsvendor.c but rarely do people do that.

One thing worth talking about is the Certicom product, namely libsb.lib and libcrvs.lib (I believe sb stands for
“soft bus” and crvs stands for “elliptic curves”). They cover almost the entire encryption field from SHA to DSA, and
the APIs employed by FLEXlm are mostly ECC (Elliptic Curve Cryptography) and RNG (Random Number Generation).
ECC is used for public/private key pairs in CRO, which is turned off in our target; but RNG is used for vendor seeds
transformation, which we cannot neglect.

As stated earlier, vendor selects 3 random seeds (LM_SEED1, LM_SEED2, LM_SEED3), but they are not used directly.
Instead lmnewgen.c!main() -> l_prikey.c!l_genrand() -> libsb.lib!sb_rngFIPS186Session() produces 4
new seeds8 (ENCRYPTION_SEED1, … ENCRYPTION_SEED4), which are output to lmseeds.h, based on them. Seed
3 and 4 are exclusively for public/private key pairs, thus we only need to worry about seed 1 and 2.

Although FLEXlm has its own RNG in l_rand.h and lm_rand3.c, here it uses the algorithm specified in DSS standard
(c.f. [7]). This algorithm is deterministic (without reading time, register value, etc.), so ENCRYTION_SEED are fixed
for given LM_SEED. It is also a one-way function, meaning that it’s practically impossible to solve LM_SEED from
known ENCRYPTION_SEED. Since it is ENCRYPTION_SEED that are built into the vendor releases and LM_SEED
appear nowhere else than l_genrand(), the best we can do is recovering the former. However, for exactly the same
reason, we deem that equally good as recovering LM_SEED.

The inner structure of FLEXlm is essentially C/S model, where integrated vendor application is the client and vendor
daemon (vni.exe here) is the server. Note vendor daemon is different from the FLEXlm license manager lmgrd.exe.
The latter simply redirects client requests to corresponding vendor daemons, which do the real work. Another factor

8
Three more seeds are generated for CRO.
is the license file format. FLEXlm offers rich licensing options and an important one is counted vs. uncounted.
Uncounted license has no restrictions on number of checkouts and does not need vendor daemon – the validation
takes place inside the vendor software that acts as client and server simultaneously. On the other hand, counted
license requires vendor daemon and lmgrd.exe running along with vendor software to serve the client. Counted
license usually floats on a network (floating license must be counted), which is a common scene at large companies:
a central node running the servers and all workstations connect to it for license checkout/checkin. Hence we can
infer that vendor daemon’s task is mainly to count, manage and coordinate the use of limited licenses.

Look at our license, it is “permanent uncounted HOSTID=ANY”, showing that it has no conditions whatsoever. This
is definitely every hacker’s dream (having no daemon process also makes tracing easier). In fact the “SERVER” and
“DAEMON” lines in the license are unneeded, two “FEATURE” lines alone are enough to guarantee cmath.exe
running unlimitedly. We will come to the topic of license types later.

Now we have cmath.exe as both client and server with vendor keys and encryption seeds built in (of course
shadowed and hidden). The good news is that CRO is disabled so there is only one set of keys. If we can dig them
out, then we can make any license file by lmcrypt.exe or makekey.exe the same way as Visual Numerics. Before
we set out to do that, we summarize some key data in key files.

lm_code.h: /* \machind\lm_code.h and \h\lm_code.h must be identical */

VENDOR_KEY1, VENDOR_KEY2, VENDOR_KEY3, VENDOR_KEY4, VENDOR_KEY5
LM_SEED1, LM_SEED2, LM_SEED3
lmcode.c:
#include "lm_code.h"
#include "lmclient.h"
VENDORCODE vendorkeys[] = { /* bad name, should use vendorcodes[] instead */
{ VENDORCODE_7,
ENCRYPTION_SEED1 ^ VENDOR_KEY5, ENCRYPTION_SEED2 ^ VENDOR_KEY5,
VENDOR_KEY1, VENDOR_KEY2, VENDOR_KEY3, VENDOR_KEY4,
FLEXLM_VERSION, FLEXLM_REVISION, FLEXLM_PATCH,
LM_BEHAVIOR_CURRENT, {CRO_KEY1, CRO_KEY2},
LM_STRENGTH, LM_SIGN_LEVEL, 0
},
};
lmseeds.h:
ENCRYPTION_SEED1, ENCRYPTION_SEED2, ENCRYPTION_SEED3, ENCRYPTION_SEED4
lm_new.c:
x = 0x3d73db2e; /* random number key5_uniqx generated in lmnewgen.c */
VENDORCODE.data /* obfuscated by key5() */
VENDORCODE.keys /* obfuscated by l_xorname() */
l_getattr.c:
#define VENDORMAGIC_V7 0x08BC0EF8 /* insignificant, just for xor canceling */
lmclient.h:
#define L_NEW_JOB l_n36_buf
l_privat.h:
#define L_UNIQ_KEY5_FUNC l_n36_buff
#define L_SET_KEY5_FUNC l_x77_buf
#define SEEDS_XOR mem_ptr2_bytes /* where seeds are hidden in job */
#define SEEDS_XOR_NUM 12

Besides, there are three numbers that are hard-coded into the binaries (just like VENDORMAGIC_V7) but seems never
used (at least in the case of cmath.exe). Maybe they are solely for certain versions or license types, we’ll see that
when we visit those functions.

lm_ckout.c!l_sg():
x = 0x6f7330b8; /* v8.x */
lmnewgen.c!VKEY5():
x = 0x6f7330b8; /* v8.x */
l_key.c!l_zinit():
z = crokey_flag ? 0x62586954 : 0x72346B53; /* v9.x */

So we have two processes: checkout in cmath.exe and keygen in lmcrypt.exe or makekey.exe. They both produce
the same correct license codes, but the two processes are not identical. We have analyzed the first process in some
depth in previous paragraphs, let’s list the important call chains in chronological order.
 1. lc_new_job() -> l_n36_buf() -> l_x77_buf()
2. lc_new_job() -> lc_init() -> l_init() -> l_sg() -> l_key() -> l_zinit()
3. lc_set_attr() -> l_set_attr() -> l_set_license_path() -> l_flush_config() ->
l_init_file() -> l_allfeat() -> l_parse_feature_line() -> oldkey() -> l_crypt_private() ->
real_crypt() -> l_string_key()
4. lc_checkout() -> l_checkout() -> lm_start_real() -> l_good_lic_key() -> l_xorname()
5. lc_checkout() -> l_checkout() -> lm_start_real() -> l_good_lic_key() -> l_sg() -> l_n36_buff()
6. lc_checkout() -> l_checkout() -> lm_start_real() -> l_good_lic_key() -> l_crypt_private() ->
real_crypt() -> l_string_key()

An interesting question is, why does l_sg() call l_key() in the 2nd chain but l_n36_buff() in the 5th? Examining
the code excerpts we see the answer is LM_OPTFLAG_CUSTOM_KEY5 and L_UNIQ_KEY5_FUNC. The latter is set by
l_x77_buf() (i.e. L_SET_KEY5_FUNC) in the first chain so in both calls l_n36_buff is not null. Then the reason is
LM_OPT_FLAG_CUSTOM_KEY5: it is switched on after calling lc_init(), that’s why l_key() is invoked in the 2nd
chain. The funny thing is, l_key() is a useless subroutine in modern FLEXlm versions (it’s for earlier versions before
l_n36_buff() is introduced). In addition, it is utterly unnecessary to call l_sg(), which decodes encryption seeds,
in the initialization stage; that should be done only at checkout time.

lm_njob.c:
int lc_new_job(oldjob, l_new_job, vcode, newjobp)
{
... ...
(*L_NEW_JOB)(vendor_name, vcode, 0, 0, 0, &sign_level);
(*L_NEW_JOB)(0, 0, 0, 0, 0, 0);
if (!(ret = lc_init(oldjob, vendor_name, vcode, newjobp)))
{
(*newjobp)->options->flags |= LM_OPTFLAG_CUSTOM_KEY5;
... ...
}
return ret;
}

lm_ckout.c:
void l_sg(LM_HANDLE* job, char* vendor_id, VENDORCODE* key)
{
... ...
unsigned long x = 0x6f7330b8; /* v8.x */
if (( job->options->flags & LM_OPTFLAG_CUSTOM_KEY5) && L_UNIQ_KEY5_FUNC)
{
(*L_UNIQ_KEY5_FUNC)(job, vendor_id, key);
return;
}
l_key(vendor_id, &(key->keys[0]), keys, 4); /* Pre v6.1 style */
... ... /* same xor operations in VKEY5() */
}

lm_init.c:
void (*L_UNIQ_KEY5_FUNC)() = 0;
void L_SET_KEY5_FUNC( void (*f)())
{
if (!L_UNIQ_KEY5_FUNC) L_UNIQ_KEY5_FUNC = f;
}

Parallel to the checkout process, we also have the call chains in the keygen process. We shall use lmcrypt.exe for
analysis because it’s more straightforward than makekey.exe (they two perform the same job).

1. lmcrypt.c!main() -> lc_init() -> l_init() -> l_sg() -> l_key() -> l_zinit()
2. lmcrypt.c!main() -> dofilecrypt() -> dofpcrypt() -> lm_crstr.c!lc_cryptstr() ->
parsefeaturelist() -> l_parse_feature_line() -> oldkey() -> l_crypt_private() ->
real_crypt() -> l_string_key()
3. lmcrypt.c!main() -> dofilecrypt() -> dofpcrypt() -> lm_crstr.c!lc_cryptstr() -> cryptfeaturelist()
-> docryptfeat() -> lc_crypt() = l_crypt_private() -> real_crypt() -> l_string_key()
Observe that cmath.exe calls lc_new_job(), which in turn calls lc_init(), for vendor & job initialization but
lmcrypt.exe calls lc_init() directly because vendor keys, seeds and name are already included in lmcrypt.exe
(put together in vendor structure by macros) so it only needs to initialize job. In both processes there are two calls
to l_string_key() and in both situations the first one returns 21D5B6E8572E, the insignificant number for
oldkey(), and only the second call matters. The two processes calls l_string_key() in slightly different ways,
basically checkout needs to provide user license key for checksum comparison but keygen doesn’t need that input.
However the part for calculating the true hash are the same.

int idx = (*job->vendor) % XOR_SEEDS_ARRAY_SIZ; /* idx = V % 20 = 86 % 20 = 6 */

... ...
memset(y, 0, L_STRKEY_BLOCKSIZE); /* L_STRKEY_BLOCKSIZE = 8, in lmachdep.h */
length = (inputlen) / L_STRKEY_BLOCKSIZE;
XOR_SEEDS_INIT_ARRAY(xor_arr) /* substitution table defined in l_strkey.h */
... ... /* memcpy() from input to newinput, and other stuff */
p = newinput;
for (i = 0; i < length; i++)
{
XOR(p, y, y); /* XOR and L_MOVELONG defined in l_strkey.h */
if (i == 0)
{
if (!user_crypt_filter && !user_crypt_filter_gen
&& (job->flags & LM_FLAG_MAKE_OLD_KEY))
{
q = y; /* SEEDS_XOR = mem_ptr2_bytes defined in l_privat.h */
L_MOVELONG(code->data[0]
^((long)(job->SEEDS_XOR[xor_arr[idx][0]])<<0)
^((long)(job->SEEDS_XOR[xor_arr[idx][1]])<<8)
^((long)(job->SEEDS_XOR[xor_arr[idx][2]])<<16)
^((long)(job->SEEDS_XOR[xor_arr[idx][3]])<<24), q)
L_MOVELONG(code->data[1]
^((long)(job->SEEDS_XOR[xor_arr[idx][0]])<<0)
^((long)(job->SEEDS_XOR[xor_arr[idx][1]])<<8)
^((long)(job->SEEDS_XOR[xor_arr[idx][2]])<<16)
^((long)(job->SEEDS_XOR[xor_arr[idx][3]])<<24), q)
}
... ...
}
if (!(job->flags & LM_FLAG_MAKE_OLD_KEY) && !demo)
our_encrypt2(y);
else
our_encrypt(y); /* our_encrypt() does not involve code or job */
p += L_STRKEY_BLOCKSIZE;
}
if (len == L_SECLEN_SHORT) /* L_SECLEN_SHORT = 0x66D8B337 in l_privat.h */
{
... ...
y[6] = y[7] = 0;
}

Since they share the same code, to compute the same hash string (otherwise what’s the point?) the arguments
passed in must be the same. Tracing result in practice is:

cmath.exe:
job = 00887630, input = 0012e170, inputlen = 0x16, code = 0012ee20, len = 66d8b337, license_key = 6d5c...
[00887630] - 00000066 f... int type;
[00887634] - 0089008e .... char *mem_ptr2;
[00887638] - a06aa84e N.j. unsigned char mem_ptr2_bytes[12]; (12 is decimal)
[0088763C] - 00c3a047 G...
[00887640] - 00660000 ..f.
[00887644] - 00000000 ....
[00887648] - 00000000 ....
[0088764C] - 00000000 ....
 [00887650] - 00000000 ....
[00887654] - 54414d43 CMAT
[00887658] - 00000048 H...

[0012E170] - ab370fd2 ..7. input string to be hashed, relies on FEATURE line info
[0012E174] - 414d4300 .CMA
[0012E178] - 88054854 TH..
[0012E17C] - 6a000113 ...j
[0012E180] - c5876e61 an..
[0012E184] - 000073d0 .s.. total length = 0x16 = 22, ends at 73
[0012E188] - 00000000 ....

[0012EE20] - 00000004 .... int type;

[0012EE24] - 52ed15b8 ...R 52xxxxb8, xxxx is random, different at each run
[0012EE28] - 75cf780f .x.u 75yyyy0f, yyyy is random, different at each run
[0012EE2C] - 7c2adb6a j.*| VENDOR_KEY1
[0012EE30] - b927f5a9 ..'. VENDOR_KEY2
[0012EE34] - 9cf311f8 .... VENDOR_KEY3
[0012EE38] - 0dbf7621 !v.. VENDOR_KEY4
[0012EE3C] - 00020009 .... FLEXlm version (here is 9.2)
[0012EE40] - 39300020 .09
[0012EE44] - 0000302e .0..
[0012EE48] - 00000000 ....

lmcrypt.exe:
job = 008C49E8, input =
0x008C49E8 66 00 00 00
0x008C49F8 00 00 00 00
0x008C4A08 00 00 00 00
0x008C4A18 00 00 00 00
0012D8D4, inputlen = 0x16, code = 004D7B48, len = 66D8B337
00 00 00 00 00 00 00 00 00 00 00 00 f...............
00 00 00 00 00 00 00 00 00 00 00 00 ................
00 00 00 00 00 00 00 00 00 00 00 00 ................
00 00 00 00 00 00 00 00 00 00 00 00 ................

0x0012D8D4 d2 0f 37 ab 00 43 4d 41 54 48 05 88 13 01 00 6a Ò.7«.CMATH.....j
0x0012D8E4 61 6e 87 c5 d0 73 00 00 00 00 00 00 00 00 00 00 an.ÅÐs..........
0x0012D8F4 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x0012D904 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

0x004D7B48 04 00 00 00 b8 39 c1 52 0f 54 e3 75 6a db 2a 7c ....¸9ÁR.TãujÛ*|
0x004D7B58 a9 f5 27 b9 f8 11 f3 9c 21 76 bf 0d 09 00 02 00 ©õ'¹ø.ó.!v¿.....
0x004D7B68 20 00 30 38 2e 30 00 00 c3 80 f4 83 2c c0 1c 77 .08.0..Ã.ô.,À.w
0x004D7B78 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................

Note here we are dealing with non-CRO short keys such as 6D5C01FD71C9, it’s composed of 12 ASCII characters
representing 6 hex bytes, that’s why we see “y[6] = y[7] = 0;” in above code. Except for VENDORCODE and job
structure all other arguments are equal9, and in these two structs not all components are important. It’s easy to
figure out that code->data[], code->keys[] and job->mem_ptr2_bytes[] are those participated in hashing the
input string.

Immediately we copy the four vendor keys revealed in the checkout process to lm_code.h (and confirm that in
memory dump), but we are still short of the encryption seeds and the mysterious VENDOR_KEY5. Now look closer
to the keygen process, it differs from the checkout in code->data[] and job->mem_ptr2_bytes[] (its job struct is
virtually empty). Why it that?

#include "lmprikey.h"
#include "lmclient.h"
#include "lm_code.h"
#include "lmseeds.h"

9
Beware that W32dasm and Visual Studio show memory differently due to little endianness.
... ...
/* set site_code.data = {ENCRYPTION_SEED1 ^ VENDOR_KEY5,
ENCRYPTION_SEED2 ^ VENDOR_KEY5}
site_code.keys = {VENDOR_KEY1, VENDOR_KEY2, VENDOR_KEY3, VENDOR_KEY4} */
LM_CODE(site_code, ENCRYPTION_SEED1, ENCRYPTION_SEED2, VENDOR_KEY1,
VENDOR_KEY2, VENDOR_KEY3, VENDOR_KEY4, VENDOR_KEY5);
... ...
int main(int argc, char **argv)
{
... ...
/* set site_code.data = {ENCRYPTION_SEED1, ENCRYPTION_SEED2} */
LM_CODE_GEN_INIT(&site_code);
if (lc_init((LM_HANDLE *)0, VENDOR_NAME, &site_code, &lm_job))
{
lc_perror(lm_job, "lc_init failed");
exit(-1);
}
... ...
/* call chain dofilecrypt() -> dofpcrypt() -> lc_cryptstr() */
estat |= dofilecrypt(infilename, outfilename, &site_code);
return 0;
}

This is the concise source of lmcrypt.c and it should explain itself. Notice carefully the two macros LM_CODE and
LM_CODE_GEN_INIT defined in lmclient.h: the former initializes site_code.data to be encryption seeds xored with
VENDOR_KEY5 (in accordance with lmcode.c), but the latter soon reverses it back to the original encryption seeds.
Did I tell you FLEXlm has lousy coding style?

Anyway raw encryption seeds and zero job->mem_ptr2_bytes are used in keygen, which is different from checkout.
The natural thing to do is copy code->data from the checkout process to lmseeds.h as encryption seeds 1 & 2 and
recompile lmcrypt.exe. But it did not work, for code->data is clearly random in the form 52xxxxB8 and 75yyyy0F
where xxxx and yyyy change at each run. The same is also true for job->mem_ptr2_bytes. We conclude that the
encryption seeds must be obfuscated and stored in two places, code->data[] and job->mem_ptr2_bytes[] (they
two should be closely coupled somehow), in vendor software because it is shipped to end users and raw seeds need
to be protected. In contrast, lmcrypt.exe is only available to vendors so encryption seeds can appear in plain form.

Obfuscation Method

Next we research how seeds and keys (yes, vendor keys too!) are obfuscated in the target. Our weapon is data flow
analysis, that is, to trace the vendor and job structure10 all the way from initialization to l_string_key() by
debugging cmath.exe.

VENDORCODE after calling l_n36_buf():

[004AEA20] - 00000004 ....
[004AEA24] - aa3342a8 .B3.
[004AEA28] - 8d112f1f ./..
[004AEA2C] - 74df9bc4 ...t obfuscated VENDOR_KEY1
[004AEA30] - b19bfb18 .... obfuscated VENDOR_KEY2
[004AEA34] - 94011f00 .... obfuscated VENDOR_KEY3
[004AEA38] - 054a2ed9 ..J. obfuscated VENDOR_KEY4
[004AEA3C] - 00020009 ....
[004AEA40] - 39300020 .09
[004AEA44] - 0000302e .0..

job after calling lc_init():

[00887630] - 00000066 f...
[00887634] - 00000000 ....
[00887638] - 00000000 ....

10
There is a third structure – config, but it’s not important for our purpose here.
 [0088763C] - 00000000 ....
[00887640] - 00000000 ....

// above are structs after lc_new_job(), then they are passed to

// lc_checkout() -> l_checkout() -> lm_start_real() -> l_good_lic_key()

VENDORCODE after calling l_xorname():

[0012EE20] - 00000004 ....
[0012EE24] - aa3342a8 .B3.
[0012EE28] - 8d112f1f ./..
[0012EE2C] - 7c2adb6a j.*| true VENDOR_KEY1
[0012EE30] - b927f5a9 ..'. true VENDOR_KEY2
[0012EE34] - 9cf311f8 .... true VENDOR_KEY3
[0012EE38] - 0dbf7621 !v.. true VENDOR_KEY4
[0012EE3C] - 00020009 ....
[0012EE40] - 39300020 .09
[0012EE44] - 0000302e .0..
[0012EE48] - 00000000 ....

VENDORCODE after calling l_n36_buff():

[0012EE20] - 00000004 .... int type;
[0012EE24] - 52ed15b8 ...R 52xxxxb8, xxxx is random, different at each run
[0012EE28] - 75cf780f .x.u 75yyyy0f, yyyy is random, different at each run
[0012EE2C] - 7c2adb6a j.*| VENDOR_KEY1
[0012EE30] - b927f5a9 ..'. VENDOR_KEY2
[0012EE34] - 9cf311f8 .... VENDOR_KEY3
[0012EE38] - 0dbf7621 !v.. VENDOR_KEY4
[0012EE3C] - 00020009 .... FLEXlm version (here is 9.2)
[0012EE40] - 39300020 .09
[0012EE44] - 0000302e .0..
[0012EE48] - 00000000 ....

job after calling l_n36_buff():

[00887630] - 00000066 f... int type;
[00887634] - 0089008e .... char *mem_ptr2;
[00887638] - a06aa84e N.j. unsigned char mem_ptr2_bytes[12];
[0088763C] - 00c3a047 G...
[00887640] - 00660000 ..f.
[00887644] - 00000000 ....
[00887648] - 00000000 ....
[0088764C] - 00000000 ....
[00887650] - 00000000 ....
[00887654] - 54414d43 CMAT
[00887658] - 00000048 H...

// above are structs after l_good_lic_key() -> l_sg(), then they are passed to
// l_good_lic_key() -> l_crypt_private() -> real_crypt() -> l_string_key()

VENDORCODE and job are changed at 3 functions: l_n36_buf(), l_xorname() and l_n36_buff() (job initialization
in lc_init() is not very interesting for it only fills zero). The last two are called within l_good_lic_key():

l_good_lic_key(LM_HANDLE* job, CONFIG* conf, VENDORCODE* key)

{
... ...
memcpy(&vc, key, sizeof(vc));
if (!(job->flags & LM_FLAG_CLEAR_VKEYS))
l_xorname(job->vendor, &vc);
l_sg(job, job->vendor, &vc); /* l_sg() would call l_n36_buff() */
 ... ...
code = l_crypt_private(job, conf, sdate, &vc);
... ...
}

For the 4 vendor keys, it is obvious that they are obfuscated in l_n36_buf() at initialization and de-obfuscated in
l_xorname(), which does nothing but some xor operations. We all know the properties of xor: x ^ x = 0, x ^ 0 =
x, x ^ 1 = ~x, so the same xor operations performed twice would cancel each other. This nice property makes xor
perfect for encoding and decoding. If our guess is right, then l_n36_buf() should also call l_xorname() in the same
manner as l_good_lic_key() does. Since l_n36_buf() resides in lm_new.c, which is generated by lmnewgen.exe,
it’s better to examine lmnewgen.c directly.

#include "l_strkey.h"
... ...
int main(int argc, char **argv)
{
... ...
strcpy(outfile, "lm_new.c");
... ...
/* initialize job, could report error and exit here */
if (lc_init(0, vendor_name, &vendorkeys[0], &job))
{
fprintf(stderr, "lc_init failed: %s\n", lc_errstring(job));
exit(1);
}
/* random number generation via sb_rngFIPS186Session() */
l_genrand(job, lmseed1, lmseed2, lmseed3, NEWSEEDSIZ, newseeds);
... ... /* convert newseeds into seed1 – seed4, and so on */
/* write the four ENCRYPTION_SEED to lmseeds.h */
if (!(ofp = fopen("lmseeds.h", "w")))
{
perror("Can't write lmseeds.h file, exiting");
exit(1);
}
fprintf(ofp, "... ...", seed1, seed2, seed3, seed4);
fclose(ofp);

/* now it’s lm_new.c */

if (!(ofp = fopen(outfile, "w")))
{
perror("Can't open output file, exiting");
exit(1);
}
... ...
/* make sure default and weak seeds are excluded */
if (!l_reasonable_seed(seed3) || !l_reasonable_seed(seed4) ||
!l_reasonable_seed(lmseed1) || !l_reasonable_seed(lmseed2) ||
!l_reasonable_seed(lmseed3))
{
... ...
fprintf(stderr, "Existing.\n");
exit(1);
}
... ...
fputs("#include <time.h>\n", ofp);
do_real(); /* write the main content of lm_new.c */
fclose(ofp);
return 0;
}

static void do_real()

{
... ... /* generate random variable and function names */
while (!key5_uniqx)
 {
key5_uniqx = our_rand(256) + (our_rand(256) << 8) +
(our_rand(256) << 16) + (our_rand(256) << 24);
}
... ... /* generate random key5_order[] */
for (i = 0; i < 200; i += 2)
random_garbage();
... ... /* more random garbage */
for (counter=0; counter<keysize; counter++)/* generally keysize=1, only one vendor */
{
key5(&vendorkeys[counter]); /* obfuscate vendorkeys->data[] */
l_xorname(vname, &vendorkeys[counter]);/* obfuscate vendorkeys->keys[] */
/* real VENDORCODE initialization */
do_ulong(vendorkeys[counter].data[0], "data[0]", counter);
do_ulong(vendorkeys[counter].data[1], "data[1]", counter);
do_ulong(vendorkeys[counter].keys[0], "keys[0]", counter);
do_ulong(vendorkeys[counter].keys[1], "keys[1]", counter);
do_ulong(vendorkeys[counter].keys[2], "keys[2]", counter);
do_ulong(vendorkeys[counter].keys[3], "keys[3]", counter);
... ...
}
l_puts_rand(ofp, fpVar, numvars); /* output lines in random order */
fflush(ofp);
uniqcode(); /* output the source of lm_new.c!l_n36_buff() */
fflush(ofp);
... ...
}

static void do_ulong(unsigned long ul, char *varname, int counter)

{
... ...
randvarname(b1, "b1"); bnames[0] = b1; /* generate junk variable names */
randvarname(b2, "b2"); bnames[1] = b2;
randvarname(b3, "b3"); bnames[2] = b3;
randvarname(b4, "b4"); bnames[3] = b4;

for (i = 0; i < 4; i++)

{
shift = i * 8;

CLEARV; /* break up ul into 4 bytes and assign them to 4 junk-named variables */

sprintf(vBuf, "static unsigned int %s = %d;\n",
bnames[i], (ul & (0xff << shift)) >> shift );
fwrite(vBuf, sizeof(char), sizeof(vBuf), fpVar);

CLEARF; /* reassemble the 4 variables back to let v->varname = ul */

sprintf(fBuf, "\tif (%s == %d) v->%s += (%s << %d);\n",
countervar, counter, varname, bnames[i], shift);
fwrite(fBuf, sizeof(char), sizeof(fBuf), fpFunc);
}
}

static void uniqcode()

{
... ...
int idx = *vendor_name % XOR_SEEDS_ARRAY_SIZ; /* idx = V % 20 = 86 % 20 = 6 */
XOR_SEEDS_INIT_ARRAY(xor_arr) /* substitution table defined in l_strkey.h */

fprintf(ofp, "static void %s(job, vendor_id, key) \n\

... ...
unsigned long x = 0x%x; \n\
struct s_tmp {int i; char *cp; unsigned char a[12];} *t, t2; \n\
if (job) t = (struct s_tmp *)job; \n\
else t = &t2; \n\
... ...", key5_fname, key5_uniqx); /* key5_fname and key5_uniqx are all random */
 for(i = 0; i < SEEDS_XOR_NUM; i++) /* decode t->a[], i.e. job->mem_ptr2_bytes[] */
{
unsigned char num;
cpp[i] = cp;
sprintf(cp, "t->cp=(char *)(((long)t->cp) ^ (time(0) ^ ((0x%x << 16) + 0x%x)));\n\
t->a[%d] = (time(0) & 0xff) ^ 0x%x;\n", /* runtime randomness */
our_rand(0xff), our_rand(0xff), i, our_rand(0xff));
cp += strlen(cp) + 1;
}
l_puts_rand1(ofp, SEEDS_XOR_NUM, cpp); /* output lines in random order */

fprintf(ofp, "for (i = 0; i < %d; i++) \n\

{ \n\
if (sig[i%%SIGSIZE] != vendor_id[i%%len]) \n\
sig[i%%SIGSIZE] ^= vendor_id[i%%len]; \n\
} \n\
unsigned long y = ((((long)sig[0] << %d) \n\
| ((long)sig[1] << %d) \n\
| ((long)sig[2] << %d) \n\
| ((long)sig[3] << %d)) \n\
^ ((long)(t->a[%d]) << 0) \n\
^ ((long)(t->a[%d]) << 8) \n\
^ ((long)(t->a[%d]) << 16) \n\
^ ((long)(t->a[%d]) << 24) \n\
^ x \n\
^ key->keys[0] \n\
^ key->keys[1]) & 0xffffffff; \n\
key->data[0] ^= y; \n\
key->data[1] ^= y; \n\
... ...", MAX_DAEMON_NAME, /* MAX_DAEMON_NAME = 10 defined in lmclient.h */
key5_order[0], key5_order[1], key5_order[2], key5_order[3],
xor_arr[idx][0], xor_arr[idx][1], xor_arr[idx][2], xor_arr[idx][3]));
}

static void key5(VENDORCODE *k) /* obfuscate encryption seeds, i.e. k->data[] */

{
... ...
/* same key5_uniqx, key5_order[], sig[] and MAX_DAEMON_NAME as in uniqcode() */
for (i = 0; i < MAX_DAEMON_NAME; i++)
{
if (sig[i%SIGSIZE] != vname[i % len])
sig[i%SIGSIZE] ^= vname[i % len];
}
y = ((((long)sig[0] << key5_order[0])
| ((long)sig[1] << key5_order[1])
| ((long)sig[2] << key5_order[2])
| ((long)sig[3] << key5_order[3]))
^ key5_uniqx
^ k->keys[0]
^ k->keys[1]) & 0xffffffff;
k->data[0] ^= y; /* k->data[0] = ENCRYPTION_SEED1 ^ y */
k->data[1] ^= y; /* k->data[1] = ENCRYPTION_SEED2 ^ y */
}

It’s a little long, but not incomprehensible once we unravel its structure. The main procedure spends much of its
time working on lmseeds.h and lm_new.c is largely procreated in do_real(). With the help of several subroutines
do_real() per se focused on making l_n36_buf() while left l_n36_buff() to uniqcode(). And what about
l_xorname()? Although not appearing in l_n36_buf() as is, it’s right there in do_real(). Therefore it is true that
4 vendor keys are ciphered and deciphered by the same xor operations at different locations, namely do_real() /
l_n36_buf() and l_good_lic_key() respectively.

Note lm_new.c itself is heavily obfuscated: there are a lot of garbage code that are wholly useless, the identifiers are
random and meaningless, the real and trash code look alike and mix together, the line order of real code are
perturbed… Apparently it’s designed in this way to fool the readers and it works, it’s almost impossible to understand
the C source, let alone the disassemblies. That’s why we need to concentrate on its parent lmnewgen.c, which can
gives us more hints. For example, l_xorname() is called in do_real(), not l_n36_buf(). It means key obfuscation
is done in-house at vendor site and encoded keys are linked into the target delivered to end users; then they are
deciphered at user runtime in l_good_lic_key(). Such scheme is intended to minimize the exposure of real keys.

But we are more concerned about encryption seeds than vendor keys. Recall it is job->mem_ptr2_bytes[] and
code->data[] that are different in the checkout and keygen processes. It is likely that FLEXlm also uses xor for seed
encoding, but it’s got to be more than that because of the randomness we saw at user runtime. We’ll analyze it step
by step.

Just one line above do_real()->l_xorname() there is call to key5(), which obfuscates the raw encryption seeds at
vendor side (nothing is done to job structure). The vendor key experience tells us that they must be de-obfuscated
somewhere at user side. This time it’s not l_good_lic_key(), but l_n36_buff(). Its source in lm_new.c is
readable, but still we prefer to study its generator uniqcode(). Before long we find out that l_n36_buff() xor-es
the same variables as in key5(), plus the extra t->a[] stuff. What’s that? Well t->a[] is just another name for
job->mem_ptr2_bytes[]. How to account for those additional xor? Let’s rewind several pages and… yes, at the
L_MOVELONG() macros in l_string_key(), right before the hashing starts.

Now we see the seed encryption/decryption is done in two steps, l_n36_buf()/l_n36_buff() and l_n36_buff()/
l_string_key(). The xor code in the corresponding functions mirrors each other, ensuring all noises are canceled
out. The runtime randomness is introduced by time(0) factor in l_n36_buff(), it affects both code->data[] and
job->mem_ptr2_bytes[] we saw in l_string_key(). Indeed FLEXlm hides the encryption seeds in a much more
obscure way than vendor keys. And we now know the meaning of the two weird-named functions (actually the only
two) in lm_new.c: l_n36_buf() – initialize VENDORCODE with ciphered seeds and keys; l_n36_buff() – undo
l_n36_buf() and impose the second step encryption.

We should emphasize that obfuscation is only for the checkout process. In keygen process lmcrypt.exe does not
invoke l_n36_buf(), l_n36_buff(), or l_good_lic_key(), and L_MOVELONG has no effects on raw seeds because
job->mem_ptr2_bytes[] is always 0. During checkout encryption/decryption there are a number of constants,
variables and arrays involved in the xor operations. They are often placed at two locations, one for encoding and the
other for decoding. A summary about them would certainly be convenient.

Related objects Encode Decode

l_xorname(),
do_real(),
VENDOR_KEY VENDORNAME,
l_n36_buf()
l_good_lic_key()
VENDORMAGIC_V7
x = key5_uniqx,
ENCRYPTION_SEED, key5_order[], do_real(), key5(), uniqcode(),
step 1 sig[], l_n36_buf() l_n36_buff()
MAX_DAEMON_NAME
ENCRYPTION_SEED, idx, uniqcode(), L_MOVELONG(),
step 2 xor_arr l_n36_buff() l_string_key()

It is time for us to retrieve the real encryption seeds from cmath.exe. The true seeds are recovered as the first
argument for L_MOVELONG(). With idx=6 and the prescribed substitution table xor_arr[], the xor operand consists
of 4 bytes indexed at 7, 3, 5, 11 from job->mem_ptr2_bytes[]. After assembling it becomes 00A0A000. Therefore
ENCRYPTION_SEED1 = code->data[0] ^ 00A0A000 = 52ED15B8 ^ 00A0A000 = 524DB5B8
ENCRYPTION_SEED2 = code->data[1] ^ 00A0A000 = 75CF780F ^ 00A0A000 = 756FD80F

Try them in lmseeds.h and… bingo! lmcrypt.exe yields the lovely 6D5C01FD71C9. Change the version to 5.0 and
we get 3F23BE3056E4, correct signature again! This assures us that doubtlessly we have acquired the authentic
encryption seeds and vendor keys of Visual Numerics. Finally we can generate any VNI license keys as we wish, for
other versions, other features, other products… And hopefully it should work without a glitch provided FLEXlm is not
materially revised.

F:\flexlm>type license.dat
FEATURE CMATH VNI 5.0 permanent uncounted 0 HOSTID=ANY
FEATURE CSTAT VNI 5.0 permanent uncounted 0 HOSTID=ANY
FEATURE CMATH VNI 5.5 permanent uncounted 0 HOSTID=ANY
FEATURE CSTAT VNI 5.5 permanent uncounted 0 HOSTID=ANY
FEATURE CMATH VNI 7.1 permanent uncounted 0 HOSTID=ANY
FEATURE CSTAT VNI 8.3 permanent uncounted 0 HOSTID=ANY
FEATURE Hello VNI 2.9 permanent uncounted 0 HOSTID=ANY
FEATURE cRaCk VNI 4.0 permanent uncounted 0 HOSTID=ANY
FEATURE CMATH VNI 5.5 permanent uncounted HOSTID=ANY SIGN=0

E:\flexlm>utils\lmcrypt -i license.dat
FEATURE CMATH VNI 5.0 permanent uncounted 3F23BE3056E4 HOSTID=ANY
FEATURE CSTAT VNI 5.0 permanent uncounted 2C60CD4570B0 HOSTID=ANY
FEATURE CMATH VNI 5.5 permanent uncounted 6D5C01FD71C9 HOSTID=ANY
FEATURE CSTAT VNI 5.5 permanent uncounted 369B56AC8B35 HOSTID=ANY
FEATURE CMATH VNI 7.1 permanent uncounted F218B30D7129 HOSTID=ANY
FEATURE CSTAT VNI 8.3 permanent uncounted CC5FA3C48B85 HOSTID=ANY
FEATURE Hello VNI 2.9 permanent uncounted 505E4E243D1B HOSTID=ANY
FEATURE cRaCk VNI 4.0 permanent uncounted 93D0E20E2D20 HOSTID=ANY
FEATURE CMATH VNI 5.5 permanent uncounted HOSTID=ANY \
SIGN=B5E1542279DC

Further Discussions

There are plenty of things worth discussing even though we have fully reverse engineered the FLEXlm protection
system. The most prominent one is, where is VENDOR_KEY5? All essays on [2] say the 5th key is used to xor
encryption seeds and offer various techniques to uncover it (I tried some of them but none worked). The source
code also strongly suggest that with all the names like L_UNIQ_KEY5_FUNC, key5(), key5_uniqx, key5_order[],
VKEY5(), etc. Yet we were able to generate correct license file without even knowing VENDOR_KEY5, isn’t that
strange?

The only logical explanation is that VENDOR_KEY5 is abandoned in the new version. In early versions (pre-8.0?)
VENDOR_KEY5 was truly vital in seed obfuscation, but it is dropped in newer versions, perhaps as a countermeasure
to hackers. In key5() encryption seeds are xor-ed (partly) with code->keys[0] and code->keys[1], the first two
vendor keys rather than the fifth. Consequently specific tricks in [2] are largely out of date. However, the old source
code are preserved in FLEXlm, bewildering everyone who tries to read it. You may argue this is done intentionally to
mislead hackers, but I think it is more likely attributed to bad project management at Macrovision.

The old encoding/decoding algorithms include several functions that are untouched in the checkout process:
VKEY5(), l_svk(), l_key() and l_zinit(). Remember the constants x and z that are never used? They’re updated
in every version upgrade (see history comment) but irrelevant in our practice. The ironic thing is that despite the
alias L_UNIQ_KEY5_FUNC, l_n36_buff() has nothing to do with VENDOR_KEY5 at all. In FLEXlm dialect KEY5 has
become a (false) symbol for obfuscation.

Back to the latest code, l_string_key() deserves a second look. Before L_MOVELONG there is a test on (job->flags
& LM_FLAG_MAKE_OLD_KEY). The macro is defined as 0x00100000 in l_privat.h, and in reality the tracing result
was job->flags = 0x00104840 for cmath.exe and job->flags = 0x00944000 for lmcrypt.exe, so they both enter
the if{} block. As the flag name indicates, we are making the old type license key – which is no surprise, after all
our keys are short and non-CRO in the first place. But what’s the new type?

The difference between the new and old style license lies on the “SIGN=” literal. The old key is a standalone hash
string while the new one has a “SIGN=” prefix. This seemingly minor detail actually matters a lot. As demonstrated
above, lmcrypt.exe determines the license type by the existence of “SIGN=” and the new checksum for CMATH 5.5
is SIGN=B5E1542279DC. Put it in a license file and cmath.exe pops up the error message. However, such format
incompatibility is restricted to the keygen process, the checkout process only cares about the signature itself. In
other words, cmath.exe would not complain about a “SIGN=6D5C01FD71C9” license line.

In the l_string_key() parameters, code->data[] has the peculiar pattern of {52xxxxB8, 75yyyy0F} where xxxx
and yyyy are random at each run. Why only the middle bytes are random but not all? Well it’s because the xor
operand from job->mem_ptr2_bytes[] is 00zzww00 where zz = ww is random. But then why does that 32-bit word
have zero head and tail with two middle bytes equal? Believe it or not, it’s purely coincidence.

First, in l_n36_buff() not all members of t->a[] (i.e. job->mem_ptr2_bytes[]) are assigned to random values.
This seems impossible because i loops through the whole array in the generator uniqcode(). The problem is the
l_puts_rand1() afterward: it’s supposed to shuffle and output the lines in random order, but the implementation
may end up with writing only certain lines depending on internal seeds. See lm_rand3.c – FLEXlm’s proprietary RA
(Random Algorithm) file – for source code. The seeds may be modified by many functions and at the time to output
t->a[] it happens to emit the following sequence: 0, 10, 3, 4, 6, 5, 5, 4, 5, 1, 2, 2. Recall the 4 byte indexes for xor
operand are 7, 3, 5, 11; no wonder t[3] and t[5] are random while t[7] = t[11] = 0.

Second, we know the t->a[] randomness comes from time(0). But standard time function’s precision is only down
to second, which is rather crude for modern processors. It takes less than 0.01s to execute l_n36_buff(), thus
t->a[3] and t->a[5] are assigned to the same random value. All these explain the 00zzww00 (zz = ww) pattern
of the xor operand.

We want to point out that the l_puts_rand1() implementation has to be classified as a bug. In fact the quality of
whole lm_rand3.c is quite low, it gives too many predictable results to be called random. The preceding t->a[]
sequence is a good example, it’s partial (not covering the entire array), fixed (VNI and I have the same
l_n36_buff()), and weak (not realizing the serious limitations of time(0)). Had it been more random, the above
exotic xor patterns would disappear. Macrovision would be much better off if they just use Certicom’s library in lieu
of their ugly code.

Macrovision did, however, do a terrific job in key/seed obfuscation. They are initialized to shadow values and not
recovered until the last moment. This makes a careless cracker to fall into traps easily, if he/she sets breakpoint
arbitrarily then most likely he/she will intercept wrong values. For instance, an article in [2] says FLEXlm validates
encryption seeds not be default ones, which is the following code section (0044414A – 00444289) from l_init().

if (!(job->options->flags & LM_OPTFLAG_CUSTOM_KEY5) && !L_STREQ(job->vendor, "demo")

&& (l_getattr(job, LMADMIN_API) != LMADMIN_API_VAL))
{
memcpy(&vc, &job->code, sizeof(vc));
l_sg(job, job->vendor, &vc); /* calls l_key(), so it does not recover true seeds */
if ((vc.data[0] == 0x87654321) || (vc.data[1] == 0x12345678))
{
LM_SET_ERRNO(job, LM_DEFAULT_SEEDS, 318, 0);
}
}
memset(&vc, 0, sizeof(vc));

I interrupted it and got vc.data[0] = 2AD430F8, vc.data[1] = 0DF65D4F, which are not real seeds. Surely this
is another bug because such validation should only appear on vendor side such as lmnewgen.c but never on user
side, moreover above code in the target does not serve its purpose. Of course if we look at the other side of the coin,
we may say such code successfully confuses hackers. I personally made many mistakes on this issue.

Eventually we achieve in defeating FLEXlm protection at three levels: 1. patch; 2. obtain license checksum; 3.
obtain vendor keys and seeds. The difficulty rises at each level. Patching is the easiest yet the sharpest weapon of
crackers. It may not be elegant but it’s very effective, whose principles apply universally to all software protection
systems. Level 2 and 3 are more ambitious, and theoretically it is possible to devise a system that is secure at these
two levels.

As mentioned previously, FLEXlm carries quite a few advanced equipments, especially CRO. Among others we think
trash code should be a top candidate. It has been proven to be very practical. Like the garbage-mixed core
encryption/decryption algorithms, the target assembly alone shed little light on what’s really going on, and we have
to resort to the source code of lmnewgen.c to grasp it. If more junk were added beyond lm_new.c the difficulty
would increase exponentially. Another practical choice is framework change at every new version. In fact
Macrovision did just that, but it is very troublesome to keep new and old versions compatible (you do need to make
your customers happy, right?). Vendors can create their own filters too, by editing utils\pc.mak, it adds one more
layer of xors.

All these measures are practical, but they are just more obfuscations, which probably has reached the end. New
breakthrough requires new theory and that’s where CRO kicks in. CRO stands for Counterfeit Resistant Option, it is
an ECC public-key encryption system introduced in v7.2. We assume that readers have basic idea about how public-
key algorithms work because that’s not our topic here. We want to concentrate on its difference from the traditional
hashing method.

The central discrepancy is symmetric vs. asymmetric and one-way vs. bidirectional. Asymmetric encryption has two
keys, public and private. The CRO public key goes to vendor software (checkout process) and private key is kept in
lmcrypt.exe (keygen process) at vendor site. So even if the public key (may be obfuscated) is compromised, the
private key is still safe provided vendor doesn’t leak it out. In theory it is practically impossible to solve the private
key from the public key. In contrast symmetric encryption has only one set of keys that are present in both checkout
and keygen processes. Once we discover it, it’s all done; that’s how we accomplish level 3. But for CRO enabled keys,
level 3 is now officially daydream.

Level 2 is not any better. We said earlier that all software protection must compare the right and the wrong to
distinguish legitimate and non-legitimate users. This is a true statement, but must “the right and the wrong” be the
right and wrong checksums? For hash calculation, the answer is yes because it is a one-way function that cannot be
performed in the other way around. The implication is that true hash code has to be calculated on the fly and a
memory peek brings us to level 2. Not that easy for CRO, public key encryption has the ability to go either direction.
Vendor keygen can take the feature line ASCII as plaintext and encrypts it with private key to produce license
signature, which is only given to paid customers. Upon checkout vendor software reads the signature from user
license file as cipher-text, decrypts it with public key, and compare the decrypted string to the feature line ASCII. In
this way (digital signature), the real signature is never calculated at user site, what gets compared is just vanilla text
visible to all (Note here feature line ASCII is the plaintext, but it’s publicly available; license signature is the cipher,
but it’s secret we want to protect; sort of mind-boggling). So level 2 is also “mission impossible” now.

There is another way to prevent the direct comparison of sensitive data. It’s widely used in password verification. At
setup the password is hashed (Windows) or used as key to encrypt a known string (Unix/Linux, DES/Blowfish/…),
the result cipher text is then stored. When user types in a password for login, the input goes through the same
process and matched to the saved cipher. The plain password never gets compared. Does this unidirectional plan
contradict to what we said? No. This plan will lose all the flexibility of license management unless ciphers are not
saved in vendor software. If they are, then vendors must know the passwords prior to product shipment, which
means it can only be static serial number. If not, then the only imaginable place to store them is vendor site. Vendors
can maintain a database of end user profiles including password ciphers and vendor software can ask users to login
to vendor website before real work begins. This may be feasible, Microsoft already forces every user to activate
Windows XP via Internet, but it will also inflict angry protest.

Fortunately we still have patching, the ultimate killer. As long as we have the vendor software we can physically
change it. We can also tinker CRO to replace vendor public key with our public key, but why bother when we can
patch much more conveniently as described in the front. This is the fundamental weakness of software sales and
why pirating can only be tamed but not eradicated. In the real world economic and legal measures are often more
useful in fighting pirating than technology. So much for FLEXlm mechanism, below we’re going to relax a little bit
and offer our 2 cents on some issues.

We worked very hard to reverse engineer FLEXlm, there is no regret because our effort pays off. As a notable brand
on market FLEXlm is very popular among numerous software vendors (e.g. ANSYS, Fluent, Cadence, Synopsys,
UG, …) and becomes industry standard. Its customer base is a big incentive for people to study it. Although I criticize
its source code harshly, to be fair, in license management it heads and shoulders the majority of other software,
which often only has the simplest serial code protection. Having the FLEXlm experience, hacking the rest should be
a piece of cake.

I have expressed my hatred toward FLEXlm coding style repeatedly and it seems I’m not alone. In a document
named “Macrovision Coding Conventions” detail instructions are given on how to write C language programs. It
sounds more like a new developer complaining about his/her frustration on the Greek-like code. By the way I think
FLEXlm should be rewritten in C++ (maybe a little Java too), even just some function wrapper also helps. License
management is a task very suitable for OOP.

A few comments on debuggers (for static analyzers IDA Pro is by far the best). VC7 has an integrated debugger that
is the No.1 choice if source code is available (developers love it). Most debuggers are for binary executables.
W32dasm is a small and efficient tool, requires little system resource and can run as a normal Windows application.
Thanks to its small size, it also lacks some advanced functionalities and handles large target poorly. The worst thing
is that its author has stopped supporting it. If it were open source software, then someone (I’d like to) could pick it
up and continue to improve this neat tool.

Ollydbg is similar to W32dasm and more powerful. It has an important feature missing in W32dasm – set breakpoint
on memory access/change. But I don’t like its UI layout. The world heavyweight champion of debuggers is of course
Numega SoftICE. This famous debugger can debug anything, even the system kernel (W32dasm and Ollydbg can
only take user applications). It’s initially intended for driver development – the implementation itself is a system
driver – but now it’s used for all kinds of operations. Its largest drawback is instability. Running at ring 0 it easily
interferes with the OS and frequently causes system crash/freeze. At last I have to uninstall it. Microsoft also has
two independent debuggers, windbg and kd. This first one is GUI application and the second is command line kernel
debugger. I have no experience on them.

There are topics we have not covered, some are not important, some need another paper. In the end we sum up
some lessons we have learned in reverse engineering:

Good tools and skillful use of them are vital to success;

Do not jump into disassembly tracing too hastily, gather as much information as possible first;
Before dynamic tracing, do a thorough static analysis;
When reading source code, compare it with tracing result to chart the control flow;
Data flow analysis is very useful;
Reverse engineering is laborious, tedious and rewarding work, be patient.

References

[1] C. Cifuentes, Reverse Compilation Techniques, PhD thesis, University of Queensland, 1994.
[2] CrackZ, FLEXlm – “Dubious License Management”, http://www.woodmann.com/crackz/Flexlm.htm, 2003.
[3] I. Guilfanov, Fast Library Identification and Recognition Technology, http://www.datarescue.com/idabase/
flirt.htm, 1997.
[4] Intel, IA-32 Intel Architecture Software Developer’s Manual, Volume 2: Instruction Set Reference, 2001.
[5] Macrovision, FLEXlm Programmers Guide 8.1, February 2002.
[6] Macrovision, FLEXlm Reference Manual 8.1, February 2002.
[7] NIST, FIPS Publication 186-2: Digital Signature Standard, 2000.
[8] Visual Numerics, IMSL C Numerical Library 5.5 User’s Guide, 2003.