Advanced Study On FLEXlm System

truth, June 23, 2004

This is a follow-up article to my previous paper “On Software Reverse Engineering”. Readers are encouraged to
consult [6] before continuing if they have not done so.

Problem Description

After successfully reverse engineered the protection of IMSL CNL 5.5, our attention moves to other FLEXlm guarded
software. There are many – Intel C++/Fortran Compiler 8.0, Intel MKL 5.2, Fluent 6.0, Maple 7.0, Gauss 4.01 –
these all have “permanent uncounted HOSTID=ANY” type licenses. A lot of software also use their own schemes
such as serial number and password, and cracking against them ranges from keygen to patching. Examples include
Mathematica, SAS, SPSS, SolidEdge, Matlab, Femlab, MathCAD, Lindo, Origin, Labview, Tecplot. What interest us
here, however, are the ones that employ FLEXlm but don’t have “permanent uncounted HOSTID=ANY” license.
Among software on my hand there are three of them: PGI C++/Fortran Compiler 5.1, Altair HyperWorks 5.0, and
ANSYS 8.0. Licenses of the first two are permanent but counted without HOSTID=ANY, the last one is not even
permanent.

At this moment we are not actively using PGI or HyperWorks, but we do use ANSYS regularly. Since it’s counted
license, FLEXlm server lmgrd.exe and vendor daemon ansyslmd.exe must be started prior to each run in order to
serve the application request for license checkout. The C/S communication is on IP/ports/sockets and it causes my
firewall ZoneAlarm Pro 3.7 to pop up window asking whether to allow such connection. This is really an annoyance
and I decide to get rid of it (I could configure ZoneAlarm to let lmgrd/ansyslmd pass every time but I don’t want to
do that). So the target in this article is ANSYS 8.0.

ANSYS has numerous features – more than any other software I’m aware of – total of 223 in original license. Here
we can only print a few lines in the front:

SERVER changsha 000795f52adc 1055

DAEMON ansyslmd ansyslmd.exe
INCREMENT ane3fl ansyslmd 9999.9999 12-nov-2006 10 1CEF0B37317C \
VENDOR_STRING=customer:00265231 ISSUED=10-Oct-2003 \
START=10-Oct-2003
INCREMENT ansys ansyslmd 9999.9999 12-nov-2006 10 B1F027206FD0 \
VENDOR_STRING=customer:00265231 ISSUED=10-Oct-2003 \
START=10-Oct-2003
INCREMENT ane3 ansyslmd 9999.9999 12-nov-2006 10 CA5505E0FE07 \
VENDOR_STRING=customer:00265231 ISSUED=10-Oct-2003 \
START=10-Oct-2003
INCREMENT anfl ansyslmd 9999.9999 12-nov-2006 10 1815F7D1236F \
VENDOR_STRING=customer:00265231 ISSUED=10-Oct-2003 \
START=10-Oct-2003

As usual, we also list our tools and so on.

Target: ANSYS 8.0

Protection: Macrovision FLEXlm v8.3b
Tools: Microsoft Visual Studio 7.1 (CL, NMAKE, DUMPBIN, LIB, …)
Microsoft WinDbg 6.1
RedHat Cygwin 1.3.5
IDM UltraEdit 9.0
Datarescue IDA Pro 4.3 (FLAIR, …)
URSoft W32Dasm 8.9
Resources: Macrovision FLEXlm 9.2 SDK source code

Preliminary Attempts

With the experience of [6], I thought it would be an easy job at first. It turns out to be not so, there are very big
hurdles waiting for us. The first one is how to debug ansys.exe. Unlike IMSL CNL or other software that has a

1
Gauss 4.0 has a CRO license – the only one I have ever seen so far.
moderate executable, ansys.exe is huge – 50.1 MB in size. It’s impossible to debug it within W32Dasm or OllyDbg
(W32Dasm has trouble just loading it). Visual Studio’s integrated debugger seems can only work with code built by
VS (I tried “Debug Processes…” but it failed). SoftICE should work, but its mouse problem makes me reluctant to use
unless there is no alternatives. Fortunately we found one, and that is WinDbg. It can perform both kernel and user
mode debugging; it has friendly GUI and standard shortcut keys; most importantly, it can handle large binaries.

The vital trick for using Windbg is how to navigate to your destination. It does not load the whole image into memory
at once. When ansys.exe is loaded, we are in ntdll!DbgBreakPoint() module at first: 77F813B1 CC int 3. To go
to the user code, we press Ctrl+G and type in 00401000. Nothing happens. The trick is to repeat the process again,
and like magic you are now at address 00401000, free to set breakpoints, etc. I have no idea why it’s like this, but
it works.

We then started tracing the program and got some results, but they were not so helpful and we worried about
spending too much time wandering in a gigantic code maze while not getting to the core (namely vendor keys and
encryption seeds). Static analysis is also a problem, it takes forever for IDA to fully analyze ansys.exe and the
database is astonishingly 274 MB large. This renders IDA virtually unusable, and without such an important tool it’s
even harder to hack the target.

Attacking the Daemon

The idea is not to attack the target directly but to attack its associated vendor daemon – ansyslmd.exe. It’s much
smaller and we know it contains the same keys and seeds as the target does. We have identified the target’s FLEXlm
version to be 8.3b, not far from our 9.2 SDK, so if we can compromise keys/seeds from the daemon then we can
build our own lmcrypt.exe to generate the “permanent uncounted HOSTID=ANY” license that we desire.

A simple run of ansyslmd yields the message “(ansyslmd) Vendor daemons must be run by lmgrd”. Documentation
[4] and [5] also mention that lmgrd is the dispatcher for different vendor daemons. Thus we headed for lmgrd.exe
as the first step. It has better command line help on options and usage.

E:\Ansys\v80\ANSYS\licensing\intel>lmgrd
... ...
16:06:54 (lmgrd) Running lmgrd in dedicated windows ...
16:06:54 (lmgrd) Use -z to run in foreground in this window

E:\Ansys\v80\ANSYS\licensing\intel>lmgrd -z -c c:\flexlm\license.dat
... ...
16:13:03 (lmgrd) License file(s): c:\flexlm\license.dat
16:13:03 (lmgrd) lmgrd tcp-port 1055
16:13:03 (lmgrd) Starting vendor daemons ...
16:13:03 (lmgrd) Started ansyslmd (pid 1024)
16:13:03 (ansyslmd) FLEXlm version 8.3b
16:13:03 (ansyslmd) Invalid license key (inconsistent authentication code)
16:13:03 (ansyslmd) ==>INCREMENT ane3fl ansyslmd 9999.9999 permanent uncount
ed 1CEF [...]
16:13:03 (ansyslmd) Invalid license key (inconsistent authentication code)
16:13:03 (ansyslmd) ==>INCREMENT ansys ansyslmd 9999.9999 permanent uncounte
d B1F02 [...]
16:13:03 (ansyslmd) Invalid license key (inconsistent authentication code)
16:13:03 (ansyslmd) ==>INCREMENT ane3 ansyslmd 9999.9999 permanent uncounted
CA5505 [...]
16:13:03 (ansyslmd) Invalid license key (inconsistent authentication code)
16:13:03 (ansyslmd) ==>INCREMENT anfl ansyslmd 9999.9999 permanent uncounted
1815F7 [...]
16:13:03 (ansyslmd) Server started on changsha
16:13:04 (lmgrd) ansyslmd using TCP-port 3056

License file c:\flexlm\license.dat is the above license clip with “permanent uncounted HOSTID=ANY” filled in.
Of course error is reported because the signature code remains the same. In fact, things are even worse. When we
start ANSYS (i.e. ansys.exe rather than lmgrd.exe) with this new trial license, the error message says “Local
checkout filter reject request (-73, 125)”. That suggests ANSYS takes advantage of the filter function supplied by
FLEXlm, a topic we left un-discussed in [6]. It will cause a great deal of trouble to us later. Following the ANSYS
prompt, we set the environmental variable ANS_FLEXLM_DEBUG=2 and obtained more details like
changsha: The 'ane3fl' license is a non-demo, uncounted
license. This particular client machine is not permitted to
run with non-demo, uncounted licenses.
(This license line's encryption key: 1CEF0B37317C)

(This could cause a local checkout filter rejection message

for the 'ane3fl' feature.)

Anyway we load lmgrd.exe into W32Dasm and IDA Pro to start tracing. Because we have source code of lmgrd.exe
in FLEXlm SDK, it is not difficult to identify function calls.

sub_00465F80 lmgrd.exe!start() ; maybe it is in MSVCRT module

00466061: call 00408900
sub_00408900 service.c!main()
00408985: call 0041B050
004089B3: call 004022EF
sub_004022EF ls_lmgrd.c!main_service_thread()
00402315: call 00415C9F
00402336: call 00402730
00402363: call 0041A470
00402421: call 0040A26B
0040264A - 0040267A
00402660: call 0040BDD0
00402672: call 00403EC0
sub_00403EC0 ls_m_main.c!ls_m_main()
00403FF5 - 0040473F
00404015: call 00404998
0040414F: call 00421AAD
0040422F: call 00408870
00404234: call 004081A5
004045A4: call 00404745
00404654: call 00414FD0
00404692: call 00404A10
00404732: call 0040488B
sub_00404745 ls_m_main.c!vendor_start()
004047C8: call 00407330
sub_00407330 ls_startup.c!ls_startup()
004077BB: call 00407B8F
sub_00407B8F ls_startup.c!NT_Startup()
00407D1E: call [00491540]
00407D92: call [0049154C]
; call service.c!main()
; call l_timers.c!l_real_getenv()
; call ls_lmgrd.c!main_service_thread()
; call wsock32.c!l__WSAStartup()
; call ls_m_init.c!ls_m_init()
; call lm_hosttype.c!lc_hosttype()
; call LOG()
; while (1) dead loop
; call ls_quorum.c!ls_quorum()
; call ls_m_main.c!ls_m_main(), **daemons = ansyslmd, select_mask = 0
; after the call EIP is in ntdll!NtWaitForSingleObject()
; while (1) infinite loop
; call ls_m_main.c!check_chld_died()
; call l_date.c!l_get_date()
; call ls_timestamp.c!ls_timestamp()
; call ls_statfile.c!ls_statfile()
; call ls_m_main.c!vendor_start(), *daemons = ansyslmd
; call l_select.c!l_select()
; call ls_m_process.c!ls_m_process()
; call ls_m_main.c!send_daemons()
; call ls_startup.c!ls_startup()
; call ls_startup.c!NT_Startup()
; call kernel32!GetStartupInfoA()
; call kernel32!CreateProcessA()

Here are some related source code.

pcsock.h:
#define WSAStartup(a,b) l__WSAStartup(a,b)
lmclient.h:
#define lm_hosttype(r) lc_hosttype(lm_job, r)
#define lm_daemon(d, o, p) lc_daemon(lm_job, d, o, p)
lsserver.h:
#define LOG(x) {ls_log_prefix(LL_LOGTO_ASCII, 0); (void) ls_log_asc_printf x;}

ls_lmgrd.c:
void main_service_thread(int argc, char *argv[])
{
... ... /* lmgrd.exe is run as app, not service, run_lmgrd_as_service = 0 */
while (1)
{
 ls_quorum(main_master_list, "", 0); /* Make sure quorum is up */
ls_m_main(&master_daemons, &select_mask); /* real stuff */
}
}

ls_m_main.c:
void ls_m_main(DAEMON **daemons, SELECT_MASK *select_mask)
{
... ...
if (havequorum && havemaster && !master_ready)
if (q->list[q->master].state & C_MASTER_READY)
{
master_ready = 1;
vendor_start(*daemons);
}
... ...
}

In the last call to kernel32!CreateProcessA(), the pszCmdline argument is

ansyslmd.exe –T changsha 8.3 –1 –c “c:\flexlm\license.dat” –lmgrd_start 40cd5776
If we try that command in the shell, ansyslmd will start – with error message, of course; actually it will stop quickly
due to heartbeat failure. This means although ansyslmd says it must be invoked by lmgrd, in fact we can run it
directly with proper command line options. We further find out that a simpler syntax will just do the job:
ansyslmd.exe –1 –c c:\flexlm\license.dat

So our effort on lmgrd.exe is a waster of time? Well… yes, but don’t get too upset, it happens in reverse engineering.
Let’s move on to ansyslmd.exe.

sub_0045AED6 ansyslmd.exe!start() ; maybe it is in MSVCRT module

0045AFB4: call 0045B3E0
sub_0045B3E0 ls_app_main.c!main()
0045B42E: call 0045B450
sub_0045B450 ls_daemon.c!ls_daemon()
0045B46E: call 0045D100
sub_0045D100 ls_app_init.c!ls_app_init()
0045D20D: call 004869E0
0045D293: call 0040ED51
0045D3A8: call 00424F50
0045D50B: call 00476670
0045E354: call 004804B0
0045E3A6: call 0045E6D4
0045E3B7: call 00465A20
sub_00465A20 ls_init_feat.c!ls_init_feat()
00465A9E: call 00465AF3
00465AB4: call 00465AF3
00465AEA: call 00472240
sub_00465AF3 ls_init_feat.c!ls_create_feats()
00465B33: call 00465FC3
sub_00465FC3 ls_init_feat.c!ls_feat_validate()
00466002: call 00434671
00466187: call 004674EE
sub_004674EE ls_init_feat.c!good_config()
004675E2: call 0041889D
sub_0041889D lm_ckout.c!l_good_lic_key()
00418CE6: call 0041B127
sub_0041B127 lm_ckout.obj!l_crypt_private()
0041B13D: call 0041B37A
sub_0041B37A lm_ckout.obj!real_crypt()
0041BE72: call 0041CE71
sub_0041CE71 lm_ckout.obj!l_string_key()
; call ls_app_main.c!main()
; call ls_daemon.c!ls_daemon()
; call ls_app_init.c!ls_app_init()
; call lm_new.c!l_n36_buf()
; call lm_init.c!lc_init()
; call lm_set_attr.c!lc_set_attr()
; call ls_s_funcs.c!ls_s_init()
; call lm_daemon.c!lc_daemon()
; call ls_app_init.c!ls_checkhost()
; call ls_init_feat.c!ls_init_feat()
; call ls_init.c!ls_create_feats(), mode = 1, not important
; call ls_init.c!ls_create_feats(), mode = 2, this one counts
; call lsprfeat.c!ls_print_feats()
; call ls_init_feat.c!ls_feat_validate(), returns 0 if failure
; call lm_config.c!l_next_conf_or_marker()
; call ls_init_feat.c!good_config()
; call lm_ckout.c!l_good_lic_key(), returns 0 if failure
; call lm_ckout.obj!l_crypt_private()
; call lm_ckout.obj!real_crypt()
; call lm_ckout.obj!l_string_key()
 0041D8AC – 0041DB6B ; if{} block containing L_MOVELONG() macro, not executed
0041DB70 – 0041DC66 ; else{} block, executed
0041DC9A: call 0041E0E3 ; call lm_ckout.obj!our_encrypt()
0041DD75 – 0041DEA4 ; for{} loop comparing signatures

See that lovely baby l_good_lic_key()? We are in business. The rest is what we have done before, trace to
l_string_key() and uncover VENDORCODE and job structures. However, due to the presence of filters, execution
of l_string_key() is different now. Therefore we need to examine l_strkey.c again.

l_string_key(job, input, inputlen, code, len, license_key)

{
int idx = (*job->vendor) % XOR_SEEDS_ARRAY_SIZ; /* idx = a%20 = 97%20 = 17 = 0x11 */
... ...
for (i = 0; i < length; i++)
{
... ...
if (!user_crypt_filter && !user_crypt_filter_gen
&& (job->flags & LM_FLAG_MAKE_OLD_KEY))
{
... ...
}
else
{
for (k = 0; k < L_STRKEY_BLOCKSIZE; k++)
{
int shift = ((k%4) * 8);
unsigned long mask = 0xffL << shift;

/* SEEDS_XOR = mem_ptr2_bytes defined in l_privat.h */

y[k] ^= (((code->data[k/4] & mask) >> shift)
^ job->SEEDS_XOR[xor_arr[idx][k%4]]);

/* job->flags not NULL, LM_FLAG_MAKE_OLD_KEY = 1 */

if (!(job->flags & LM_FLAG_MAKE_OLD_KEY))
y[k] = reverse_bits(y[k]);
}
}
... ...
}
... ...
for (i = 0; i < j; i++) /* compare user-input and real checksums */
{
... ...
if (user_crypt_filter)
(*user_crypt_filter)(job, &x, i, y[i]);
if (x != y[i])
return 0;
}
if (user_crypt_filter_gen) /* executed only in keygen, not in checkout */
{
j = L_STRKEY_BLOCKSIZE; /* L_STRKEY_BLOCKSIZE = 8, in lmachdep.h */
if (len == L_SECLEN_SHORT)
j -= 2;
for (i = 0; i < j; i++)
(*user_crypt_filter_gen)(job, &y[i], i);
}
... ...
}

Filters cause several crucial branches. Recall that l_string_key() is called in both the checkout (ansyslmd.exe)
and keygen (lmcrypt.exe) processes. In the checkout process user_crypt_filter != 0, user_crypt_filter_gen
= 0 while in the keygen process user_crypt_filter = 0, user_crypt_filter_gen != 0. In particular
user_crypt_filter = 004098EA for ansyslmd. These two functions provide one additional layer of symmetric
transformation.
Nevertheless, before that layer in the else{} block, code->data[] and job->mem_ptr2_bytes[] are still xored to
reveal the raw encryption seeds. Hence we can acquire vendor keys and seeds in the old way. Below are the
structures dumped from memory.

VENDORCODE in ansyslmd.exe!l_string_key()
[0012E148] - 00000004 .... int type;
[0012E14C] - b31813e3 .... code[0], random and different at each run
[0012E150] - 21bf965f _..! code[1], random and different at each run
[0012E154] - 928689b2 .... true VENDOR_KEY1
[0012E158] - cd5480b3 ..T. true VENDOR_KEY2
[0012E15C] - fc17f999 .... true VENDOR_KEY3
[0012E160] - 05dd4739 9G.. true VENDOR_KEY4
[0012E164] - 00030008 .... FLEXlm version (here is 8.3)
[0012E168] - 38300062 b.08
[0012E16C] - 0000332e .3..
[0012E170] - 00000000 ....

job in ansyslmd.exe!l_string_key()
[002F4068] - 00000066 f... int type;
[002F406C] - 009800de .... char *mem_ptr2;
[002F4070] - 5513f8bc ...U unsigned char mem_ptr2_bytes[12]; random and different at each run
[002F4074] - 006ee662 b.n.
[002F4078] - 00bf0000 ....
[002F407C] - 00000000 ....
[002F4080] - 00000000 ....
[002F4084] - 00000000 ....
[002F4088] - 00000000 ....
[002F408C] - 33656e61 ane3 feature to checkout
[002F4090] - 00006c66 fl..

Note idx = 17 and the substitution table gives xor_arr[17][0] = 0, xor_arr[17][1] = 2, xor_arr[17][2] =
4, xor_arr[17][3] = 8, thus the xor operand assembled from job->mem_ptr2_bytes[] is 006213BC. Therefore
ENCRYPTION_SEED1 = code->data[0] ^ 006213BC = B31813E3 ^ 006213BC = B37A005F
ENCRYPTION_SEED2 = code->data[1] ^ 006213BC = 21BF965F ^ 006213BC = 21DD85E3

Before we study how filtering works, we point out several things worth noticing. First, the initialization is done in
ls_app_init(), where we see clearly how it handles command line options. This not only explains the behavior we
saw earlier on ansyslmd, it also implies that vendor daemon can be run as long as argc >= 4.

keysize = (*L_NEW_JOB)(vendor_name, &vcode, 0, 0, 0, 0);

memcpy(vendorkeys, &vcode, sizeof(VENDORCODE)); /* vendorkeys[] comes from lmcode.c */
for (i = 1; i < keysize;i++)
(*L_NEW_JOB)(vendor_name, &vendorkeys[i], 0, 0, 0, 0);
... ...
if (ls_a_user_crypt_filter)
lc_set_attr(lm_job, LM_A_USER_CRYPT_FILTER, (LM_A_VAL_TYPE)ls_a_user_crypt_filter);
... ...
if (argc < 4) /* check the command line argument number */
{
if ((argc > 1) && (!strcmp(argv[1], "-v") || !strcmp(argv[1], "-V")))
{
printf("%s v%d.%d%s - "COPYRIGHT_STRING(1988) "\n",
lm_job->vendor, lm_job->code.flexlm_version,
lm_job->code.flexlm_revision, lm_job->code.flexlm_patch);
exit(0);
}
else
{
LOG((lmtext("Vendor daemons must be run by lmgrd\n")));
exit(EXIT_BADCALL);
 }
}
... ...
master_list = ls_checkhost(master_list);
ls_init_feat(*list);
... ...

Second, in ls_init_feat.c the function ls_create_feats() is called twice. The first call is insignificant, only the
second call does real work. So remember to bypass the first one in debugging.

#define CREATE_SUITES 1
#define CREATE_FEATURES 2
... ...
ls_init_feat(LM_SERVER * master_list)
{
... ...
ls_create_feats(features, master_list, &f, CREATE_SUITES);
ls_create_feats(features, master_list, &f, CREATE_FEATURES);
... ...
ls_print_feats();
}

Third, after obtaining vendor keys and encryption seeds, we set out to rebuild FLEXlm SDK to verify them. During
the tracing we got the hash code for feature ane3fl to be y=4BE34851A0A5 before filtering took place. lmcrypt.exe
should give us the same number. But we encountered a compiling error when building utils directory.

lmnewgen ansyslmd -o lm_new.c

v8.1+ FLEXlm, non-CRO
lc_init failed: Invalid FLEXlm key data supplied
FLEXlm error: -44,49

This error haunted me for quite some time, I even doubted if vendor keys were also obfuscated by filters (this error
occurred before lmseeds.h was generated). Everything is checked carefully, including the ansyslmd vendor name
(all lower case) in lm_code.h and pc.mak, but still no luck. With no clues we had to debug lmnewgen.exe to figure
out what was going on. Finally we traced the error down to three constants that was claimed to be useless in [6].

lm_ckout.c!l_sg():
x = 0x6f7330b8; /* v8.x */
lmnewgen.c!VKEY5():
x = 0x6f7330b8; /* v8.x */
l_key.c!l_zinit():
/* z = crokey_flag ? 0x67607419 : 0x3cde3ebf; v8.x */
z = crokey_flag ? 0x62586954 : 0x72346B53; /* v9.x */

It turns out that they may not be used in the checkout process, yet they do participate in keygen for vendor key
verification, at least that’s true for z in l_zinit(). We did not bother to analyze this in [6] because SDK built fine
there. The problem is evidently version number: in [6] it’s 9.2, here it is 8.3, and that does make a difference in
l_zinit(). After we change the commenting in l_zinit(), compiling/linking pass successfully. Then modify
lmseeds.h, rebuild and run lmcrypt.exe, the output is 4BE34851A0A5 again. Now we are sure the vendor keys and
encryption seeds are indeed correct.

FLEXlm Filters

Filter is a mechanism offered by FLEXlm for customers who need extra security. Macrovision manuals [4] and [5]
only have a very brief description about it. [1] has one essay on this issue, but the version is 6.1. Basically we need
to figure it out by ourselves. The first hints come from one line in utils\pc.mak.

E:\flexlm\utils>lmrand1 -filter_gen 0x23456789 0x3456789a 0x456789ab

** Filter-Generator: Additional security **
This generates 2 source files, which you must *never edit*:

lmappfil.c: must be linked into vendor daemon, and all applications

calling lc_checkout(). These applications must call
 lc_set_attr(lm_job, LM_A_USER_CRYPT_FILTER, (LM_A_VAL_TYPE)user_crypt_filter);
after lc_new_job(). Also, lsvendor.c must have
extern void user_crypt_filter();
void (*ls_a_user_crypt_filter)() = user_crypt_filter;

lmkeyfil.c: this must be linked into all license generators:

makekey, lmcrypt, programs that call lc_cryptstr().
In these programs, after lc_init(), call
lc_set_attr(lm_job, LM_A_USER_CRYPT_FILTER_GEN, (LM_A_VAL_TYPE)user_crypt_filter_gen);

lmrand1.exe takes in three random seeds2 – for convenience we will call them FILTER_SEED1, FILTER_SEED2,
FILTER_SEED3 – and generate two files, lmappfil.c and lmkeyfil.c, which link to vendor application and keygen
lmcrypt respectively. What we have in the target is the traceable image of lmappfil.obj and our goal is to recreate
lmkeyfil so that we can build the new filter-enabled lmcrypt.exe. Needless to say, we must peruse the preceding
sample files and understand them first. The source code is not hard to read, but it’s better to read the generator
code in lm_rand3.c first where the call chain is lmrand1.c!main() -> lm_rand3.c!l_filter_gen().

#define MAX_CRYPT_BYTES 20 /* must be equal to numnames[] size, num0 – num19 */

#define NUM_BITS MAX_CRYPT_BYTES * 8
#define MASK_INIT_VAL 40 /* mask values are 0 - 7, 40 is invalid */
char mask[NUM_BITS];
unsigned char xor_vals[MAX_CRYPT_BYTES];

API_ENTRY l_filter_gen(int argc, char **argv)

{
... ...
setup_vals();
print_files();
return 0;
}

static void setup_vals() /* setup mask and xor_vals[] */

{
int byte, bit, r;
memset(mask, MASK_INIT_VAL, sizeof(mask));

srand8(seed[0] & 0xff, (seed[0] >> 16) & 0xff, seed[1] & 0xff);
for (byte = 0; byte < MAX_CRYPT_BYTES; byte++)
{
xor_vals[byte] = (rand8() / 256) % 256;
}

srand8((seed[1] >> 16) & 0xff, seed[2] & 0xff, (seed[2] >> 16) & 0xff);
for (byte = 0; byte < MAX_CRYPT_BYTES * 8; byte += 8)
for (bit = 0; bit < 8;)
{
r = (rand8() / 256) % 8; /* between 0 and 7 */
if (mask[r + byte] == MASK_INIT_VAL)
mask[r + byte] = bit++;
}
}

static void print_files()

{
int byte, bit, i, j;

for (byte = 0 ; byte < MAX_CRYPT_BYTES * 8; byte += 8)

{
... ... /* print variables num0 – num19, bit0 – bit7, x_0 – x_19 for both files */

/* for license generator part 1 */

*xorlinesp++ = ++xoroutcp;

2
Note that filter seeds are completely independent from vendor keys and encryption seeds.
 sprintf(xoroutcp, "\tif (idx == %s)\tin_c ^= x_%d;\n", numnames[byte/8], byte/8);
xoroutcp += strlen(xoroutcp) + 1;

for (bit = 0; bit < 8; bit++)

{
/* for license generator part 2 */
*liclinesp++ = ++licoutcp;
sprintf(licoutcp, "\tif ((idx == %s)\t&& ", numnames[byte/8]);
licoutcp += strlen(licoutcp) ;
sprintf(licoutcp, "\t(in_c & %s))", names[mask[bit + byte]]);
licoutcp += strlen(licoutcp) ;
sprintf(licoutcp, "\tc |= %s;\n", names[bit]);
licoutcp += strlen(licoutcp) + 1;

/* for application part 1, reverse of license generator part 2 */

*applinesp++ = ++appoutcp;
sprintf(appoutcp, "\tif (idx == %s)\n\t{\n", numnames[byte/8]};
appoutcp += strlen(appoutcp) ;
sprintf(appoutcp, "\t\tif (in_c & %s)", names[bit]);
appoutcp += strlen(appoutcp) ;
sprintf(appoutcp, "\tc |= %s;\n\t)\n", names[mask[bit + byte]]);
appoutcp += strlen(appoutcp) + 1;

/* for application part 2, reverse of license generator part 1 and bitwise test */
*appxorlinesp++ = ++appxoroutcp;
sprintf(appxoroutcp, "\tif (idx == %s) \n\t{\n\t\t", numnames[byte/8]};
appxoroutcp += strlen(appxoroutcp);
if (xor_vals[byte/8] & (0x1 << bit)) /* xor lines, the reversal part */
{
sprintf(appxoroutcp, "if (c & %s)\tc &= ~%s;\n\t\t", names[bit], names[bit]);
appxoroutcp += strlen(appxoroutcp);
sprintf(appxoroutcp, "else\t\tc |= %s;\n\t\t", names[bit]);
appxoroutcp += strlen(appxoroutcp);
}
sprintf(appxoroutcp, /* test each bit, return junk if failed */
"if (((c & %s) ^ (expchar & %s)) & 0xff) {*inchar = c ^ %d; return;}\n\t}\n",
names[bit], names[bit], (rand8()>>2) % 0xff);
appxoroutcp += strlen(appxoroutcp) + 1;
}
}
... ...
}

lmappfil.c and lmkeyfil.c each has one and only function, not surprisingly named as user_crypt_filter() and
user_crypt_filter_gen() respectively. They all share three arrays, declared individually as
num0 = 0, num1 = 1, num2 = 2, ... ... num19 = 19
bit0 = 0x01, bit1 = 0x02, bit3 = 0x04, ... ... bit7 = 0x80
x_0 = xor_vals[0], x_1 = xor_vals[1], ... ... x_19 = xor_vals[19]
Actually bit0 – bit7 have some random higher bits, but only the last byte (i.e. the 8 least significant bits) is used, so
we ignore those trash bits here. It is obvious that bit0 – bit7 are bit selectors.

The three filter seeds are applied to generate two random arrays, xor_vals[] and mask[], using Macrovision’s own
RNG. The process is practically a one-way function and FILTER_SEED are of no use to us once we recover the two
arrays because then we can remake lmkeyfil with arrays alone. It is similar to the relations between LM_SEED and
ENCRYPTION_SEED. The difference is that there it’s 3 to 4 and here it’s 3 to many, 20+160 to be precise.

The two function bodies both consist of two parts, let’s call them part 1 and part 2 in natural order. As symmetric
transformation, part 1 of user_crypt_filter() is the reverse of part 2 of user_crypt_filter_gen(), and part 2
of user_crypt_filter() is the reverse of part 1 of user_crypt_filter_gen() plus testing. To illustrate how the
reversal is done, certain lines are excerpted as examples in below. Note idx is the input byte index number, in_c =
*inchar is the input byte copy, *inchar = c is the output byte that overwrites the input, and expchar is the
expected correct byte for comparison.
 lmkeyfil.c!user_crypt_filter_gen() lmappfil.c!user_crypt_filter()
if (idx == num5)
{
Part 1 if (idx == num5) in_c ^= x_5;
if (in_c & bit6) c |= bit3;
}
if (idx == num5)
{
if (((c & bit4) ^ (expchar & bit4)) & 0xff)
{*inchar = c ^ 5; return;}
}
... ...
if ((idx == num5) && (in_c & bit3))
Part 2 c |= bit6;
if (idx == num5)
{
if (c & bit1) c &= ~bit1;
else c |= bit1;
if (((c & bit1) ^ (expchar & bit1)) & 0xff)
{*inchar = c ^ 235; return;}
}

The two reversals use different operations, one is OR and the other is XOR. OR is used to set certain bit in a byte
according to the permutation table mask[]3. In this example, bit 3 and bit 6 permutes in the sixth byte (idx = 5),
in lmkeyfil.c it’s (5, 3, 6) and in lmappfil.c it’s (5, 6, 3). XOR is controlled by xor_vals[] to negate certain bit
in a byte. In part 2 of lmappfil, besides the testing code, additional code is added for those negated bits to reverse
it back. Here that means bit 1 is negated in the sixth byte while bit 4 is not. In other words, bit 1 is set in x_5 while
bit 4 is not.

As for the testing, the de-filtered byte is compared against the expected byte bit by bit. At the first bit discrepancy,
random garbage is xored to the output and function immediately returns. This will almost surely fail the next test in
l_string_key(). We say “almost surely” because there is a chance that the random XOR would coincide with the
correct XOR, but realistically that probability is safely negligible. In conclusion when filters are turned on, real
comparison takes place in user_crypt_filter() rather than l_string_key().

Now we understand what filters do, next is how do we dig out xor_vals[] and mask[] from the target? Thanks to
IDA, we pinpoint the location of user_crypt_filter() and even the variables.

sub_004098EA ansyslmd.exe!user_crypt_filter()
004098EA – 0040A66E ; part 1
0040A670 – 0040BDCA ; part 2

typical part 1 code block ; ebp = idx, al = c, dl = in_c

0040A20A: cmp ebp, [004875C8] ; if (idx == num5)
0040A210: jnz 0040A222
0040A212: mov cl, [00487628] ; cl = bit3
0040A218: test cl, dl ; if (in_c & bit3)
0040A21A: jz 0040A222
0040A21C: or al, [00487600] ; c |= bit6

typical part 2 code block ; ebp = idx, al = c, dl = in_c, cl = expchar

0040BCAB: mov edi, [004875C8] ; edi = num5
0040BCB1: cmp ebp, edi ; if (idx == num5)
0040BCB3: jnz 0040BCDC
0040BCB5: mov edx, [004875D0] ; edx = bit1
0040BCBB: test dl, al ; if (c & bit1)
0040BCBD: jz 0040BCC7
0040BCBF: mov bl, dl
0040BCC1: not bl ; bl = ~bit1
0040BCC3: and al, bl ; c &= ~bit1
0040BCC5: jmp 0040BCC9

3
mask[] is a bad name, it should be called something like permute[] or map[].
 0040BCC7: or al, dl ; c |= bit1
0040BCC9: movzx esi, cl
0040BCCC: movzx ebx, al
0040BCCF: xor esi, ebx
0040BCD1: test edx, esi ; if (((c & bit1) ^ (expchar & bit1)) & 0xff)
0040BCD3: jz 0040BCDC
0040BCD5: xor al, 24h ; *inchar = c ^ 36
0040BCD7: jmp 0040BDC0 ; return immediately

global variables
dword_004875C4 dd 0Fh ; num15
dword_004875C8 dd 5 ; nunm5
dword_004875CC dd 8 ; num8
dword_004875D0 dd 102h ; bit1
dword_004875D4 dd 0Eh ; num14
dword_004875D8 dd 3 ; num3
dword_004875DC dd 2 ; num2
dword_004875E0 dd 0CC80h ; bit7
dword_004875E4 dd 0A720h ; bit5
dword_004875E8 dd 6 ; num6
dword_004875EC dd 0Ah ; num10
dword_004875F0 dd 7 ; num7
dword_004875F4 dd 12h ; num18
dword_004875F8 dd 0Dh ; num13
dword_004875FC dd 10h ; num16
dword_00487600 dd 740h ; bit6
dword_00487604 dd 11h ; num17
dword_00487608 dd 2801h ; bit0
dword_0048760C dd 1 ; num1
dword_00487610 dd 0Ch ; num12
dword_00487614 dd 13h ; num19
dword_00487618 dd 0Bh ; num11
dword_0048761C dd 9 ; num9
dword_00487620 dd 2704h ; bit2
dword_00487624 dd 3210h ; bit4
dword_00487628 dd 3108h ; bit3
dword_0048762C dd 4 ; num4

num0 is missing from the variables, but that’s not important, what matters is the order of variables and code blocks
are shuffled to increase the cracking difficulty. Typical code sections of part 1 and part 2 are also shown, again idx
= num5. Attention! Although it looks almost the same as the above sample source code, they are totally separate
things! One is from sample filters we built and the other is from real target.

ansyslmd.exe!user_crypt_filter() our sample user_crypt_filter()

if (idx == num5) if (idx == num5)
{ {
Part 1 if (in_c & bit3) c |= bit6; if (in_c & bit6) c |= bit3;
} }
if (idx == num5) if (idx == num5)
{ {
if (c & bit1) c &= ~bit1; if (c & bit1) c &= ~bit1;
Part 2 else c |= bit1; else c |= bit1;
if (((c & bit1) ^ (expchar & bit1)) & 0xff) if (((c & bit1) ^ (expchar & bit1)) & 0xff)
{*inchar = c ^ 36; return;} {*inchar = c ^ 235; return;}
} }

Note carefully the difference in part 1, the permutation of ansyslmd.exe is (5, 3, 6) while that of our sample
lmappfil.c is (5, 6, 3). They are NOT equivalent! In fact the former also has mappings of (5, 6, 2) (5, 2, 0) and the
latter has (5, 3, 2) (5, 2, 4); this way the difference becomes apparent. The symmetry of permutation is valid only
between affiliated lmappfil and lmkeyfil but not between two unrelated user_crypt_filter(). As to part 2, the
fact that both x_5 has bit 1 set is purely a coincidence. A good evidence for that is the two junk bytes differ from
each other (36 vs. 235).

This is about as far as static analysis can go, to discover the buried mappings and xor_vals[], we have to trace.
However, there are a number of tips that should be observed based on our experience.
The maximum length of signature is 20 bytes, that’s too large. Fortunately in our case ANSYS deploys short
license key composed of only 6 bytes (e.g. 1CEF0B37317C), which means idx ranges from 0 to 5. Hence the
number of permutation pairs reduces to 6 x 8 = 48 and xor_vals[] also has only 6 elements/bytes.
Beware of compiler optimization that fetches operand directly from register instead of memory. This disables
the identification of bit variable from its address and is one reason why static analysis alone cannot do the job.
The solution is to tell from register value. For instance, if EBX is 0x740, then we know it is bit6.
Be vigilant in tracing, especially in part 2 because a single mismatch would get the function returned instantly.
To avoid that we need to modify EAX frequently and set breakpoints at the right place. In both parts there are
8 total idx matches (1 byte = 8 bits) that we can count, but in part 2 the number of flagged bits is unknown,
requiring extra carefulness.

I have to say this tracing work is truly laborious, costs remarkable time and energy (maybe also eyesight), and I
dislike it. But the outcome is juicy. Let’s savor it:

Mappings
(0, 6, 1), (0, 7, 2), (0, 4, 6), (0, 5, 3), (0, 2, 5), (0, 1, 7), (0, 0, 0), (0, 3, 4)
(1, 3, 3), (1, 0, 7), (1, 2, 2), (1, 1, 6), (1, 5, 4), (1, 7, 1), (1, 4, 0), (1, 6, 5)
(2, 2, 5), (2, 5, 7), (2, 1, 3), (2, 6, 2), (2, 3, 4), (2, 7, 6), (2, 4, 1), (2, 0, 0)
(3, 4, 5), (3, 5, 3), (3, 7, 1), (3, 6, 7), (3, 3, 4), (3, 0, 2), (3, 2, 6), (3, 1, 0)
(4, 4, 7), (4, 7, 3), (4, 1, 2), (4, 3, 5), (4, 2, 6), (4, 5, 0), (4, 0, 4), (4, 6, 1)
(5, 7, 1), (5, 1, 5), (5, 4, 7), (5, 5, 3), (5, 2, 0), (5, 6, 2), (5, 3, 6), (5, 0, 4)

byte index bits set xor_vals[] value

0 4, 1, 0, 3 00011011 = 0x1B
1 3 00001000 = 0x08
2 1, 2, 4, 3, 0 00011111 = 0x1F
3 1, 5, 2 00100110 = 0x26
4 4, 5, 2, 1 00110110 = 0x36
5 2, 4, 3, 1, 0 00011111 = 0x1F

xor_vals[2] = xor_vals[5]? That’s also a coincidence. With these data at hand we can recreate lmkeyfil.c, of
course not identical source but equivalent functionality. The source code of our lmkeyfil.c is listed in the appendix.
To link it with lmcrypt.c we have one more thing to do: insert
lc_set_attr(lm_job, LM_A_USER_CRYPT_FILTER_GEN, (LM_A_VAL_TYPE)user_crypt_filter_gen);
after calling lc_init() in lmcrypt.c and put declaration
extern void user_crypt_filter_gen();
on the front.

Rebuild lmcrypt.exe and run it, we get the new license codes. For example, the signature for feature ane3fl is
4BE34851A0A5 before filtering, after filtering it becomes 18DBD99F53B3. Copy the new license file

SERVER changsha 000795f52adc 1055

DAEMON ansyslmd ansyslmd.exe
INCREMENT ane3fl ansyslmd 9999.9999 permanent uncounted 18DBD99F53B3 \
HOSTID=ANY
INCREMENT ansys ansyslmd 9999.9999 permanent uncounted 31260633E118 \
HOSTID=ANY
INCREMENT ane3 ansyslmd 9999.9999 permanent uncounted E8FC2C1BCC6E \
HOSTID=ANY
INCREMENT anfl ansyslmd 9999.9999 permanent uncounted DE173C2200B1 \
HOSTID=ANY

to c:\flexlm\license.dat and start ansyslmd, Wham! All features check out successfully. This confirms that our
new filter-enabled keygen works superbly.
Setback and Turnaround

At this point I thought it was over, just generate the whole license for 223 features and ANSYS should run without
daemon. But the unbelievable thing happened, ANSYS still reported the same error! I was caught totally off guard.
How could this be possible, when ansyslmd.exe accepts the license but ansys.exe rejects it?

I contemplated some potential explanations. Maybe the application and vendor daemon have different vendor keys
or encryption seeds? That’s highly unlikely. Different filter seeds or filter subroutines? Equally unlikely. Maybe the
filter resides only in vendor daemon and the application lacks its copy so that ANSYS has to contact ansyslmd for
checkout service? Well, if that’s true then there is no way we can run ANSYS as standalone program. The error
message says “The ane3fl license is a non-demo, uncounted license. This particular client machine is not permitted
to run with non-demo, uncounted licenses.”, suggesting that uncounted license is restricted to certain special hosts.
In that case we are also screwed. Finally maybe it’s simply our license missing something? If so, what can it be?

We should clarify that our license is valid if vendor daemon is used to start ANSYS, but that way the only progress
we made is the “permanent HOSTID=ANY” part. Our aim is primarily to eliminate the annoying C/S communication
and the hassles of maintaining the daemons, so it is the “uncounted” part of license that we are pursuing. I tried
keeping the original license intact as much as possible, but the resulting license (see below) still did not work.

SERVER changsha 000795f52adc 1055

DAEMON ansyslmd ansyslmd.exe
INCREMENT ane3fl ansyslmd 9999.9999 permanent uncounted B5C21B677341 \
VENDOR_STRING=customer:00265231 HOSTID=ANY ISSUED=10-Oct-2003 \
START=10-Oct-2003
INCREMENT ansys ansyslmd 9999.9999 permanent uncounted 919D058C6B26 \
VENDOR_STRING=customer:00265231 HOSTID=ANY ISSUED=10-Oct-2003 \
START=10-Oct-2003
INCREMENT ane3 ansyslmd 9999.9999 permanent uncounted B7789420A321 \
VENDOR_STRING=customer:00265231 HOSTID=ANY ISSUED=10-Oct-2003 \
START=10-Oct-2003
INCREMENT anfl ansyslmd 9999.9999 permanent uncounted D94AA7B07B4A \
VENDOR_STRING=customer:00265231 HOSTID=ANY ISSUED=10-Oct-2003 \
START=10-Oct-2003

Frustrated and desperate, I had no choice but go back to debug ansys.exe. This is so ironic because we believed
attacking the daemon is a smarter idea4 than attacking the application straight, yet after defeating the daemon we
had to pick up WinDbg again. To comfort ourselves a little bit: at least we have overcome the obstacle of filters and
have a working lmcrypt.

As stated before, IDA cannot handle the colossal ansys.exe, but we find a way out here. Under ANSYS directory
there is a folder E:\Ansys\v80\ANSYS\custom\ for users to build their own customized version. The batch script
\custom\user\intel\anscust.bat is intended for that particular purpose. Most of ANSYS code is written in Fortran
(GUI is using TCL/TK) and on Wintel platform the compiler is CVF exclusively. For customized building, the ANSYS
core comes in the form of libraries, mainly \custom\lib\intel\ansys1.lib and \custom\lib\intel\ansys2.lib.

E:\Ansys\v80\ANSYS\custom\lib\intel>dumpbin /archivemembers ansys1.lib | grep -i tlcka

E:\Ansys\v80\ANSYS\custom\lib\intel>dumpbin /archivemembers ansys2.lib | grep -i tlcka

Archive member name at 8827F6: tlcka_util.obj/
Archive member name at 884EFA: tlcka.obj/
Archive member name at 9DF2EA: systlcka.obj/

E:\Ansys\v80\ANSYS\custom\lib\intel>lib /extract:tlcka.obj ansys2.lib

Microsoft (R) Library Manager Version 7.10.3077
Copyright (C) Microsoft Corporation. All rights reserved.

E:\Ansys\v80\ANSYS\custom\lib\intel>

They are ordinary COFF libraries and where we crack in. Using dumpbin and lib tools the same way as in [6] we can
extract object files that we are interested in from the two libraries. Moreover, IDA is now able to perform excellent

4
Actually it is, it’s just that we need to do more work in this case.
analysis on these small object files, which proved to be really helpful.

sub_00401000 ; user code entry point

… …
00405404 call 00404F15 ; show splash screen
00405579 call [02C15484] ; call kernel32!AllocConsole()
004058CF call 00410640 ; show error message
sub_00410640
00410769 call 00458348 ; call ansys1.lib!kwinit.f!kwinit()
0041076E call 009367C0 ; call ansys2.lib!pdinit.f!pdinit()
0041077F call 004584C0 ; call ansys1.lib!begin.f!begin()
sub_004584C0 ansys1.lib!begin.f!begin()
004586A8 call 0099C4A0 ; call ansys2.lib!sysarg.f!sysarg()
00458A83 call 009A0750 ; call ansys2.lib!noticewrite.f!noticewrite(), show license agreement
00458A8B call 00487860 ; call ansys1.lib!config.f!config(), returns 7
00458A90 call 009A0B78 ; call ansys2.lib!systlcka.f!systlcka()
sub_009A0B78 ansys2.lib!systlcka.f!systlcka()
009A0CA8 call 0096E090 ; call ansys2.lib!tlcka.c!tlcka()
sub_0096E090 ansys2.lib!tlcka.c!tlcka()
0096E164 call 009485B0 ; call ansys2.lib!tlcka.c!tlcka_msg(), show debug level info
0096E1EE call 00957C70 ; call ansys2.lib!tlcka.c!tlcka_blddt(), show executable build date
0096E280 call 009485B0 ; call ansys2.lib!tlcka.c!tlcka_msg(), show client info
0096E952 call 0095A020 ; call ansys2.lib!tlcka.c!tlcka_out(), returns error code -73
sub_0095A020 ansys2.lib!tlcka.c!tlcka_out()
0095A0B9 call 00956BD0 ; call ansys2.lib!tlcka.c!tlcka_init()
0095A1B6 call 017679C0 ; call ansys.exe!lc_checkout(), returns -73
sub_017679C0 ansys.exe!lc_checkout()
01767A39 call 01767A8E ; call ansys.exe!l_checkout()
sub_01767A8E ansys.exe!l_checkout()
01767C1C call [03950130] ; call ansys.exe!lm_start_real(), [03950130] = 01768192
sub_01768192 ansys.exe!lm_start_real()
017682E2 call 0176FFC1 ; call ansys.exe!l_next_conf_or_marker()
017683D9 call 01768ABE ; call l_local_verify_conf(), returns 0=failure
01768580 call 0176922D ; call l_good_lic_key(), returns 1=success
sub_01768ABE ansys.exe!l_local_verify_conf()
01768AE3 call [ecx+F8h] ; call ansys2.lib!tlcka.c!tlcka_cofilter() = 00958000, returns –1
sub_009572C0 ansys2.lib!tlcka.c!tlcka_getvendval()
00957463 lea edx,[esp+18h] ; load edx = 04A8BF7C = _lgv_specunc_str
00957467 lea eax,[esp+290h] ; load eax = 04A8C1F4 = VENDOR_STRING
00957470 call 02C15638 ; call _imp_strstr() to compare [edx] and [eax]
sub_00958000 ansys2.lib!tlcka.c!tlcka_cofilter()
0095813D call 00957680 ; call ansys2.lib!tlcka.c!tlcka_procvend() to set _lgv_specunc
00958157 call 00957D80 ; call ansys2.lib!tlcka.c!tlcka_unclic(), returns -1
sub_00957680 ansys2.lib!tlcka.c!tlcka_procvend()
00957853 mov edi, 0392BBCC ; load _lgv_specunc_str
00957889 call 009572C0 ; call ansys2.lib!tlcka.c!tlcka_getvendval(), returns -1
00957892 mov [esp+38h],ebp ; set _lgv_specunc = 0 if tlcka_getvendval() returns nonzero
009578BC mov [esp+38h],1 ; set _lgv_specunc = 1 if tlcka_getvendval() returns 0
sub_00957D80 ansys2.lib!tlcka.c!tlcka_unclic()
00957E52 mov eax, 03932448 ; load string “NT/2000”
00957EC1 mov esi, 0393109C ; load _lgv_specunc

The key module here is tlcka.c, yes! that’s a C file according to dumpbin /symbols ansys2.lib, albeit the majority
of components are Fortran files. It is integrated with FLEXlm APIs to provide ANSYS specific checkout service. This
is both good and bad news. The good news is that we know ansys.exe does have local copy of FLEXlm; the bad
news is that tlcka_cofilter() is not even normal filters we discussed earlier, it is a special ANSYS version of filter
that we need to research (sadly, without source code).
An important thing here is that WinDbg failed to correctly identify FLEXlm APIs. For instance, instruction 0095A1B6
is shown as “call SPLIT_ELEM+0043AB60” and 01767C1C as “call G_ETUDE_EST_PPM+0000ADBC (01768192)”.
Then how could we label those functions in above list? We did it by matching parameters and comparing assembly
code. Again, at instruction 0095A1B6 the arguments in stack are (053B9B40, 0392F8C8, 0392F870, 1, 0, 0392AF28,
00004000). Looking at memory we recognized some familiar structures, 0392AF28:VENDORCODE; 053B9B40: job;
0392F870: 2003.0930; 0392F8C8: ansys; etc. Plus some peek into the function body, we know this is a call to
lc_checkout(job, feature, version, nlic, flag, key, dup_group). Similarly we can determine instruction
01767C1C is calling lm_start_real() with same parameters.

Notice here l_good_lic_key() returns 1 but l_local_verify_conf() returns 0, so the problem is not license hash
but config structure, which is defined in lmclient.h. We usually focus more on VENDORCODE and job structures and
tend to ignore config, in fact it is equally important. The call chains are

1. l_local_verify_conf() -> tlcka_cofilter() -> tlcka_procvend() -> tlcka_getvendval()

2. l_local_verify_conf() -> tlcka_cofilter() -> tlcka_unclic()

where all parameters are 053B9378: config. tlcka_cofilter() is invoked via a function pointer at the beginning
of l_local_verify_conf()

/* job->options->outfilter = tlcka_cofilter() */
if (filterflag && job->options->outfilter &&
(*LM_CALLBACK_OUTFILTER_TYPE job->options->outfilter)(conf)) /* make call here */
{
if (job->lm_errno == 0)
LM_SET_ERRNO(job, LM_LOCALFILTER, 125, 0);
return 0;
}

In the end it all boils down to tlcka_unclic() for the final return value. The following is its main disassembly code
adapted from IDA. Observe that the static IDA address is different from the real-time WinDbg tracing address, but
there is a correspondence: (IDA)0030C8AB = (WinDbg)00957EBD.

0030C8AB: mov eax, [esp+14h] ; eax = address of config

0030C8AF: mov esi, ds:_lgv_specunc
0030C8B5: mov edi, 0Dh ; edi = HOSTID_DEMO = 13 = 0Dh
0030C8BA: mov edx, [eax+50h] ; if (config.users != 0)
0030C8BD: test edx, edx
0030C8BF: jnz 0030C902
0030C8C1: mov ecx, [eax+78h] ; if (config.idptr == 0)
0030C8C4: test ecx, ecx
0030C8C6: jz 0030C902
0030C8C8: cmp [ecx+2], di ; if (idptr->type == HOSTID_DEMO)
0030C8CC: jz 0030C902
0030C8CE: cmp esi, 1 ; if (_lgv_specunc == 1)
0030C8D1: jz 0030C902
0030C8D3: mov ecx, ds:_lgv_display_level1_debug_msgs
0030C8D9: test ecx, ecx
0030C8DB: jz 0030C8F7
… … ; print out level 1 debug info
0030C8F7: pop edi
0030C8F8: or eax, FFFFFFFFh ; return -1
0030C8FB: pop esi
0030C8FC: add esp, 8
0030C8FF: retn 4
0030C902: mov ecx, ds:_lgv_display_all_debug_msgs
0030C908: test ecx, ecx
0030C90A: jz 0030C938
… … ; print out level 2 debug info
0030C938: pop edi
 0030C939: xor eax, eax ; return 0
0030C93B: pop esi
0030C93C: add esp, 8
0030C93F: retn 4

Annotation: config.idptr is a pointer to structure HOSTID, which is also defined in lmclient.h. It is clear that in
order to return 0 (success), one of the four conditions must be met: (1) number of users is not zero, i.e. it’s counted
license; (2) HOSTID does not exist; (3) HOSTID type is demo; (4) _lgv_specunc = 1. The first three are impossible,
we want uncounted license, uncounted license requires HOSTID to be specified, and we want non-demo fully
functioning license. Therefore the fourth condition is our last hope.

By the way real tracing shows idptr->type = HOSTID_ANY = 3, in accordance with our license. If we modify it to
be HOSTID=000795F52ADC, then idptr->type = HOSTID_ETHER = 2. Anyway the four conditions also explain the
error message – “This particular client machine is not permitted to run with non-demo, uncounted licenses”.

What’s _lgv_specunc? It’s a variable set in tlcka_procvend(). In that procedure a string called _lgv_specunc_str
is loaded and passed to tlcka_getvendval() -> strstr() for comparison (strstr is a standard ANSI C function).
What on earth are getting compared? Watching the memory we see _lgv_specunc_str is “specunc” and the other
is “customer:00265231”, the VENDOR_STRING! This indicates we should set “VENDOR_STRING=specunc” in the
license so that the comparison returns affirmative and _lgv_specunc would be set to 1.

We did that right away. Our expectation is right, tlcka_getvendval() returns 0, _lgv_specunc = 1, and eventually
l_local_verify_conf() returns 1. Everything seems to be working, but ANSYS still reports error! This time with
a new message: “ANSYS Error 103. Please report this error to your ANSYS Sales Representative or ANSYS, Inc.”.

I almost give up. I have no energy left to trace a clueless 103 error. At this very moment Goddess of Fortune (Tyche?
Fortuna?) smiled to me. By accident I deleted the SERVER and DAEMON lines in the license, and… it worked! ANSYS
main window appeared smoothly. We soon realized the problem is that SERVER and DAEMON lines direct ansys.exe
to contact ansyslmd, which per se is a conflict to the “uncounted HOSTID=ANY” status. So they have to be removed.
Actually, aside from VENDOR_STRING, other auxiliary fields can all be removed risklessly.

INCREMENT ane3fl ansyslmd 9999.9999 permanent uncounted F81FC239606B \

VENDOR_STRING=specunc HOSTID=ANY
INCREMENT ansys ansyslmd 9999.9999 permanent uncounted E62969FCA1E7 \
VENDOR_STRING=specunc HOSTID=ANY
INCREMENT ane3 ansyslmd 9999.9999 permanent uncounted C10CC93EA95B \
VENDOR_STRING=specunc HOSTID=ANY
INCREMENT anfl ansyslmd 9999.9999 permanent uncounted F7B48DE10B9F \
VENDOR_STRING=specunc HOSTID=ANY

In a short time we came up with a whole new license file (partially shown above) of all 223 features. It worked to
our full satisfaction (As it says, chance favors the prepared mind). It is interesting to see the debug info after
victory:

(DBG-2) ******************************** NOTE *********************************

(DBG-2)
(DBG-2) Being permitted to run non-demo, uncounted 'ane3fl' license
(DBG-2) on this non-Windows 98/ME client machine because of presence
(DBG-2) of the 'specunc' keyword in license.
(DBG-2) (This license line's encryption key: F81FC239606B)
(DBG-2)
(DBG-2) ***********************************************************************

Discussions

It is time to look back and summarize what we have learned in our work. We start with a little technical detail. The
version 9999.9999 in the license file looks quite strange, is it exaggeration? After all the target version is only 8.0.
It turns out FLEXlm has two types of versions: one is the conventional x.y version such as IMSL CNL 5.5 and Fluent
6.0, the other is the date version used in ANSYS, Maple and Intel Compilers. ANSYS 8.0’s incremental version is
compared to its build date 2003.0930 instead of 8.0, so it’s absolutely necessary to have a large version number like
9999.9999 in the license file.
The No.1 motive for our hacking effort is to remove daemon from the picture. We attempted to capture the C/S
traffic, which is caught by the firewall, and analyze it. Although ZoneAlarm can intercept C/S communications, it
provides no packet viewing functions. So we tried WinPcap and WinDump (c.f. [3]), but nothing was captured. To be
fair we tested WinDump on Internet traffic of browsers, RealPlayer, SSH, etc. and it worked very well. We can see
the bit stream of every packets sent from and received at our machine. Unfortunately it stops working for FLEXlm.
We think there has to be some internal engine difference between ZoneAlarm and WinDump.

Attentive readers may have noticed that we missed one major cracking technique – patching. Why don’t we patch
ansys.exe? It should be much easier than the ordeals we underwent. In fact we did try that. There are numerous
ways to patch and the point we select is the calling of tlcka_cofilter() in l_local_verify_conf().

01768AC4 837D2000 cmp [ebp+20h],0 ; if (filterflag == 0)

01768AC8 747B jz 01768B45 ; change to jmp 01768B45
01768ACA 8B4508 mov eax,[ebp+8]
01768ACD 8B486C mov ecx,[eax+6Ch]
01768AD0 83B9F800000000 cmp [ecx+F8h],0 ; if (job->options->outfilter == 0)
01768AD7 746C jz 01768B45
01768AD9 8B550C mov edx,[ebp+C]
01768ADC 52 push edx
01768ADD 8B4508 mov eax,[ebp+8]
01768AE0 8B486C mov ecx,[eax+6Ch]
01768AE3 FF91F8000000 call [ecx+F8h] ; ecx = 053B84D0, [053B85C8] = 00958000 = tlcka_cofilter()

Searching for binary string 83B9F800000000746C in UltraEdit leads us to the following offsets in ansys.exe

01368AC0 EC 83 EC 7C 83 7D 20 00 74 7B 8B 45 08 8B 48 6C
01368AD0 83 B9 F8 00 00 00 00 74 6C 8B 55 0C 52 8B 45 08
01368AE0 8B 48 6C FF 91 F8 00 00 00 83 C4 04 85 C0 74 55

The plan is to change 01768AC8 jz 01768B45 into 01768AC8 jmp 01768B45 so that conditional jump becomes
unconditional jump and the call to tlcka_cofilter() is then bypassed. Look up in [2] we find out the only patch
needed is to modify the instruction code 747B to EB7B. We did that and… the result?

It depends on what license file we use. At first the license had SERVER and DAEMON lines, and ANSYS Error 103 was
reported. This was before we figured out the reason of error 103 and I thought it might be some integrity hash check
(anti-patching measure to ensure the binary is not modified). That’s why it was such a devastating news to me when
the same error happened after setting VENDOR_STRING=specunc. But later we found out the source of error 103,
then we tried the license below (partially shown) and the patched ansys.exe launched successfully.

INCREMENT ane3fl ansyslmd 9999.9999 permanent uncounted 18DBD99F53B3 \

HOSTID=ANY
INCREMENT ansys ansyslmd 9999.9999 permanent uncounted 31260633E118 \
HOSTID=ANY
INCREMENT ane3 ansyslmd 9999.9999 permanent uncounted E8FC2C1BCC6E \
HOSTID=ANY
INCREMENT anfl ansyslmd 9999.9999 permanent uncounted DE173C2200B1 \
HOSTID=ANY

So we have two solutions, specunc license and patched ansys.exe, both need filtered signatures5. What are the
differences? On is in ANSYS main window “Help -> About ANSYS”, it shows “Customer: 99999999” for specunc
license and “Customer: 00000000” for patched program. Another one is the debug info. Compare the following
patching printout to the previous specunc one.

(DBG-2) ************************** CHECKOUT SUCCESS ***************************

(DBG-2)
(DBG-2) License checkout was successful for:
(DBG-2)

5
Of course we could patch ansys.exe to also bypass l_good_lic_key(), then arbitrary signatures would be OK.
 (DBG-2) Server:
(DBG-2) Product: ANSYS Multiphysics
(DBG-2) Feature: ane3fl
(DBG-2) Code: 18DBD99F53B3
(DBG-2)
(DBG-2) ***********************************************************************

A natural and interesting question arises, what does specunc, unclic, and tlcka stand for? What is UNC, Univ. of
North Carolina? Well, unless the software is SAS, but this is ANSYS. I believe unc means “uncounted” and specunc
stands for “special uncounted”. No wonder those variables are called _lgv_specunc and _lgv_specunc_str. The
debug info suggests that it is a special VENDOR_STRING requirement to run ANSYS with uncounted license on
non-Win98/ME OS. Maybe specunc is unnecessary on Win98/ME? I didn’t try and don’t know. As for tlcka, I have
no clue what that is. Maybe it’s “Type License Checkout ASCII”? Sounds like a joke.

We will not go to in-depth analysis about that 103 error in this article because we did not trace to its source except
knowing that deleting SERVER and DAEMON lines can solve the problem. There may be other problems like this, I
remember seeing ANSYS Error 102, but within the scope of this paper we are happy about what we got.

Back to FLEXlm, the l_zinit() constant glitch is a topic that never got chance to be emphasized before. We know
vendor keys are assigned by Macrovision while vendors have the freedom to choose LM_SEED and FILTER_SEED. It
is no surprise that there may be some test on vendor keys, e.g. correlation to vendor name/ID. It only happens in
keygen process where raw vendor keys are supplied. And Macrovision takes great length to upgrade that vendor key
verification in every new version. However, it poses little trouble for hackers who possess FLEXlm SDK. The simplest
workaround is to circumvent such test in l_init().

The final comment is on FLEXlm filters. We should say it is a successful security measure, well designed and
implemented. Surely it is beatable for it is still a symmetric transformation. Like those obfuscations, they are all
reversible. Unlike obfuscation, which uses only XOR operations, filters employ both OR and XOR. OR is used for bit
setting while XOR is used for bit negation.

The highlight of filtering is that it greatly increases hacker’s workload. In our case, license key is short (only 6 bytes),
and it yields 48 mappings and 6 bytes that we have to uncover from the target. The tracing process is really arduous.
If the signature were longer (up to 20 bytes), the workload would be too heavy to carry out manually. The solution
should be automated debugging, where tracing can be performed automatically by a program and result be
recorded. It should be doable, but we didn’t do it so we cannot say.

References

[1] CrackZ, FLEXlm – “Dubious License Management”, http://www.woodmann.com/crackz/Flexlm.htm, 2003.

[2] Intel, IA-32 Intel Architecture Software Developer’s Manual, Volume 2: Instruction Set Reference, 2001.
[3] Loris Degioanni, WinDump: tcpdump for Windows, http://windump.polito.it, 2004.
[4] Macrovision, FLEXlm Programmers Guide 8.1, February 2002.
[5] Macrovision, FLEXlm Reference Manual 8.1, February 2002.
[6] truth, On Software Reverse Engineering, April 2004.

Appendix

/* lmkeyfil.c for ANSYS 8.0

* Recreated keygen filter with data, namely xor_vals[] and map[],
* uncovered from ansyslmd.exe, the daemon that has lmappfil.c
* built in. Note these data are for ANSYS 8.0 only and cannot be
* used elsewhere. However, function user_crypt_filter_gen() is
* good as long as Macrovision does not change the filtering algorithm.
*/

#include <lmclient.h>

#define MAX_CRYPT_BYTES 6 /* short license key */

/* values converted from binary 8-bit sequences */

unsigned char xor_vals[MAX_CRYPT_BYTES] =
{0x1B, 0x08, 0x1F, 0x26, 0x36, 0x1F};
/* mappings, or permutation table */
unsigned char map[MAX_CRYPT_BYTES * 8][3] = {
{0, 6, 1}, {0, 7, 2}, {0, 4, 6}, {0, 5, 3},
{0, 2, 5}, {0, 1, 7}, {0, 0, 0}, {0, 3, 4},
{1, 3, 3}, {1, 0, 7}, {1, 2, 2}, {1, 1, 6},
{1, 5, 4}, {1, 7, 1}, {1, 4, 0}, {1, 6, 5},
{2, 2, 5}, {2, 5, 7}, {2, 1, 3}, {2, 6, 2},
{2, 3, 4}, {2, 7, 6}, {2, 4, 1}, {2, 0, 0},
{3, 4, 5}, {3, 5, 3}, {3, 7, 1}, {3, 6, 7},
{3, 3, 4}, {3, 0, 2}, {3, 2, 6}, {3, 1, 0},
{4, 4, 7}, {4, 7, 3}, {4, 1, 2}, {4, 3, 5},
{4, 2, 6}, {4, 5, 0}, {4, 0, 4}, {4, 6, 1},
{5, 7, 1}, {5, 1, 5}, {5, 4, 7}, {5, 5, 3},
{5, 2, 0}, {5, 6, 2}, {5, 3, 6}, {5, 0, 4}};

void user_crypt_filter_gen(LM_HANDLE *job, char *inchar, int idx)

{
int i, j;
unsigned char num[MAX_CRYPT_BYTES], bit[8];
unsigned char c = 0;
unsigned char in_c = *inchar;

for (i = 0; i < MAX_CRYPT_BYTES; i++)

num[i] = i;
for (i = 0; i < 8; i++)
bit[i] = 0x01 << i;

/* first pass */
for (i = 0; i < MAX_CRYPT_BYTES; i++)
if (i == idx) in_c ^= xor_vals[i];

/* second pass */
for (i = 0; i < MAX_CRYPT_BYTES; i++)
for (j = 0; j < 8; j++)
if ((i == idx) && (in_c & bit[map[i*8+j][2]]))
c |= bit[map[i*8+j][1]];

/* output */
*inchar = c;
}