Netscape Documentary cC 50

Tape 10 Jim Roskind Interview

So uh the big discussion came in January uh in general uh thered been some talk
guess

back and forth about things like this over number of months and all of sudden word

came down that was thinking of doing this and they wanted to
management really seriously

know is there any reason why we couldnt do it Uh and they also had suggestion that

they wanted to release it in the first quarter which is by 3-31 uh hence the name of this

Uh it
was uh-uh the big question was what-wotthbtop-us what
ongoing project

was the reason why we couldnt do this And they gave us think almost somewhere

between three days and week to try and decide if there was something that we were not

about and uh what they could do you know how we could do it how we
thinking

about getting involved and finally said definitely were going to do this mean

fundamentally whats going on is uh you know Microsoft is uh competing very strongly

against us and uh this-the idea is to try and roll the dice little bit andmake this
guess

more interesting for everyone Uh theres some old quotes about uh terrain being the

most critical in combat situation and think were smaller company able to move faster

and the hope is that we can bring this to bear so that Microsoft cant compete as well Fact

Ive really seen positive things come out of this where uh weve been rated as better

browser becaus now were cost competitive But anyway so it-it came out in late January

uh there Iliad meeting with Bob Lisbon the Senior VP of all of client division
wer
and theyve gtT20 other people saying Are there any show stoppers Are there
any

show stoppers No one could really think of uh a-a dead problem and then uh guess

Mark Mark Andreeson went public with this that we were definitely going to do it

And then the question is how can we do it

quickly Now 3-31 is coming up pretty
fast

The decision was made mid to late January so this gave us little over two months and the

tmth is if
you back off the schedule and you say well if
you want to be done on 3-31 you

really
need to be aim for 3-15 so that youre dead youre dead done and then you have

some slack to track care ofrSsizlasruokuattwuds.AtQ1ti

Ahem the goal then is really to get
done by 3-15 We had really two months
Two
.0Et7
Page
 Netscape Documentary
Tape 10 Jim Roskind Interview ci

months is an incredibly short piojt cycle In some sense its not product In another

sense it
really is Were gonna show the code to people Uh and so there-theres the first

that people were nervous about is all sorts of profanity that might sit in the code Anything Kiu

uh-uh people didnt like certain variant of UNIX and uh there sort of abusive terms

relating to the operating system and-and truth be known lot of us consider this non

professional mean uh theres there are examples where companies have had things

appear accidentally on their disk profanity which is even outside of the code and gotten all

77 r--
sorts of flaks-flak from different groups you know its not part of thcir able code so

were very concerned about that There was concern that were going to be infringing on-

on or trademar-ar-or of restricted code The

copyrights any sort thing is its hard in this

business to know whats patented Patent things things are moving along quickly
very

uh theres certain chance that were using something and dont even realize it People

from the outside cant-cant see what theyre doing what were doing but once you look

at the code you go oh theyre using this algo-rhythm They are using this-this

technique But for engineer it would very difficult to see what source code it would

very straight forward to see it not that we stole it but simply independent

invention and patent its its So

moving along a-its very scary business at this point

ub we started looking at that The big thing that hit us that we didnt really forsee in that

first week is the complete consideration of third pirty cride It turns out the numbers in
are

excess of about..ttydifferent third parties that contributed of code to the

portions system

that we consider communicator And now the realization was gee we cant this
give away

third party code without their agreement So now there are-there are several possibilities..

one is to get them to sign up to the actual uh they can call it NPL Netscape public lisense

and try and convince them to for it Uh but you know in truth of these people_are
go alot

making money by selling this code mean there are-there are versions of code where you

\pay thousand dollars

you get whole big library including source code And you

bundle it into and your liense allowed

your product saysyoure to see and use the source

Page2
 -1
Netscape Documentary
Tape 10 Jim Roskind Interview

code and ship the binary Thats common scenario Uh as an example Microsoft has

all sorts things built into their

compiler uh that are standard routine such as sorting

routine of sorts uh Quick Sort And be it there are no unstandard algorhythms but

theyll actually give you copy of their source But the intent is not that you will

redistribute that source and make your own compiler using their system but rather youll

just use it as component of the system youre shipping So we started talking to

.JesV A-
different One of here has been
players my jobs historically the job the job-of security

architect obia littl bit the liason through J-Java for source 3-31 one as we call it Um
and so had to consider the issue of Java and approach Java Soft Am running on too

long

So uh so laison to Java had to raise the elements of what can we do about Java Java

represents large amount of source which is bundled into the communicator and is clearly

restricted in source form by Java Soft The question is can we ship it What-Whats the

scenario which we can

under ship it And can talk about that if youd like uh but it

fundamentally going to all these different vendors Java Soft gives

Maybe good

examples of-of ways that we tried to consider working around the

system Uh Java Soft

actually distributes their source under fairly uh easy to get lisense The one restriction is

when you go to their site

you have to agree to take it for non-commercial use If you want

commercial use you have to you pay you know dollars and you-you get certain rights to

use it But noncommercial use people get to see it So the approached that another large

uh software distribution or free software distribution

system Ticklinex is that they

distribute that
patch is list of changes from the freely distributable source Now the

idea is linex gives this and you get from Java Soft the
you actual
underlying source

Then there are tools that just merge them together and now you have really home-brewed

version So idea is we dont

the
really ship the restricted source we say go to Java

Soft and get it Well just shif-ship patch But everyone and their brother knowsthe

Page
Netscape Documentary
Tape 10 Jim Roskind Interview

put them we made

intent is all In our case
just together so many changes to Java Softs-

virtual machineJava Softs source code that it was hard to use tool to do this So the

approach that took is that we went to Java Soft and we said hey could we ship your

source with the changeweve made under your highly restrictive lisense Now it doesnt

help the outside world them much but at least it

gives them something to play with it

gives them model \Yeu harc to want to

develop a-an alternate YM an alternate uh Java

virtual machine They can plug maybe Cava theres whole bunch of different

Obi
variations Also other vendors can think about integrating it All VM vendors that are

concerned about doing integration already have lisenses can say oh this is how we plug

it in and theyll have much easier time integrating So it would nice for everybody

although it wouldnt be the perfect scenario it would be pretty good Uh the problem is

with two months

that to go from basically 1-15 to 3-15 we have to put deadline of

listen youve got to agree to this weve got to get it in writing if were gonna move

ahead not we have out We

say Its
If to pull the
just plug and have to tell the public

and thats exactly what happened We passed 2-24 February 24th that was the official

deadline and at that point uh we can no longer wait for agreement from Java Softwace

We really were gonna be put in position where we couldnt out of- get out of this
get

trouble uh so at that goint we began cutting So weve been cutting now roughly 2-24 2-

26 that week ____

iIpuk1 Java out And this represents big effort to not just remove it but

you want to gracefully remove it You want move Java but still leave navigator

That sounds
functioning like saying okay we decided we dont want to side window

Uh and maybe you have to now remove it but you dont air out so you
really blowing

have to out how

figure to plaster that
up so that uh this thing is
someway useable Yes

you cant look out the window You cant do everything you could do with the window

c4
but it still functions like so its an interesting thing removing pieces and trying to leave

functional underneath
body SIn some cases weve had harder troubles with this uh theres

database that we were using its part of mail system and in that
case not only couldnt

3S
/0
Page
Netscape Documentary
Tape 10 Jim Roskind Interview

we ship it and couldnt really get all the stuff straight but this was fairly pivotal to all of

our mail side of this product and so the bottom line is we ended up think the current plan

is still lithe bit in the air is fear that this will be viewed
up is
...eres
little as

you do
the
misrepresentation to net but my answer is the best
you can you tell the net

what youre doing you try to be honest an open and thats think what were going for

right now Youll see that number of different pieces theyll be funny sections uh

security is another example of major piece of code Now this is probably the most

interesting one because the third party we unintelligble all the code but theres this

ery
restriction by the Federal Government on how we can distribute this code its getThe

fekb4t4code So now the hard part is what can we do to the whij\

satisfy government

effectively is the third party and ship it Um and this is even stickier because now were

not at risk of being sued were at risk of being

put in prison which we consider bad

would slow our software

thing it
development significantly So anyway the hard

question we put to the government is how much do we have to cut and their answer is

pretty simple they said you have to make it look like there never was any security in the

system you have to make sure theres no sign of crypto or photography encription if
you

can do that then

you can ship it and now the bad news its not
is
cryptography is hidden

hidden its tied deeply into the system its almost like with car saying you can ship it

but you cant use electricity Now

in
your car w-ll csll
you the lightbulbs oh no you

dont need lighting the cars only driven by day thats right but the wires you cant

leave the wires in because it shows how you would wire the car and so theres
very

large effort think this was Jeff Wienstein and hes put in quite an effort to try and

remove this thing which is

very in fact one of the of
integral initially major components

navigator one of the things that differentiated from think

it related products Mosaic was

that we had SSL secure socket layer this is of talking

cryptographically strong way to

sites Alot of commerce much more

feels comfortable banking and buying feels much

more comformtable communicating across this but this uses cryptography uh and the

Pages
 iY/
Netscape Documentary
Tape 10 Jim Roskind Interview

bottom line is we couldnt leave that in at all People started to wonder well gee heard

weak
you can export cryptography but not strong cryptography why wont they allow you

to export source for weak cryptography the answer is typically the difference between
ti
weak and strong is whether 40 or 128 oneoint which visible So
you type at is
very

theres no real
way to narrow it down This is an example of the same idea of cutting

things out but the goal is to make it useable so now comes question what happens

when you go to site that to use SSL how does the and
expects you system respond is it

exceptable And so the cutting has to be very delicate Im sure you can imagine if you

had no electricity all of sudden youdf have to think know how do we fire the
you

chambers well OK its diesel engine thats right we use diesel engine no electronic

ignition no headlights uh we cant have warning lights on it so all sorts of things have to

be 44d out and the end result is

yes you do have car it wouldnt surpirse me if some

early cars had very little in the

way of electrics or could be in that
managed way but its

alot of work to this done And

get so theres you_see java going away you see crypto

going away uh in the end some people will come and say hey why did you say you were

shipping this and were hoping that the Net will be understanding of some of this uh we

cant

it
its
interesting

has to just be completely

with the

unclear
crypto

where
laws we cant

the
crypto
even

goes
leave holes

in There are
for plugging

laws that
in

youre
crypto

not
ii
allowed to whats of plug and play crypto oh
you cant make
ship called sort even it so

this where you would plug

is in
crypto module but were not allowed to plug it in It

has to be completely devoid of connection so thats its very difficult this is all theres

uhole lot of arguing about whether this is or

right wrong it
certainly impacts us but in this

case you have too months to go you cant go about any slow
this in
way you have to be

very direct the most direct ways complete removal

Describe the process of code writing

/0 /7
rz.

Page
Netscape Documentary
10 Jim Roskind Interview
Tape

and
There is some code creation in the analogy said you remove window you have to

doesnt blow out and see find intenable untenable

put in something so that the air people

to car uh so you have to put in wooden plank in the window

so really pain driving

used be crank to crank this up

uh now that has to connect to the other pieces like there to

and down you have to tie that in you have to in and make delicate adjustment the bad
go

may decide oh that crank is third

news is while youre doing this someone else really

party uh software you have to replace that crank with uh uh an electric power

window just to get by no more cranking up and down just locking so theyre chaning

for the
the lock youre changing the window and when you go to connect you each look

of the window and goes

you know
bolts they the crank looks for the
guy changing part

wow its gone the guy putting in the window looks for the crank and goes theres nothing

to connect with this issue of when you have many people working on the project at the

same time and really rushing you cant just say OK you replaced the window OK

youre done and now that youre done OK you replace the crank People are trying to

this large code base think its in the of million lines of code and
modify neighborhood

theyre running around we have Im not sure what the count of engineers is certainly Idl

in of 100 involved this in various ways all

say theres excess theyre running
in
engineers

around with different tasks and then they have whats called merging problem basically

with youre of the car which is the code the code is really
youve go off snapshot design

you know design and you come back with the changes that youve made with your

design and youre about to check it in and make it

officially part of the design and you

notice right where it checks in that the picture has been changed and now you have to

merge you have to carefully look at their changes and your changes find way to work

them together Now when youre working on veryjiow schedule or if its just one

person everything is nice and sequential everyone knows whats going on Uh when

with 100 its difficult lets see if can site an

youre working people very very

example of merge problems Uh here am Im working on group withftom Ronsi te

ii
Page
 iC

Netscape Documentary

Tape 10 Jim Roskind Interview

of security
and were Well java it turns out makes alot
andUbert Shaw tearing out java

should we do about those security calls Well the

calls and now the question is what

in and cut out all the security so they actually went into
security group is assigned to go

At the same time we said well were not going

Java and put little cut marks around that

lets just adjust this of code so that it doesnt make the call
to call so ptI piece
anymore

chcck and we realize they changed code we changed Now in

no we go to it in suddenly

in the of 50 files and very large

the case of java were about to check in
think its range

number of changes and now were about to check it in and it

gets pretty scarry knowing

technical
that its about to break if it
gets put in Now if you break the build thats the

in that doesnt allow complete car to be rolled off see

if something
phrase you put

they build the car and they test

drive it
everyday they take snapshop from that design

off the assembly line doesnt work and they keep

So if ever its the case that car coming

evolves and they just stop and

testing
them time and again its this as this design

and clear whats broken all they know is

panicks little its usually not even
everyone

Imin the turned it and nothing happened And well

the car put my key in ignition

thats
you now have to crank the car the
the electricity
you took away
all
the answer is if

new way and the oh didnt even realize yeu had to crank the car and jill
guy says

where the starting motor goes but the

the in the crank connected to
maybe guy putting

You know so theres all these miscommunications

whole starting motor housing is gone

the on the floor everyone gets scared And

and so the problem is that once system falls

the problem
the goal is to stop changing anything uh the hard part in debugging or finding

if$ have short of time were

is to locate the actual problem In this ease very span

sure theres only been couple of changes and we know to look at exactly those changes

as to which one of the many changes

If have long of time it
gets very
span cousing

what broke the So typical1 once the system breaks we just stop dead
really
is system

and we to then do little and look back so we can see whats chang and
try autopsy

we want to but the bad

what we can fix So uh the idea is that to cause everything stop

iS

Page
Netscape Documentary

Tape 10 Jim Roskind Interview

in theyve made their changes they

news is else is racing to put their changes
everyone

And now when the whole system

and then go on to the next task so
put them
want to in

this line all comes to halt and now you

assembly it
you have this is
stops suddenly

view
have people spinning their wheels saying well want to check in my new side

cant this new side view mirror will

mirror uh but the whole door is apart really see if

hub caps but then the bad news

go work on the
work and4f-ycu put it down then can

wait for half day and

Im to what was doing and Ill just wait Ill jsut
going forget

this stuff checked in But now what youve done is youve instigated inefficiency
get

have
the system Again two months its desperate schedule you cant any ineffeciency7

that theres to be confIii

the deal is when you go to check in if you see going
so typical

down carefully poll do

rather than risking breaking the system you have to slow yourself

of the system that really worked

the integration be sure now you have perfect snapshot

and go check
and finally say ah now Im done and then you turn around face the system

has changes and theres another piece

it
again and much to your chagrin something lese

merge hell You were mentioning

this
that has to be merged Yocbuld actually call this

the time that this

in some of the meetings different people are in merge hell basically

stretches across alot of the system then

that
occurs if
you have very large change really

And fix one thing but you wait

its almost guarenteed to hit someone else everytime you

something else will break So finally the

two hours four hours day youre guarenteed

call whats called the car pool and they say listen youre in merge hell we can get you out

in its not broken its all

the the world stops people checking
stop changing system stop

take breath fix all the differences check your

working you get to poll calmly deep

and then release the tree will be the phrase So then youll see
drawing change in re-open

the tree from

this commonly happen when people get into merge hell theyll stop

its to land
the car car pool really easy
changing theyll allow pool
call
merge they

small change if all you want to do is crss off the DOT sticker you know on the window

in and do that and be done And then you go to another window and do
you can go

Page
Netscape Documentary
Tape 10 Jim Roskind Interview

another DOT If you have to remove all the glass in the car at once and it turns out that

the glass is related to other things electricity thats an easier example you cant jsut say

well Ill remove the battery so then the car stops working You have both remove the

battery remove the starter insert crank you have to remove the ignition switch you

have to insert all these things have to be done at once you cant get away with doing one

without the other And so thats why you have this large change Its always nicer some

take out the lights the

people will try to break 1arget theyll say well could

electricity has to go Ill take out the lights first OK thats out Now can remove this

other But at some point you get down to the starter motor and the ignition system
piece
CoCV r4pC
and it all has to be changed out at once you have to switch
fro ignition system to

diesel and you cant do meal and theres lot of other people doing skip checks
it
piece

Yesterday

Yesterday was Monday the start of the week and people were getting really nervous

cause now its 3-10 and theyre hoping to be done really by 3-15 basically by Friday

and there are two major car pools that hadnt landed One is the security car pool and the

second is the java car pool Now there was sort of race between the two of us to see

who can get out there fat faster or be ready faster there was maybe almost there was

another thing we talk about here called schedule chicken uh schedule chicken is where uh

one group realizes maybe my group could realize we couldnt be done by Thursday but

we tell them we could be done by Wednesday knowing that the other group wont be done

by Friday and all the attention will be focused on them and then we can be done by

Thursday and no one ever knew that we were really ever going to be late So its game of

whos CCn In
chicken the question is going to tac cnt7

Page 10
Netscape Documentary
Tape 10 Jim Roskind Interview

they announced they were

and the going to
..tape begins finally yesterday at meeting

check uh and was going to in the afternoon the tree was being closed so that
it it
happen

meant check in the tree was going to moving and

also that since they were going to stop

call now the back burner They told us at the time at the meeting
certainly a14 p001/ put on

worse case could be done by Wednesday which is thought because again Friday
it
scary

cGr
is the goal we have major eØl pool coming in were all so we dont want to be

scaredk

on the hot seat we dont want to be the one to cause the prob the program to slip uh so

car
yesterday there was alot of discussion about the landing the security call pool and what

what is...exactly is uh the uh the Java lIpool gonna do about this the first thing we

mentioned we were then now were talking about recovery plans mentioned we took

snapshots of where things were before the security Si pool started landing in their change

caused too much bustage again another clever technical term uh the-the chanebes we

could always roll them back out and let us go first and let them think about what they were

e4 Iccrd
going to do little bit more clearly Uh Ynfle mentionedfl Jeff Weinstein or his

people coming from his group his goal he was just going to stay all night and be sure the

call pool cleared everytime there was an error he was going to turn it back around The

tree wasnt going to be moving so the ground wasnt going to be shaking under his feet

and he was going to get

it all done The good news is actually think by about Im not

sure if it was or 11 oclock at night he actually was completely done So then we got

call was here Rahman and Yube and uh Hey their call pool cleared how are you
ri

whether we not at that Its little

guys feeling lockcditWtui1aiiitnrbuitthng- point

scary going in late at night when whole bunch of changes have landed because you dont

have the full support of the other engineering staff if there was problem outside your urea

e4 expertise its
pretty problematic Youll see today in meetings theyll be talking about

the Java call

cc
Carnrnnland make it What
pool.. is it
going to exactly are the risks

involved Last night we did merge the tip which is trying to adjust our design to be as

LC
11
Page
Netscape Documentary
Tape 10 Jim Roskind Interview

close as possible to the most recent most recent set of code and uh merged i3
design it
up .j

nicely but then when we brought it

up all of sudden we couldnt it called clicking on

links you know major function you bring up the whoeage you see an underlined word

it talks about some interesting part tbati-4terpiiig4ttllpprttugetber and it

just

sort of stares you in the face and go holy cow somethings brokenl and now came

question.. did someone else break the system before we entered this new sytem status or

did we just break it when our merges conflicted with items that were sitting there So we

were trying probably til about three in the morning to figure out what the heck is going

on finally decided the answer is take break go home get some sleep come back in

today Tuesday uh have some more people with more expertise in Unix to look at the exact

Unix problem were facing and uh see if we can diagnose it So were little bit on the

hot seat We thought we were good to go were about ready to check in were doing

final final final checks making sure everything merges again We cant afford to land

unless we can assure safe landing Uh and now we suddenly had a-a problem Were

going around were circling the clock is ticking its Tuesday theyre going to wonder

are we going to make it Theyre on the outside theyre listening to me and they have to

wonder. is Jim really understanding whats going on are these guys misrepresenting

intentionally common theres an old joke uh in uh computer software where

Es really
they say you know are you gonna be done and the answers oh this is the last bug

here This is the very last bug.t And obviously every

time you get one more bug more

things work and then often you find another bug But each time you find one everything

up to there ran perfectly so you have this funny feeling that everythings working

wonderfully without there being problem in the design uh and so its pretty nerve-

for
wracking management Management commonly has this problem with software

that
engineering its-theres alot of stuff going on its
easy to misunderstand details uh

you know its-the-anothe/guery is that ninety percent of the work is finished in the last ten

of the time This is of building house Its true of building

percent true so many things

Page 12
Netscape Documentary

Tape 10 Jim Roskind Interview

hot
they dont
want to be done want to be in the
and uh it
gets very confusing People

know for that theyre

theyre done but they dont have good way to sure
seat they think

Im pretty have good team think were

dead done So uh confident pretty

Youll see alot of nervousness in

bit of termoil
representing it well but youll see little

confidence
Michael Uh hes driving this uh ovei time theres rapport and theres

him were in hes going to tend to believe

thats built up so when tell that good shape

what So
me clock and you have to see happen
11
But at the same time the is ticking

be going out today alot of worries about that We do

big stuff thats
thats-thats the to

alot of through alot of little items

the tree up to the call pool so little people got
open

night and released

these are not little people people got to inject small changes last so that

the to long for call pool you slowly get

some to the tension note that if
you close tree

ii cr
slowed down Soa-eah
more and more people standing up and shouting theyre being

does does the of

person but group
it
pool is meant to expedite this one it it at expense

and constrantly have to trade off if you slow the group too much and you finally have
you

them go by So its this interesting delicate balance and the

to get out of the way and let

constant question Michael is wondering. are they being honest with me Can they

done in just few hours Or should let the people who are stacked up behind
get this

them go And he has to mak at judgement call And uh guess thats why they pay

him some money

now have one-on-one

Uh you can do that Right now uh oficially right
three

have check to see if theyre in

discussions a-aside from the fact that a-unintelligible first

yet They havent come in

yet thinls
unless unintelligible

Oh yes yeah totally

Page 13
Netscape Documentary
Tape 10 Jim Roskind Interview

most of the most the stuff in thc room

Uh oh the answer is.. stuff yet-got is everyone

going why is this breaking sheet whats going on here Last night you know we are

and theyre i-iits It

looking why is the suddenly breaking going goshw-w-w-
f
doesnt thet that you see is people working Theres the three of us on Tuesday
p-nti

nights and Thursday nights have been trying to go out and play raquet ball -.hfec.ea1lLL

And this is now like unless we get this check in theres no raquet ball tonight and
looking

uh so it will be interesting in general try you have to realize also as us managerse

comic that
callid pointy-head people its take off on certain strips that represent managers

dont want come and look over guys shoulder

way To be good manager you really
to

every two seconds and say Hey are you done yet eally doesnt help them It

the
adds to the tension Uh the truth is my job is to try to shield the pressure so go to

meetings and Im going to sit through and take the flack about us not being in and us not

whats going on and allow them to focus as much as possible on

being ready explain

done So thats-thats and have to sort of weigh this delicately and

getting the job my job

In
be nice guy as come around asking without interrupting being curteous to system

fact you know my job bring cookies bring water Uh if they have problems and they

want me Ill sit down and look at code with them as well uh It varies so its just

moving along wthei4-ache.4 and time you never know when you have the last bug

You know when the last on the main is about hope it all works And
never thing system
VS
good does time again-is immemorial here
the truth is if youre doing it
really

Thats it And it
really does work Okay that was the problem and were good to go but

know uh you get good reinforcement when then

you dont this and its happens
it
quite

you hit your head on another one and youre going Oh darn its not over you know

the marathon hasnt ended could see the-the light at the end of the tunnel but
thought

it..

Page 14
 SUM4UUJ
1301 p%wwox wg