Sie sind auf Seite 1von 153

JMP105 How Stuff Works Domino Style!

Jess Stratton
Susan Bulloch

| Independent Consultant | Solace


| Support SWAT Engineer | IBM

2011 IBM Corporation

Who are Jess and Susan?

Susan Bulloch

15+ years working with IBM Lotus Domino and Notes and related products
12 years with IBM
Support SWAT Engineer
High touch, often on-site analysis/repair work
Technote/whitepaper/wiki article author
Certifications
Part time blogger notesgoddess.net

Jess Stratton

10+ years working with IBM Lotus Domino and Notes as a developer and
administrator
Blogger, tweeter (@NerdGirlJess)
Written technical articles and been a contributing author for textbooks
Currently an independent consultant!

2011 IBM Corporation

What are we learning, here, today?

We are learning how the basic principles of how many modules of an IBM Lotus
Domino Server work.

This presentation is modeled after the brilliant example of HowStuffWorks.com

From HowStuffWorks.com:
"Our premise is simple: Demystify the world and do it in a simple, clear-cut
way that anyone can understand."

2011 IBM Corporation

How HowStuffWorks.com works...

2011 IBM Corporation

Agenda
How Notes Named
Networks work

How Replication works

How Cross-Certifying works

How Connection
Documents work

How Clustering works

How View Indexes work

How Mail Routing works

How Busytime works

How Transaction Logging


works

How Domino Domains work

How Agent Manager works

How Program Documents


work

How SMTP works

How AdminP works

How Smart Upgrade works

How Access Control Lists


work

How ID Files work

How Readers fields work

How Recertifying works

2011 IBM Corporation

How Notes Named Networks Work

IN THIS SECTION

Introduction to Notes Named Networks

Routing and Notes Named Networks

Users and Notes Named Networks

Setting the Notes Named Network

2011 IBM Corporation

Introduction to Notes Named Networks

In Domino, the Notes Named Network (NNN) that a server is on defines what
other Domino servers it can see.

Each of those protocols runs on a different port.

Each server on the same NNN is running the same LAN protocol.
TCP, SPX, NetBEUI, etc.

The available ports to choose from is taken from the Ports list in the server's
Notes.ini.
For example, cluster servers are frequently connected via dedicated cable and
running on their own NNN.

If a Domino server is on a different port, running a different protocol, it cannot


be on the same NNN.

How would the servers know how to talk to each other?

2011 IBM Corporation

Introduction, cont.

Domino servers can have multiple NNN's.

While other uses are possible, the primary usage for having Notes Named
Networks is for mail routing scenarios.

2011 IBM Corporation

Routing and Notes Named Networks

It's simple: Domino servers running on the same NNN can route mail without
the need for connection docs.

Remember they are running on the SAME port and protocol.


There still has to be all that DNS magic behind the scenes to make sure they
really CAN talk to each other, though!

Domino servers running on different NNN's can still route mail, but connection
documents must be in place.

Because they are running different network protocols, the connection document will
tell the servers how to find each other.
They can use a bridge to find each other:
An intermediary Domino server running a second matching NNN.

2011 IBM Corporation

Users and Notes Named Networks

When users open a database and browse Other in the server list, the list of
server names that will automatically show up are those servers in the same
NNN as the user's home server!

When users find a replica of a database, it will automatically find replicas of


databases also based on servers in the same NNN as the user's home server.

2011 IBM Corporation

10

Setting the Notes Named Network

Make sure the port is first set up and available in Notes.ini of all servers that will
use it.

In the Server doc, go to the Ports-->Notes Network Ports tab.

Enter the port name. Domino will automatically set the protocol.

Enter the Notes Network name.

Enter the net address of the port.

Best to put in the protocol and what it's for, ie. TCPIP Cluster
This MUST be reachable by other servers that will use the port for routing!

Finally, enable the port.

2011 IBM Corporation

11

How Connection Documents Work

INSIDE THIS SECTION

Intro to Connection Documents

Keeping it Simple

Tips and Gotchas

2011 IBM Corporation

12

Intro to Connection Documents

Connection documents provide servers with the information necessary to


connect to other servers for mail routing and replication purposes.

A connection document provides two types of information

For Mail Routing, two documents are required

Network information
Defines which server to connect to
How that connection is made and what protocol to use
Schedule information.
Defines when and how often activities are performed
Replication and routing to a particular server
One for each server
Unless the servers are in the same Notes Named Network (NNN)

For Replication only one document is required

And it can be one of the mail routing documents

2011 IBM Corporation

13

Keeping it Simple

Servers in different NNNs only route mail on schedule

As defined in the Connection document


Plan accordingly

Keep the number of connection documents between servers to a minimum

Design your mail routing topology so that servers are grouped into Notes
Named Networks for routing without connection documents

Implement a hub-and-spoke replication topology to keep connections simple


This will help you avoid nasty troubleshooting problems

2011 IBM Corporation

14

Tips and Gotchas

Most code that needs information in Connection docs looks to the same place

A hidden view called $Connections


If the information in this view is stale, stuff won't work
Like mail routing!

Never, ever mess with the design of the views in your Directory
Bad things can happen

Make sure the View Indexes stay current

2011 IBM Corporation

15

How Mail Routing Works

INSIDE THIS SECTION

Introduction to mail routing

Basic Mail Routing

The Routing Table

Routing Mail Efficiently

Transfer Queues and mail.box

2011 IBM Corporation

16

Intro to Mail Routing

Mail Routing for Lotus Notes uses the server task Router

All mail goes first through a mail.box

Recipient is on another server

Router Performs Message Delivery

Clients can deposit mail there


Other servers can deposit mail there

Router performs Message Transfer

Lotus Notes can transfer NRPC (Lotus Notes) and SMTP (Internet) mail
We'll discuss NRPC mail
The router creates and controls the mail.box(es)
Uses them to process messages

Recipient is on the same server as the mail.box

Router depends on a routing table

It tells where to send the mail

2011 IBM Corporation

17

Basic Mail Routing

The Router task initializes

For example, at server startup


After a restart command

Router builds the routing table,

Router determines the maximum number of threads to allocate

Router checks MAIL.BOX.

If the Router finds new or changed messages in mail.box


Its main thread builds the main message queue
The message queue contains
An entry (a list item) for each new or modified message
Summary information about the message

Recipients, destination, date sent, etc.

After building the message queue, the main thread dispatches each message in
the queue

2011 IBM Corporation

18

The Routing Table

Routing tables

Use Connection documents, Domain documents, and Server documents in the


Domino Directory,
Settings in the server notes.ini file

Router builds the routing table and determines which servers are connected

The Router periodically checks certain views in the Domino Directory for edits
and additions to the documents

$Connections, $Domains, and $Servers

The Router does not pick up a change until one of the views changes

If you change a value in the Server document, the Router is not aware of it until the
view refreshes
View indexing is important!

2011 IBM Corporation

19

The Routing Table (con't)

By default the router polls for changes every 5 minutes

Router basically builds a graph

Of all servers in the domain


Plus all servers that use Adjacent domain documents to connect to other domains
In a format that ensures fast lookups.

The Router tries to check the views before dispatching each message.

Use "Tell router update config" to process changes immediately


Or restart the router task
Global Domain Document settings are changed only on server restart

If it finds changes, it reloads the routing table.

To view the routing table

tell router dump routingtables


Since Version 8.5.1

2011 IBM Corporation

20

Routing Mail Efficiently

When Router finds users that are on the same server as the mail.box holding
the message

When Servers are in the same Notes Named Network (NNN), mail is
transferred to the server without using connection records

If a large message is being delivered, the opportunity for a logjam occurs

If there are multiple MAIL.BOX databases on the destination server

Remember this when planning connections

When the transfer thread connects to MAIL.BOX on a server, it performs a


NoteUpdate that temporarily locks MAIL.BOX

It marks the message for local delivery


Delivers the message to one or more mail files on the server

A transfer thread can move to an available MAIL.BOX if one MAIL.BOX is locked

Multiple transfer threads and multiple MAIL.BOX databases on the receiving


server result in faster processing of messages for that destination

2011 IBM Corporation

21

Transfer Queues and mail.box

Message Transfer queues are in-memory queues that move a message from
one server to another.

Each transfer thread handles a single message for a single destination, but deals
with all recipients of that message at that destination
The sending server tries to transfer as many pending messages to a destination as
possible over an open connection for maximum efficiency

Sending servers only use multiple threads to a single destination if the


connection is fast enough

Use Tell router show to see the transfer queues

The console displays the destination, number of messages pending, and state for
each active transfer queue.
You can sometimes see your slow connections!

2011 IBM Corporation

22

How Domino domains work

IN THIS SECTION:

What is a Domino domain used for?

What do servers in a domain share?

What are the types of Domino domains?

2011 IBM Corporation

23

What is a Domino domain used for?

A Domino domain is a group of Domino servers that share a Domino Directory

A Domino domain can be used to control several things:

names.nsf

Server access
Domino server groups
Clustering
Mail routing

A Domino domain is specified at the end of a user's common name, org units,
and certifier.

Jessica Stratton/Qualinesti/Solace@Solace

2011 IBM Corporation

24

What do Domino servers in a domain share?

Busytime

The Domino Directory

Or clubusy.nsf if they are clustered (more on that later!)


names.nsf

The Administration Process

adminp.nsf

Monitoring

The Certification Log

These databases need replicas of each other on each server in the domain!

2011 IBM Corporation

25

What are the types of Domino domains?

Adjacent Domain document

Non-adjacent Domain document

These are used when a bridge Domino domain is used to connect two Domino
domains not connected via Connection documents.
These can be used for next-hop instructions, and also restriction/allow rules.
Calendar servers can be specified here for free-time lookups between domains

Foreign Domain document

These are most commonly used to restrict or allow mail to be sent to/from Domino
servers through Domino domains.

These are used to specify a routing link between Domino and a non-Domino
application, such as a fax gateway.

Foreign SMTP Domain document

Use these when a Domino server does not use SMTP and needs to locate a nexthop server for DNS lookups to route outbound Internet mail.

2011 IBM Corporation

26

How SMTP Works

IN THIS SECTION

Introduction to SMTP

Configuring an Internet domain for SMTP

Configuring Domino for SMTP

What happens when a message is sent

What happens when a message is received

2011 IBM Corporation

27

Introduction to SMTP

SMTP stands for Simple Mail Transfer Protocol.

In its simplest form, mail is sent from one server directly to another over port 25.

2011 IBM Corporation

28

Configuring an Internet domain for SMTP

A domain name, once registered, needs certain records placed in the Domain
Name System (DNS).

An A record, or host record, tells DNS what IP address is matched to your domain
over port 80, to get to a website.
An MX record, or mail exchange record, tells DNS what IP address is matched to
your domain over port 25 for the SMTP protocol.

These records do not have to go to the same server.

So basically, whatever IP address you set your MX record to, MUST have an
SMTP-enabled server that is listening on port 25.

Which, fortunately, Domino is!

2011 IBM Corporation

29

Configuring Domino for SMTP

Domino must be configured to SEND mail (the Router task), and RECEIVE mail
(the SMTP listener task).

Domino must have a Mail.box, or multiple Mail.boxes.

To SEND mail:

Make sure DNS is set up correctly from Domino


Enable SMTP to be used outside the local Internet domain (Config doc)
Specify a relay host when sending message outside the local domain (Config doc)
Set up mail relay restrictions to avoid spammers (Config doc)
Set up outbound email address format (Domain doc)

To RECEIVE mail:

Enable the SMTP listener task (Server doc)


Populate your Domino Directory with usernames and Lotus Notes as their mail
system.
Each name must have an Internet address specified, and a physical .nsf
mailbox
2011 IBM Corporation

30

What happens when a message is sent

Domino checks the message format for a <localpart>@<domain> format, which


is how it knows how to look up the domain name over the Internet through
DNS, rather than routing it internally.

The Router task looks up the least-cost route to a Domino server that has
SMTP allowed for all messages.

If enabled, the Router task converts the message to MIME format.

DNS looks up the MX record for the <@domain>, which returns an IP address.

Domino then initiates a conversation with the server at that IP address, over
port 25.

Domino says, OK, now I know where to deposit the email.

It connects to the server, and deposits the message.

The connection is closed, as the transaction is now over.

2011 IBM Corporation

31

A conversation between two servers...

The sending server's conversation with the receiving server is via a shared
language.

A text line is sent, and a reply is sent by the receiving server.


This happens with EVERY outbound email through Domino!

2011 IBM Corporation

32

What happens when a message is received

The SMTP listener task listened, and had a conversation with an incoming mail
server through the Router task.

The Router picks up the incoming message and deposits it into Domino's
mail.box

The Router looks up the incoming address against the $Users view in the
Domino Directory.

If it finds a match (and there are no restrictions against the address in the
Configuration document) it looks up the user's home server in their Person
document.

The routing tables finds the least-cost path to deposit the message in the
mail.box of the user's home server.

Finally, the home server's Router task looks up their mail file in their Person
document, and if the server has sufficient access in the ACL, deposits the
message into the database.

2011 IBM Corporation

33

How ACLs Work

IN THIS SECTION

Introduction to the access control list

The access levels and user types

Roles and readers fields

Enforcing a consistent access control list

The maximum Internet name and password

Understanding -Default-

2011 IBM Corporation

34

Introduction to the Access Control List

The access control list is a property of EVERY Lotus Notes database that
controls who can read and write documents, change the design, and manage
the physical database.

Because every aspect of Domino Administration is database-driven, the ACL is


the first step in troubleshooting any issue.

The Administration Process is a database, admin4.nsf


The Domino Directory itself is a database, names.nsf
Replication success relies on the servers having the proper access
Busytime/calendaring and scheduling relies on being able to access user calendars

2011 IBM Corporation

35

The access levels and user types

There are 7 access levels of a Lotus Notes database.

Each level higher gets all the rights of the levels below it.
A user can never be granted more rights in a database than what's been given to
them in the ACL, no matter how the database is designed or what the Domino
Directory says about them.
There are switches to fine-tune each access level, also called privileges.

2011 IBM Corporation

36

The seven access levels:

NO ACCESS

DEPOSITOR

You can only create documents into a database, you cannot read them.

READER

You cannot access the database. Period!

You can only read documents in a database, you cannot deposit them.

AUTHOR

You can create documents, but read only documents you've created.
You can read documents that you are listed in an Author's field.
You may not be able to create documents, but you can edit any document in which
you are listed in an Author's field.
Creating documents is a privilege for the Author access level.

2011 IBM Corporation

37

The seven access levels (cont)

EDITOR

DESIGNER

Can create and edit all documents.


Can delete documents if given fine-tuned permission.
Can create design elements, including LotusScript agents.
Designers can also create shared agents.

MANAGER

Can delete the physical database.


Can add/edit the database access control list

2011 IBM Corporation

38

The five user types

There are 5 user types, in addition to the access levels.

These are so Domino knows what ID file type it should be expecting to go along
with a name/Certifier combination.

Leaving a user or group as Unspecified in the ACL will make it possible for a
user to create a local group titled as a person's name to gain higher access.

2011 IBM Corporation

39

The five user types (cont)

PERSON & PERSON GROUP

SERVER & SERVER GROUP

The server knows what groups you are in before you even try to access a database.
If you have a local group that is the same name as a person, you won't be able to
access the database by that group name.
This settings can lock down databases who should only be accessed via back-end
processes, as only an ID of type Server, or a group whose membership is only
servers can access it this way.

MIXED GROUP

Users, groups, and servers can have access, as long as it is one of those three.
This allows you to cover your bases, and not leave an entry as Unspecified.

2011 IBM Corporation

40

Roles and Readers fields

Roles are a sub-group that can only be used in the scope of the database it's
listed in.

Roles can be added to a database for form/document development.

Hide/Whens can be tested on Role membership.


Readers fields can be populated by Role membership.

First you create the role, then you assign ACL entries to the role.

You can add servers to roles, also!


In fact, this is crucial for Readers fields!

2011 IBM Corporation

41

Enforcing a consistent access control list

With the option Enforce a consistent access control list selected, a local
database replica's ACL can NEVER change a person to higher rights than they
have on the server replica.

If you TRY and change the ACL locally to give yourself higher rights, all replication to
and fro the database will stop!
This prevents local ACL changes from propagating back up to servers and being
replicated around an organization.

Also, Readers fields will not be honored on local databases UNLESS this value
is checked.

2011 IBM Corporation

42

The maximum Internet name and password

This option was created as a security feature to ward off potential database
pitfalls over the web.

What does a user need to do to a database while accessing it over the web that
would require Designer or Manager access?
Even background agents online use server access to run.
Why open a can of worms?

2011 IBM Corporation

43

Understanding -Default

Any user that accesses a database and is NOT assigned an access level will
get what you have specified as -Default- access.

Setting it to No Access will not let anyone in if they are not explicitly listed in the
ACL as either a person or as group membership.
Setting it to Reader will allow anyone who access it to read the contents, rather
than putting in an entire organization's group membership in the ACL.
This will vary depending on the database!

Use Anonymous to cover any Internet user who has not authenticated.

There may be a reason for this to be different than Default access.


If you don't use Anonymous, Default is used.

2011 IBM Corporation

44

How Readers Fields Work

INSIDE THIS SECTION:

Introduction to Readers fields

When would you use Readers fields?

Readers fields vs. other security measures

How do you implement Readers fields?

What are some common gotcha's with Readers fields?

2011 IBM Corporation

45

Introduction to Readers fields

Readers fields are a field type that explicitly states who can read documents
created from the form the field is currently located on.

Some other field types are:


Text
Date/Time
Number
...etc.

The contents of a Readers field can be Usernames, Groups, or Roles that have
been created in the current database's ACL.
2011 IBM Corporation

46

When would you use Readers fields?

Readers fields are used when you need to hide documents for certain users.

If a user is not listed in the Readers field, then to them, the document doesn't exist in
the database!
It is, however, counted in the Document Count field in Database Properties.

Readers fields are used when you need TRUE document security in Lotus
Notes.

There is no backdoor access, unless you are listed as a Full Access Administrator.
There is no changing to another view.
Well, you CAN, but the document won't be visible in that view, either.
There is no right-clicking on Document Properties to see field values.
Why? Oh yeah there is no document!
If your ID file Username is not listed in a Reader field, you don't even know about the
document!

2011 IBM Corporation

47

Readers fields vs other security measures

Controlled-Access sections

View Access

Just because a user can't access a VIEW does not mean they can't access a
document.

Hide/When field formulas

These use ACL and Group memberships to control who can expand a section on a
document.

This data is still easily viewable by the Fields tab in document properties!

Readers fields

These are considered true security measures as it trumps the database ACL.
A user can be listed as a Manager in the ACL and not see documents if they are
not listed in the Readers field on a document.
We will cover this in the gotcha's section!
It will only taketh away; you cannot grant a No Access user access by listing
them in a Readers field.

2011 IBM Corporation

48

How do you implement Readers fields?

Create a new field on a form and give it a field type of Readers.

Make sure it is a multi-value field.

Add allowable readers of the document using any of the following, separated by
a :

Roles in the ACL (recommended)


Explicitly listed user names in Canonical format
Groups
Other field names on the same form containing resolvable names.
Some examples:
Administrators:CN=Jessica Stratton/O=Solace:[HR]

2011 IBM Corporation

49

What are some common gotcha's of Readers


fields?

HELP! I'm the administrator and I locked myself out of all the documents!

Always create a role in the ACL called [Admins] and give the following items the role:
The database admins
LocalDomainServers
The Administration server
This role should be your first entry in the Readers field list!
You can also take Full Access Administrator to grant yourself access.

The documents aren't replicating to other servers, even though I can see
them all!

See item #1.


All servers need to be listed as allowable readers too, otherwise the documents are
invisible to them, too.

2011 IBM Corporation

50

Common gotcha's (cont.)

Abbreviated names will not work. Canonical is best!

GOOD: CN=Jessica Stratton/O=Solace, Jessica Stratton


BAD: Jessica Stratton/Solace

Readers fields aren't being honored in local replicas!

Enforce a consistent ACL must be enabled in the ACL in the advanced properties
tab.

2011 IBM Corporation

51

How Replication Works

INSIDE THIS SECTION

Introduction to replication

Preparing for a replication infrastructure

Implementing replication

Ways to initiate replication

The science behind replication

Deletion stubs and replication

2011 IBM Corporation

52

Introduction to replication

A database replica is a special copy of a database on another server, or local


workstation.

It shares a common property called the replica ID that even an OS filecopy will
retain.
The only way to break the replica is by creating a new copy instead. This will
issue a new replica ID to the database.
Databases can have different titles, filenames, even a different design template.
But if they share a replica ID, they can replicate!

2011 IBM Corporation

53

Introduction to replication (cont)

When change is made in a database, the change can be propagated to any


other replicas of the same database.

So any user on any server who needs an up-to-date version will always have it.

The changes are propagated most often by Connection documents in the


Domino Directory between servers.

You can schedule the connection documents on any interval, and even have multiple
schedules according to database priority.
In the database replication settings, you can specify a subset of documents to
replicate, say, for a particular physical work location.

2011 IBM Corporation

54

Introduction to replication (cont.)

If you have multiple servers, they need to replicate common databases even if
you have no production databases that are used by users.

Remember: servers need to replicate, even if users don't!

Servers in the same domain share:

Names.nsf (The Domino Directory)


Adminp.nsf (The Administration Process)
Certlog.nsf (The Certification Log)
... more.
These databases need to stay in sync!

2011 IBM Corporation

55

Preparing for a replication infrastructure

Make sure both servers have a level of access at least as high as they need to
each databases they need to replicate.

If any form in the database that will replicate has Readers fields, make sure the
servers are listed in those fields.

The Replicator server task should be listed in the Notes.ini file under
ServerTasks.

This task is enabled by default!

Create connection documents, specifying:

What database will replicate


When they will replicate, and how often the replication interval will be
Which server initiates the replication

2011 IBM Corporation

56

Preparing, cont.

The following are types of replication in the connection document:

Pull-Push
One server first pulls changes, then pushes its changes to the second.

Pull-Pull
One server pulls the changes, then the other server pulls the changes.

Push-Only
One way replication in which the initiating server pushes changes to the second.

Pull-Only
One way replication in which the initiating servers pulls changes from the
second.

2011 IBM Corporation

57

Table of server access levels for replication:


(from Notes Help)

Access level

Allows a server to push these changes

Assign to

Manager

ACL settings
Database encryption settings
Replication settings
All elements allowed by lower access levels

Servers you want to use as a source for ACL


changes. For tight database security, give this
access to as few servers as possible. In a huband-spoke server configuration, you typically give
the hub server Manager access.

Designer

Design elements
All elements allowed by lower access levels

Servers you want to use as the source for design


changes. Use Manager access instead if you want
one server to control ACL and design changes.

Editor

All new documents


All changes to documents

Servers that users use only to add and modify


documents. In a hub-and-spoke configuration, you
typically give the spoke servers Editor access.

Author

New documents

No servers. You don't typically use this access for


servers.

Reader

No changes; server can only pull changes

Servers that should never make changes. Servers


in the OtherDomainServers group are often given
Reader access.

Depositor

New documents. Also prevents the server from


pulling changes.

No servers. You don't typically use this access for


servers.

No Access

No changes. Also prevents the server from pulling Servers to which you want to deny access.
changes.
Servers in the OtherDomainServers group are
sometimes given No Access.
2011 IBM Corporation

58

Implementing Replication

After database ACLs are set, Connection docs are created, and the replica task
is running, replicas must be created on the second server.

What are some ways to get replicas on the second server?

Manually
File-->Replication-->New Replica
OS copy/paste
Windows Explorer via mapped drive
Or, USB stick into the Data directory of the second server
From the Administrator Client
Go to the Files tab
Highlight all databases to be copied
Select Database-->Create Replica(s) from the right Tools pane.

2011 IBM Corporation

59

Ways To Initiate Replication

Manually at the Domino console

Some server console command examples:


Replicate Server1/Acme
Replicate Server1/Acme names.nsf
Push Server1/Acme names.nsf
Pull Server1/Acme

From the database icon

Replication-->Replicate

From scheduled Connection documents

From Program documents

2011 IBM Corporation

60

The Science Behind Replication

It starts when scheduled replication begins, and the Replicator task springs into
action!

Then, each server makes sure they can authenticate with each other through
matching, signed certificates.

The initiating server searches the second server for databases.

It constructs a list of databases that contains matching replica IDs on each server.

The initiating server looks at the replication history to find the last time the two
replica databases replicated.

The initiating server looks at its own database for changes since that last time.

The Replicator task performs a special search on the database.


The search results are a list, containing the Originator IDs of all the documents that
have been modified.
The Originator ID is a long string containing the doc UNID, a sequence number
(how many times the doc has been modified), and a timestamp when it was last
modified.
2011 IBM Corporation

61

The Science (cont)

The initiating server's Replicator task then performs another search on the
second server, using the UNIDs from the modified docs of the initiating server.

It compiles a list of the Originator IDs, just like it did from the first server.

Now that the Replicator task has two OIDs for the same document on each
server, it compares them.

If it can't find a matching OID for a document, then it is a new document and must be
pushed to the server that doesn't have it.
If the OIDs are the same, no changes have been made to either document.
If the OIDs are different, the Replicator task looks at the contents of the $Revisions
field on each document.
(The $Revisions field gets updated during replication so it will be cleared for
next time)
The revision history tells the Replicator task whether a conflict exists.

2011 IBM Corporation

62

The Science (cont)

If a conflict exists, the Replicator task checks for a $ConflictAction field.

If $ConflictAction contains 1, then the database properties has Merge replication


conflicts set and the changes can be merged successfully.
HOWEVER If the sequence number tells the task that the same field has
been changed, a merge can not occur no matter what, and a replication conflict
will still occur!
The document with the most recent time stamp in its sequence number is the
winner. The other document is the loser and is the conflicting document.
The same tests occur during replication for design elements, but the loser's
changes are simply removed, there are no conflict documents created.

2011 IBM Corporation

63

Deletion Stubs and Replication

Deletion stubs remain in a database after a document has been deleted.

It's to let the Replicator task know to also remove the document from a replica
database.
As long as Deletions is checked off in the Remove incoming: field in
Replication settings.

You can specify how long deletion stubs will remain in the database before they
are purged.

If the database replicates AFTER the deletion stub is gone, the Replicator task will
not know to delete the document from the second server.
Worse, it will think the still-existing document in the second database is a new
document and replicate it back over again.
So now, the first database will have the deleted document re-appear.

2011 IBM Corporation

64

How Clustering Works

INSIDE THIS SECTION

Introduction to clustering

Cluster Manager

Other Clustering Tasks

The Cluster Replicator Task

Tweaking CLREPL

How all this Cluster Stuff Works

2011 IBM Corporation

65

Intro to Clustering

A cluster is a group of two or more servers

That provides users with constant access to data


Balances the workload between servers
Improves server performance

There are several components that make clustering function

The Cluster Manager


The Cluster Database Directory
The Cluster Database Directory Manager
The Cluster Administrator
The Cluster Replicator
AdminP

2011 IBM Corporation

66

Cluster Manager

Cluster Manager runs on each server in a cluster

When it starts, the Cluster Manager


Looks in the ClusterName field in its Server document for its cluster name
Then looks at the cluster membership list for the names of the other servers
It maintains this information in memory in the server's cluster name cache

Each Cluster Manager exchanges messages (probes) with the other servers in
the cluster

Cluster Manager uses the NSPingServer command to probe the other clustermates
The Cluster Manager issues the NSPingServer command once every minute by
default
Change with Server_Cluster_Probe_Timeout NOTES.INI setting

2011 IBM Corporation

67

Cluster Manager

The cluster name cache contains the following information:

When a session is closed (server or client), this is saved to cluster.ncf

The name of the cluster


The names of the servers in the cluster
The server availability index of each cluster server, sorted by the most available first
The state of the server if it is BUSY, MAXUSERS, or RESTRICTED
The cluster probe timeout value
This file populates the cache initially on restart
This file has been known to contain erroneous data

To see what's in the cluster name cache, type show cluster at the console

Or open with a text editor

2011 IBM Corporation

68

Other Clustering Tasks

The Cluster Database Directory (CLDBDIR.NSF) is on each server in a cluster

The Cluster Database Directory Manager task (Cldbdir)

Creates the Cluster Database Directory and keeps it up to date


But sometimes the database has stale data
Note that the first hint you have of this is the Resource Reservations system
stops working!

Since version 8.5, cldbdir runs automatically on clustered servers

It contains a document for each database and replica in the cluster.


Contains the database name, server, path, and replica ID, and other info

All earlier versions need this in servertasks

The Cluster Administrator

Does the housekeeping


Adds/removes databases and clustermates

2011 IBM Corporation

69

The Cluster Replicator task

The Cluster Replicator task (clrepl) keeps the data in sync within the cluster

Whenever a change occurs clrepl immediately pushes the change to the other
replicas in the cluster
By default, one clrepl runs per server, but you'll want more

The Cluster Replicator gets it information from CLDBDIR.NSF

If that info is wrong, clrepl has a problem

The Cluster Replicator task pushes changes only to servers in the cluster

Cluster_Replicators=

The standard Replicator replicates changes to and from servers outside the cluster

This is high traffic stuff, so use a dedicated LAN connection for this traffic

Your users will thank you


Or they just won't hurt you

2011 IBM Corporation

70

Tweaking CLREPL

Pausing Cluster Replication

When you pause cluster replication vs stopping it, it retains the data it has in memory
Use tell clrepl pause/resume

By default, clrepl polls for changes every 15 seconds

If you feel your servers can handle it, try this


CLREPL_POLL_INTERVAL=10

2011 IBM Corporation

71

How All this Cluster Stuff Works

With all clustermates up and running correctly, nothing happens

When one server goes down

Mail routing uses the information in the cache to determine which clustermate gets
the mail meant for that server

When a user requests something from a server that's down

The local cluster.ncf data is used from the cache to find the next available server

2011 IBM Corporation

72

How Busytime Works

INSIDE THIS SECTION

Intro to Busytime

How it works

Where it looks

Clustering Busytime

Doing a Free Time Search

Tips and Gotchas

2011 IBM Corporation

73

Intro to Busytime

The busytime system is responsible for tracking the availability of people and
resources

Two tasks populate busytime


Schedule Manager (Sched) and Rooms and Resources Manager (RnRMgr)
One task communicates between servers (calconn)

If busytime ain't happy, nobody's happy

Users, Rooms, Resources, all depend on busytime being correct

2011 IBM Corporation

74

Intro to Busytime (con't)

When not in a cluster

Each server contains a Free Time database (BUSYTIME.NSF)


It includes scheduling information for all users and resource databases (RRDB) on
that server

In a cluster

There is one Free Time database for all users and RRDBs whose servers are in the
cluster (CLUBUSY.NSF)
Every server in the cluster contains a replica of this database
Since the replicaID must be unique to the cluster, its not like other system databases
The replicaID is not taken from the Directory
Its taken from the CLREPLID field in the server documents

2011 IBM Corporation

75

How it Works

Schedule Manager

Scans and processes any database that has a calendar profile and a calendar
It does this at 2 AM or when the server is restarted
Uses fields in Person documents to determine where the database is located
(which server) and the databases file name
If schedule manager finds a database with a valid calendar profile and a calendar,
It creates a record in busytime.nsf
When databases have calendar entries, Sched stores them in busytime.nsf and, as
appropriate, marks time as busy
If a database exists and has no matching person document, Sched lets you know it
SchedMgr: Error processing calendar profile document (NoteID: NT00000926)
in database mail\ABCDEFG.nsf: Cannot find user in Domino Directory
This means that the calendar profile was not valid
This user will not go into busytime.nsf

2011 IBM Corporation

76

How it Works

Rooms and Resources Manager

Runs first if both Sched and RnRMgr are on a server


RnRMgr looks for any database with a Room/Resource profile document
But it looks at all databases with a profile document along the way
Also validates the profile, creates a record in busytime.nsf and marks times as busy
RnRMgr uses fields in the Resource documents in the Directory to validate against
SchedMgr: Error processing calendar profile document (NoteID: NT001568EA)
in database resource.nsf: Cannot find user in Domino Directory
This means that no resource document exists in the Directory for this resource

That's REALLY bad

2011 IBM Corporation

77

Where it Looks

Validation of Users and Resources is done in one place

$Users View in the Domino Directory


This must be kept up to date
Good place to start when troubleshooting!

Certain fields are required in order to add users/resources or free time

If any of the fields are incorrect, errors will occur


For Sched In Person Documents
User Name Field: FullName = $busyname in the calendar profile of the mail
database Canonical format
MailServer Field = server the busytime task is running on
MailFile = correct mail file for the user
CalendarDomain (Miscellaneous tab): Used if user is on a non-Notes calendar
system
If this field is used, sched will not list the person in the busytime.nsf.
ForwardingAddress: If used, user is not listed in busytime

2011 IBM Corporation

78

Where it Looks (con't)

RnRMgr looks in $Users view also

Searches for Resource forms


The field names are the same
FullName = Resource or Room Name
Must be canonical and match the value on the Resource Document in the
RRDB
MailServer = Home server of the RRDB
MailFile = Name of the RRDB (with path)

Manually editing Resource documents is a major cause for problems

Be vewwy careful if you must do this


Spaces are hard to see

2011 IBM Corporation

79

Clustering Busytime

When you add a server to the cluster,

Sched deletes BUSYTIME.NSF and creates CLUBUSY.NSF


When removing a server from the cluster, it does the reverse
It then replicates with the other servers in the cluster

Rebuilding clubusy.nsf if very difficult to do right

And when needed, it must be done right


Check the create date on all copies in the cluster

There should be no duplicate entries

There should be no rep/save conflicts

2011 IBM Corporation

80

Doing a Free Time Search

When a user searches for free time for another user or room

When a user in a cluster looks for free time

The user's home server looks at the $Users view for the requested User/Room
Determines whether to look in its own BUSYTIME.NSF or ask CalConn to go fetch
from another server
Or whether to pass the request off to an outside system
But that's beyond the scope of this session
The server looks in its own CLUBUSY.NSF first if other users are on clustermates
For users whose mail servers are outside the cluster, CalConn sends out for the
information

When a user outside a cluster makes a request about a user in a cluster

The request fails over to another server in the cluster if the user's mail server is
unavailable

2011 IBM Corporation

81

Tips and Gotchas

Easy troubleshooting tip

If theres a gray line where there


should be information

Touch that gray line with your mouse


and look at the bottom of your
screen

The message usually tells you


whats wrong

In this case, the system couldnt find


Jess

2011 IBM Corporation

82

Tips and Gotchas

RnRMgr and Sched respond to the same Tell commands

Tell Sched Val does nothing useful

Tell Sched Show User Name/OU/O

Unless followed immediately with Tell Sched Check


Dumps the information the server has about the user's appointments
This may not match calendar

Tell Sched List User Name/OU/O

Shows what the server thinks is the user's preferences


Again, this may not match reality

2011 IBM Corporation

83

How Agent Manager Works

INSIDE THIS SECTION

Intro to Agent Manager

Queues and Rights

When it Works

Keep an Eye on Stuff

Tips and Gotchas

2011 IBM Corporation

84

Intro to Agent Manager

Agent Manager is the server task that manages and runs agents on a server.

It determines who can run agents,


Where they should run
When they should run
It handles Scheduled and Event-driven agents (there are other kinds)

It sets up the context for the agents to run in,

It checks security,
Handles loading, invocation, caching, error reporting, and logging

The Agent Manager runs by default on a server

Depending on the type of agent

Agent Manager may perform database operations for simple actions, evaluate
formulas, or call back-end classes
For event-triggered agents, the Agent Manager monitors events and then
determines when the agents should be invoked
It also watches the clock and invokes scheduled agents when the right time comes.

2011 IBM Corporation

85

Queues and Rights

The Agent Manager has three different queues

Scheduled and Event-triggered agents stay in their queue until its time to run

One queue for agents that are eligible to run (E)


Another queue for agents that are scheduled to run (S)
A third for event-triggered agents that are waiting for their event to occur (V)
Then they move to the eligible to run to wait for their turn

The creator or signer of the agent must have the rights to run on the server and
the database

Rights are set in the Server document as well as the database ACLs

2011 IBM Corporation

86

When it Works

Agent Manager periodically checks to see if it has any new agents that it needs
to schedule.

Two environment variables control how quickly an agent gets into the queue:
AMgr_SchedulingInterval specifies a delay (in minutes) between running of the
Agent Manager's scheduler.

AMgr_UntriggeredMailInterval specifies a delay (in minutes) between running of


the Agent Manager's check for untriggered mail.

Valid values: one minute to 60 minutes. The default value is one


minute.

Mail is "untriggered" when it's added to a database through


replication, which does not cause a new mail trigger to fire.
Valid values: one minute to 1440 minutes
The default value is 60 minutes.

The variables are tunable depending on your needs

2011 IBM Corporation

87

When it Works

Scheduled agents are executed in the order in which they are found in the
queue by Agent Manager.

If more than one agent in a database is scheduled to run at the same time
The one that is saved and enabled first will be more likely to go into the queue
first and therefore be executed first

Agent Manager doesn't run ALL agents

Router does some (before new mail arrives)


Server does some (run on server)
The HTTP task does those invoked by a browser

2011 IBM Corporation

88

Keep an Eye on Stuff

Agent writers can get a little ambitious

Poorly written agents have been know to bring down servers or lock up CPUs
There's an easy way to watch for these
Use Domino Domain Monitoring
Use Tell Commands to get more information

Look at the queues using the Administrator client

Run more than one instance if queues back up

2011 IBM Corporation

89

Tips and Gotchas

Most Agent Manager problems are really problems with bad agents

Or agents without the proper rights


Use Tell Commands for quick results
Tell amgr sched

E S 04:03 PM Today agent1 CENTRAL.NSF


S S 05:04 PM Today agent2 CENTRAL.NSF
V U agent3 CENTRAL.NSF

Enable Agent logging if having problems

Log_AgentManager=value
0 - Do not log agent execution events
1 - Log agent execution events (partially and completely successful)
2 - Log agent execution events (completely successful only)

2011 IBM Corporation

90

Tips and Gotchas

Enable Debug (short term) for more info

Debug_AMgr=*
Have lots of time to read the output!
See technote on the Stuff page for less info

Study and use the Tell Commands to make stuff work better

Cancel, Run, debug, Load

2011 IBM Corporation

91

How AdminP Works

INSIDE THIS SECTION

Introduction to AdminP

How AdminP requests are created

How AdminP requests are processed

How AdminP Works

Example AdminP Request

Tips and Gotchas - AdminP

2011 IBM Corporation

92

Introduction to AdminP

AdminP is a server task that automates stuff in Domino

It was created for name changes but does much more today
It usually works on items that must be exact

It runs on every server in the domain

The admin4.nsf database controls and records all requests

Requests process on varying intervals

It is created when a server is created


Shares a replica ID with every server in your domain
This database must replicate properly and be kept healthy
Most are adjustable

Requests process at varying places

The administration server of the domain


All servers in the domain
On specific target servers

2011 IBM Corporation

93

How AdminP requests are created

Something initiates the process

A completed request generates the next step request

An administrator requests a name change


A user delegates her mail file
A server gets upgraded
A database gets clustered
And depending on the request and settings, it lingers for up to 21 days or more
Name change requests can be made to last for months

It works like multiple mini-batch files

Each request goes off and does what its asks, then reports back to the controller
(admin4.nsf)
Then the next thing happens

2011 IBM Corporation

94

How AdminP requests are processed

The AdminP task looks for new documents in the AdminP database

It builds a table of requests to process


Looks for new requests first
Then for requests to re-run a request
Builds the table of Note Ids of the documents

When a request is processed a response document is created

This document tells the result of the request


Its called an Administration Process Log document
In earlier versions of the product it was created even when no work was done
Check your server doc for the proper setting

2011 IBM Corporation

95

How AdminP Works

Servers are asked to process certain items

Like changes in ACLs


Each action has its own code (ProxyAction)

The Administration Process is multi-threaded

The main administration process thread looks for new work


Then creates a table of note IDs.
Worker threads wait until a new copy of the note ID table is available.
One thread then begins processing the first ID; a second thread processes the
next; and so forth.
After processing a request, a thread looks for another ID and the process
continues until all IDs are processed.

By default, three threads are assigned to the Administration


Process
This is adjustable in the Server Document
May be needed in times of high activity

2011 IBM Corporation

96

Example AdminP Request

Delete User from Domino Directory

This isn't even a complicated request

See Admin Help for other flow charts

2011 IBM Corporation

97

Tips and Gotchas - AdminP

AdminP works best when left alone

Make sure admin4.nsf replicates to all production servers correctly

Consider using a replication formula on hub servers to keep the database size
in check

Destroy all non-production servers

See me for an example

Consult the multitude of resources online to find Best Practices and other
details

2011 IBM Corporation

98

How ID files Work

IN THIS SECTION

Introduction to the Notes ID in the Domino Security Model

What's in an ID file?

Who needs an ID file?

How public and private key pairs work

How encryption keys work

How validation and authentication work

2011 IBM Corporation

99

Introduction to the Notes ID in the Domino


Security Model

Lotus Notes and Domino is built on a layered security model.

After firewall traffic, the Notes ID file is the next highest layer.

The ID file gains you access to the server.


The ACL gets you access to your databases (using your ID file).
Document and Form fields have Reader and Author access fields on them.
These all use your ID file, too!

The ID file is based on authentication.

Each user or server has to get through each layer to access the next system below
it.

Encrypted data in a physical file, a key, to prove you are who you say you are.

Notes ID file authentication is based on certificates.

A stamp by a Domino Certifier.

2011 IBM Corporation

100

What's in an ID file?

Certificates

Encryption keys

Recovery info (if you use it)

Public and private key pairs

Finally, the user's name and Lotus Notes license number

2011 IBM Corporation

101

Who needs an ID file?

A user who needs to authenticate with a Domino server that has public key
checking enabled.

A user who needs to encrypt their mail, or a document.

A user who needs to sign an email.

A user who needs an encryption key to see the contents of a certain field.

A Domino server

2011 IBM Corporation

102

How public and private key pairs work

Public keys are located in server docs, person docs, and certifier docs.

Why?
Because anything encrypted with a public key can ONLY be decrypted with the
private key on the ID file!

Your public key is stored on your Person doc and your ID file.

However, your private key is ONLY available on your ID file!

When a public and private key are put together, they are mathematically related
and identify the user.

When someone sends you encrypted mail, they are reading your public key
from the Domino directory.

Your ID file's private key decrypts the mail!

2011 IBM Corporation

103

How encryption keys work

It's possible for a developer to encrypt just one field on a form.

A Secret key is created from an ID file

A credit card number, salary, or social security number


File-->Security-->User Security
The key is mailed to the users who need access to the field.
Those users import the key into their ID file.

It IS possible to place all sorts of restrictions on the key...

Who can access it, can it be mailed, etc.

The form properties Security tab is then changed to have the key as the
default encryption key

2011 IBM Corporation

104

How validation and authentication work

When a user tries to connect with a Domino server, two things happen:

Domino uses three rules to trust a public key for validation:

VALIDATION: The user's ID file is validated against the certifier.


AUTHENTICATION: Once validated, the ID file sends back a message.

Trust the public key of the server's ancestors if they are in a user's ID file.
Trust any public key from a valid certificate issued by ancestors.
Trust any public keys certified by a trusted certifier.

Those rules, translated:

Domino checks each certifier in the ID file for a trust.


If an ID file's public key is trusted, Domino sends a number challenge out.
The workstation encrypts the number with the private key from the ID file and sends
back a response.
The same public key is used to decrypt the number on the server side.
If the numbers match, the user's ID file has successfully authenticated.
2011 IBM Corporation

105

How Recertifying works

IN THIS SECTION

What does a certificate contain?

Where is the certificate stored?

When would you need to recertify a user?

How do you recertify a user?

Recertifying vs renaming vs moving certifiers

2011 IBM Corporation

106

What does a certificate contain?

Like an ID file, a certificate (or certifier) contains several pieces of info:

The certificate owner's name


The certificate expiration date
The certifier name
The public key
A digital signature to prove authenticity
However, unlike a user or server ID file, a certificate does not contain a private key.

2011 IBM Corporation

107

Where is the certificate stored?

It's stored as a physical ID file.

It's also stored in the Domino directory.

Along with any other issued certificates that have been cross-certified.

Wait, it's stored in the Domino directory??

Remember, the certificate itself is public.


It's the contents that's on the physical certifier ID file that must be protected!

2011 IBM Corporation

108

Why would you need to recertify a user?

Certificates are issued to users with a shelf life they expire.

The Certification Log (certlog.nsf) will tell you everything!

If a user ID file is allowed to expire, it cannot be used to gain access to a server if it's
found one day.
Basically, a user ID is set to expire specifically so it can no longer access a server.
It will tell you who is registered with what server, and when they were registered.
It will tell you when they will expire.
It will tell you their recertification history and Notes license.

If you don't have one, create one!

You only need one for your entire Domino system.


Call it Certification Log using the certlog.ntf template.

2011 IBM Corporation

109

How do you recertify users?

Select the users from the Domino Administrator client.

On the Tools pane, select People-->Recertify


Select the SAME certifier.
You can recertify batches of users, but they must be using the same certifier,
otherwise the process will end with an error.

2011 IBM Corporation

110

Recertifying vs renaming vs moving certifiers...

Recertify a user when their existing ID file is due to expire.

Rename a user when you change their common name.

Marriage, adding an initial, fixing an incorrectly spelled last name.


Renaming does not alter or change the certifier in any way.

Move a user to a new certifier when you want to change them in the hierarchy.

All you are doing is appending new expiration dates to their ID file.
Think of it as renewing.

Uses the Request move to new certifier toolbar option under Rename.
Must be approved by Adminp.

Name and certifier changes will be updated in all ACLs, readers, names and
authors fields whose databases have an Administration Server set.

2011 IBM Corporation

111

How Cross Certification Works

INSIDE THIS SECTION

Introduction to Cross Certitifcation

Why use Cross Certification

Details of Cross Certification

Tips and Gotchas

2011 IBM Corporation

112

Intro to Cross Certification

Notes ID Files, certifications and recertification works great for users in the
same Notes Domain

When a merger happens, things change

Where all IDs are created from the same root certifier

Users need to share data from servers in separate Domains


Or you may simply need to exchange data with another Notes shop on a regular
basis

Cross certification allows a trust network to be established

2011 IBM Corporation

113

Why use Cross Certification

Notes Cross Certification allows users in different domains to send encrypted


Notes mail

Servers in other organizations can be accessed by users or other servers

If the proper ACL rights are granted


Remember that even with cross certification, ACLs and Reader Lists help keep your
data more secure

Cross Certification is useful especially if you plan to merge domains down the
road

It can be a temporary solution, or a permanent one

2011 IBM Corporation

114

Details of Cross Certification

Each organization provides the public key for their Organization or OU to


another entity

There are several ways to do it


The easiest way is to create a SAFE.ID using the Administrator client
Send it to the other Organization
And have them so the same
For complete cross certification

Contents of the SAFE.ID file

Your Canonical Name of the ID you're sending


The Public Key

2011 IBM Corporation

115

Details of Cross Certification

You can even cross certify using the phone

By reading out that incredibly long number to the other Administrator


Not a fun thing to do

The certificates are stored in the Domino Directory

Or the user's personal address book


When placed into edit mode, the entire public key is visible

2011 IBM Corporation

116

Tips and Gotchas

Choose which level you want to cross certify

The entire organization or just a certain OU?

Users can copy the cross certificates from the Domino Directory to their
personal Address Book if they need to
Sometimes handy for troubleshooting

Use cross-certification if you need to sign and encrypt across Notes Domains
A similar process exists for Internet Domains
But its a little more complex

2011 IBM Corporation

117

How the Indexer works

IN THIS SECTION

Introduction to the Indexer

View indexes vs full-text indexes

Updall vs Update

How view design affects the Indexer

2011 IBM Corporation

118

Introduction to the Indexer

The Indexer is a Domino process that keeps database views and full-text
indexes up to date.

The Indexer is simply comprised of two server tasks:

They need to be kept up to date after documents are added, removed, or modified.

The Updall task


The Update task

These tasks use two NOS services:

NIF (Notes Index Facility)


Full-Text services

NOS = Notes Object Services. They are portable C++ functions that can access
information in databases.
2011 IBM Corporation

119

Introduction to the Indexer (cont)

NIF is a process that stores collections of documents in a binary tree structure


to present to the user in a Notes view.

A b-tree data structure consists of three nodes:

A left pointer
A right pointer
A data element

The left and right pointers can point to smaller subtrees on either side.

A null pointer is a b-tree element with no elements.

SO... recursively speaking:

A binary tree is either a null pointer (empty), or a single node where the left and right
pointers point to another binary tree.
This structure allows for incredibly fast lookups and/or insertions.

2011 IBM Corporation

120

View indexes vs full-text indexes

A full-text index is a collection of files that indexes text in a Notes database to


process user search queries.

A full-text index on a database must be created manually on the database


properties tab, with four options to update it:

Daily
Updall task runs nightly
Scheduled
Must have a program document to run Updall
Hourly
Chronos keeps a timer to trigger the Update task
Immediate
Update task runs immediately

2011 IBM Corporation

121

View indexes vs full-text indexes (cont)

A view index is a collection of the data structure on every view.

The developer decides how often it updates.


The collection contains:
An index of parent-child relationships in the view.
An index sorted by note number.
Indexes defined by the view sort columns (and user-defined sorts!).

Both view indexes and full-text indexes utilize the Update server task.

2011 IBM Corporation

122

Updall vs Update

Update does the following :

Updates all views in the Domino Directory first


Updates views in all other databases
If there have been at least 20 other note changes in that database
And if the database has been accessed in the last 7 days
Updates full-text indexed databases

Update is run on server startup, and then continually.

It is listed in Notes.ini to run on startup:

Servertasks=Update,Replica,Router,AMgr,AdminP,CalConn,etc.

2011 IBM Corporation

123

Updall vs Update (cont)

Any views that have recorded changes gets the entire database placed in the
$UpdateQueue

Recorded change = A document is added, removed or modified, remember?

This $UpdateQueue is checked every five seconds.

When a database enters the $UpdateQueue, Update waits 15 minutes to


update all view indexes in that database in case any more views need updating.

This maximizes the Indexer efficiency.

The Update tasks checks a modified date of each view against the last time the
Update task ran.

This is how it knows which views in the database to update.

2011 IBM Corporation

124

Updall vs Update

The Updall task is similar if not the same as Update, HOWEVER:

It runs when it's told to:

It does not run continuously, it is a one-off, or single instance of the Update task.

Entered manually via Domino console command


From a program document
Nightly from a Notes.ini parameter to update FT Index frequencies marked Daily
ServerTasksAt2=Updall

Updall also performs additional tasks that Update does not:

Purges deletion stubs


Discards unused view indexes (every 45 days unless otherwise specified)
You can also specify parameters to turn off and on certain functions
Or even just run it on a single database
Runs on ALL databases, not just ones flagged as changed in a queue like Update

2011 IBM Corporation

125

How view design affects the Indexer

Reader fields = more Indexer work

A view is evaluated for Reader access before it can even display the view to the
user, and will refresh based on what documents it is allowed to show.

@Now or @Today in views

This will cause the view to be refreshed on a constant basis.

More views = more Indexer work

More sortable columns = more Indexer work

The alternatively sorted data is available immediately when clicked, thanks to the
Indexer.

2011 IBM Corporation

126

How Transaction Logging works

INSIDE THIS SECTION

Introduction to Transaction Logging (T/L)

Benefits of Transaction Logging

Requirements for Transaction Logging

Under the Hood

Types of Transaction Logging

Implementing Transaction Logging

How it works

2011 IBM Corporation

127

Introduction to Transaction Logging

A transaction is a single API call

A transaction log is a history of actions executed on a system

A binary file with an extension of .txn in Lotus Domino

Transaction Logging is a process that allows these actions to be recorded


before they are saved to disk

Includes creating, reading, modifying documents

Uses write-ahead logging to write a sequential record of changes


The changes can be saved shortly afterward.

The transaction log consists of log extents and the log control file
(NLOGCTRL.LFH).
A log extent is one of the log files into which the transaction logs are written.
It has the form Sxxxxxxx.TXN, where x represents a seven-digit number
that is unique to that server.

2011 IBM Corporation

128

Benefits of Transaction Logging

Better data integrity

Faster restart time

Domino can defer some work during periods of high activity

Faster Backup/Restore time

Servers start faster from a cold start


Servers start dramatically faster after unplanned outages

More efficient processing

Reduces the chance of data loss to virtually zero

Only changes are backed up, not full databases

Enables you to use DAOS

You should use DAOS

2011 IBM Corporation

129

Requirements for Transaction Logging

The T/L directory must be placed on a separate physical device devoted solely
to transaction logging

Any performance gains that you thought you'd get will disappear
Having the T/L on the same disk normally results in a 25% decrease in performance
Or worse

Each Domino server requires its own T/L drive, including partitioned servers.

The rule of thumb is one log device per data directory, and one data directory per
log device.

A dedicated, mirrored RAID1 device

If using Archive style transaction logging, a dedicated backup and restore


system is required

2011 IBM Corporation

130

Requirements for Transaction Logging on a SAN

Use a separate file system, separate pathway, and separate disks for the
transaction logs.

Consider the following

Use larger disk block size and matching Stripe size


OS file system cache is not used. NVRAM cache in the disk subsystem helps.

Use 2 GB Fibre Channel rather than 1 GB. Have dedicated channels and avoid
using data switches

Use RAID 1/0 (Stripe mirror) or mirrored pair (RAID 1), rather than RAID 5.
Use the fastest, most reliable disks available.
Configure the device with a Hot Spare available in case a disk physically fails
Do not share the disk controller (SAN and NAS) with any other users, if possible

Make sure you have adequate I/O capacity for transaction logging.

If the logs are placed on a SAN, they should be placed on dedicated devices
within the SAN

Each DPAR should have its own HBA connection to the SAN.

2011 IBM Corporation

131

Under the Hood

The Recovery Manager and Logger tasks are used to implement transaction
logging

The Logger records information passed to it by the Recovery Manager

The Recovery Manager

Is the sole NSF component that interfaces with the logger


Writes transaction-undo to the logger
Writes database recovery records to the logger
Reconstructs databases after a server has crashed
Replays database recovery records from the log
Undoes partially-completed database transactions using transaction-undo from
the log

2011 IBM Corporation

132

Under the Hood (con't)

You've seen them working:

Opening dirty previously logged DB F:\Lotus\Domino\data\mail2\baddb.nsf still


logged, containers being dropped.
DbOpen AutoFixup of Dirty Logged DB F:\Lotus\Domino\data\mail2\baddb.nsf
Clearing DBIID 0DB5EC6C for DB F:\Lotus\Domino\data\mail2\baddb.nsf
Completed consistency check on mail2\baddb.nsf
Recovery Manager: Assigning new DBIID for F:\Lotus\Domino\data\mail2\baddb.nsf
(need new backup for media recovery).

2011 IBM Corporation

133

Under the Hood (con't)

A unique database instance ID (DBIID) is generated for a database when it is


first created

It is meaningful only if the database resides on a server with T/L


It changes later whenever the database is compacted with any option other than -b
Or when fixup is run

When Domino records a transaction in the log, it includes this DBIID. During
recovery, Domino uses the DBIID to match transactions to databases

Why the DBIID changes

Compacting and fixup temporarily disable transaction logging for a database


This prevents large amounts of logging data from being spewed out to the
recovery log
Because a large amount of I/O is done during database maintenance
Since T/L was disabled, there will be a gap in the data for that database
The DBIID is changed to prevent these logs from being applied to the newly
compacted databases

Databases must be backed up immediately after maintenance


2011 IBM Corporation

134

Types of Transaction Logging

Circular logging

Server reuses a fixed amount of disk space (up to 4GB) for transaction logs.
After the log fills, the server starts overwriting old transactions
Use circular logging if the size of the log needed between full database backup
intervals is less than 4GB.
Does not support incremental backups
Most useful for faster server restarts

Linear logging

Is like circular logging, except allows more than 4GB


Use linear logging if the size of the log needed between full database backup
intervals is greater than 4GB
And you are not using archive media

2011 IBM Corporation

135

Types of Transaction Logging (con't)

Archive logging :

Creates log files as needed


Simplifies backup and restore
Provides online and partial backups
The log files are not overwritten until you archive them
Requires a backup utility to back up the filled log extents
So that they can be reused when needed.
If you do not have a backup utility or it does not work properly

The server continues to create log extents


It fills up the disk space, and then panics
Plan accordingly

2011 IBM Corporation

136

Details and Gotchas

Make sure your database ODS is the latest to get full functionality

For most things actually

Never put logs on the same disk as your server

Use the latest improvements

Manage your disk space

Create_R85_Log=1
Don't run out of room

Make sure your backup solution supports newer versions before you upgrade

2011 IBM Corporation

137

Implementing Transaction Logging

Prepare your environment properly

Enable in the Server document

Use the admin client to disable logging on any databases who wish to

Select the options you want


Some databases can be not logged
Those that have replicas everywhere (AdminP)
Those that constantly write (clubusy.nsf)

Restart the server

2011 IBM Corporation

138

How it Works

On a normal day

The Logger writes to the .txn file


The log fills and is reused or backed up then flushed
Depending on the T/L style
The server is a little faster

On a bad day

The Logger writes to the .txn file


The server stops unexpectedly
After the server restart,
The changes not saved are played back by the Recovery Manager
All is well

2011 IBM Corporation

139

How Program Documents Work

IN THIS SECTION:

What are program documents?

What can program documents be used for?

How to pass parameters to program documents

2011 IBM Corporation

140

What are program documents?

A program document will automatically run a server task at a scheduled time.

Just like a Windows batch file.

You can pass parameters to it.

You can specify which Domino server to run it on.

You can specify the days of the week, time, and repeat interval.

Setting a repeat interval of 0 means the program document will only run once.

2011 IBM Corporation

141

What can program documents be used for?

Scheduled server shutdowns and restarts

Late-night fixups, updalls and compacts

Running Domino batch files

Replication

Scheduled Domino console commands

Any command on an individual database

When program documents are used, Domino issues a NEW INSTANCE of the
program or task.

2011 IBM Corporation

142

How to pass parameters to program documents

To send Domino console commands, make sure the program name is correct!

Program name: nserver


Command line:
-c tell router update config
-c dbcache flush
-c tell amgr q
Program name: compact
Command line: -B
Program name: nserver
Command line: -c "c:\lotus\domino\data\compact.txt"
Compact.txt contains:
load COMPACT apps\ -B
load COMPACT mail\*.nsf -B

2011 IBM Corporation

143

How Smart Upgrade works

IN THIS SECTION:

Introduction to Smart Upgrade

The steps to set up Smart Upgrade

The trigger and user experience

2011 IBM Corporation

144

Introduction to Smart Upgrade

Smart Upgrade is a built-in Notes feature that will automatically install client
upgrades to users.

It can track who has upgraded, and who failed with issues
It can upgrade the mail file template at the same time
Users can defer the upgrade until a date you specify
You can build a custom MSI package to fine-tune the installation.
But, you don't have to!

2011 IBM Corporation

145

The steps to set up Smart Upgrade

1. Create the Smart Upgrade database

2. Update the Server configuration document with a doclink pointing to the


database you just created.

User the Smart Upgrade Kits template.

You can also limit the concurrent downloads via the Smart Upgrade Governor here.

3. (Optional) Create the Smart Upgrade Tracking Reports database using the
template.

This needs a mail-in document for that database on the server!

2011 IBM Corporation

146

The steps to set up Smart Upgrade (cont)

4. Create an upgrade kit document in the Smart Upgrade database.

A kit is just a fancy word for the install files!


Point it to a network share, or attach the upgrade files to the kit document.
Specify who can use it in the allowable users field.
Optionally include silent install switches here!

5. (Optional) Create a Desktop policy to track smart upgrades.

6. (Optional) Create a Desktop policy to upgrade their mail file at the same time.

2011 IBM Corporation

147

The trigger and user experience

How is Smart Upgrade triggered via the client?

A user authenticates with their home server


Or, it can be triggered manually via Tools-->Notes Smart Upgrade

Smart Upgrade does NOT use the DCC (dynamic client configuration).

It uses its own process


However, as Smart Upgrade tracking is pushed down via policy, the tracking uses
DCC

2011 IBM Corporation

148

The user experience

The Notes client release number is matched up to a source version of the Kit
document and install type.

Just the Notes client vs. Client, Administrator and Designer


Notes Basic vs. Notes Standard

If a matching kit is found, the SU process sends back a flag to the Notes client.

The user is then prompted with a dialog box prompting to upgrade.

If the user has a policy that sets a deadline, they are forced to upgrade!
There will not be a prompt if the upgrade Governor has been set and the concurrent
upgrade totals have been reached.

When a user finishes the upgrade, it sends a message to the Domino server to
decrease the current active upgrade count for the Governor.

If a user is downloading the kit instead of installing across a network share, they
will not receive notification to upgrade until the kit has completed downloading
and the installation is ready to execute!

2011 IBM Corporation

149

Whew! In Closing

Thanks for your attention

Please fill out your evals they DO matter

Contact Jess:

At GURUPalooza
And Wednesday 3PM in Swan 7-10
jstratton@solacelearning.com
She blogs on momelettes.com
twitter.com/NerdGirlJess

See Susan in the Solutions Development Lab on the Showroom floor

Also at GURUPalooza
Where she won't answer questions because she's technically not a guru
She's notesgoddess.net
Also susan_bulloch@us.ibm.com

2011 IBM Corporation

150

Related Sessions

ID103: Reducing Costs Through Improved Serviceability

ID104: IBM Lotus Domino Server Availability: Best Practices and Tuning Tips

ID105: DAOS Deployment and Best Practices

ID107: Smarter IBM Lotus Domino Monitoring: From Activity Trends to Statistics
and Reporting

BP101: Adminblast 2011

BP103: Got Problems? Let's Do a Health Check

BP107: Performing Your Own IBM Lotus Domino Security Review

BP108: Admin for the Developer: Build and Secure Your Own IBM Lotus
Domino Server Playground -- in an Hour!

BP110: Discovering the Mysterious and Dangerous Secrets of STATREP.NSF

BP112: 10 Tips to Make You an Admin Star (While Reducing Your Workload)

BP116: Backup 101: The What, How, and When


2011 IBM Corporation

Related Sessions (cont)

BP117: IBM Lotus Domino Server and Application Performance in the Real
World

BP118: Proactive Server Management: Learn How to Maximize Your Server


Uptime

BP119: Ground Control to IBM Lotus Notes: Client Management Explained

SHOW102: Statistics and Events Base Camp: Proactive Monitoring of Your IBM
Lotus Domino Servers

SHOW104: Crispy Certificates with Spicy SSL Salsa

SHOW109: How To Build a Better Cluster From a Standing Start

GEEK101: Speedgeeking!

GURU101: GURUpalooza!

NERD101: Nerd Girl Panel: Work is Not a Romance Novel or a Football Game

2011 IBM Corporation

Legal Disclaimer
IBM Corporation 2011. All Rights Reserved.
The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without
warranty of any kind, express or implied. In addition, this information is based on IBMs current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of
the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors,
or altering the terms and conditions of the applicable license agreement governing the use of IBM software.
References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change
at any time at IBMs sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor
shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by
customer.
IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or
both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.
HowStuffWorks, A Discovery Company is the trademark of HowStuffWorks, Inc. and/or Discovery Communications, LLC, A Silver Spring, MD company in the United States
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
All references to Blossom refer to a fictitious company and are used for illustration purposes only.

2011 IBM Corporation

153