Clock Synchronization

Topics in Multiprocessing 2004

Presented by: Alon Stern

Outline

Introduction Event Ordering

Partial Ordering Total Ordering Model Overview The Basic Theorem External Synchronization

Clock Synchronization

Importance of clock synchronization

Communication is not enough. Cooperation is needed synchronization.

Distributed synchronization is needed for: Transactions (bank account via ATM) Access to shared resources (network printer, databases) Ordering of events (network games where players have different ping times)

Clock Synchronization Problem

Goal Obtain the tightest achievable estimate of the readings of a remote clock in any given execution of the system.

Outline

Introduction Event Ordering

Partial Ordering Total Ordering Model Overview The Basic Theorem External Synchronization

Clock Synchronization

Event Ordering

The concept of one event happening before another in a distributed system is examined. Defining a partial ordering of events. Defining total ordering of events.

Defining Our System

The system is composed of a collection of processes. Each process consists of a sequence of events. The events of a process form a sequence, where a occurs before b in this sequence if a happened before b. Sending or receiving a message is an event in a process.

Causality

Event ordering linked with the concept of causality:

Saying that event a happened before event b is the same as saying that it is possible for event a to causally effect event b. If events a and b happen on processes that do not exchange any data, their exact ordering is not important.

Partial Ordering

In a distributed system it is sometimes impossible to say that one of two events occurred first.

The relation happened before is therefore only a partial ordering of the events in the system.

The Happened Before Relation

The happened-before relation in a distributed system: 1. If a and b are two events in the same process, and a comes before b, then a b. 2. If a is the sending of a message, and b is the receipt of that message, then a b. 3. If a b and b c then a c. Two distinct events a and b are said to be concurrent if a b and b a.

Example

p1 q2 (rule #2) q2 q4 (rule #1) q4 r3 (rule #2) r3 r4 (rule #1)

p1 r4 (rule #3)
p3,q3 are concurrent (neither can causally effect the other)

Logical Clocks

Problem: How do we maintain a global view on the systems behavior? Solution: Use logical clocks on each process.

Logical Clocks

Define a Logical clock Ci for each process Pi to be a function which assigns a number Ci(a) to any event a in that process. Monotonically increasing counter. No relation with physical time: Just a way of assigning a number to an event.

Logical Clocks
The clock condition: If a b then c(a) < c(b). The clock condition is satisfied if:

If a and b are events in process Pi, and a comes before b, then Ci(a)<Ci(b). If a is the sending of a message by process Pi and b is the receipt of that message by process Pj, then Ci(a)<Cj(b).

Logical Clocks

Note: We can not expect the converse condition. Reason: That would imply that any two concurrent events must occur at the same time. Example: p2 and p3 are both concurrent with q3. they must occur at the same time as q3. C(p2)=C(q3)=C(p3). But, p2 p3.

Logical Clocks in Space-Time Diagram

There must be a tick line between any two events on a process line. Every message line must cross a tick line.

Satisfying the Clock Condition

Each process Pi maintains a local counter Ci and adjusts this counter according to the following rules:
1.
2.

3.

Ci is incremented before each event. Ci=Ci+1. If event a is the sending of a message m, then the message m contains a timestamp Tm= Ci(a). Whenever a message m is received by a process
Pj, Pj adjusts its local counter Cj: Cjmax{Cj+1,Tm+1}.

Satisfying the Clock Condition

public class LamportClock { int c; public LamportClock() { c = 1; } public int getValue() { return c; } public void internalAction() { c = c + 1; //action execution } public void sendAction() { c = c + 1; msg.header.timeStamp = c; //send message } public void receiveAction(int src, int sentValue) { c = max(c, sentValue) + 1; }

Total Ordering With Logical Clocks

Problem: it can still occur that two events happen at the same time. Solution: Attach a process number to an event: Pi timestamps event e with {Ci(e),i}.

The relation defines a total ordering. ab if and only if either:

Ci(a)<Cj(b) or Ci(a)=Cj(b) and i<j

Example of Total Ordering

Events (1,0) and (1,1) were considered to be concurrent. Now we can say (1,0) (1,1)

(1,0)
(2,0) (5,0) (6,0)

(1,1) (2,1) (3,1) (4,1) (5,1) (6,1)

(1,2)

(2,2)

(4,2)

(7,1)
(7,0)

Anomalous Behavior

Presence of outside interactions.

Carrying a diskette from one machine to another. Dictating file changes over the phone.
Explicitly introduce into the system the necessary information about the ordering. Strong clock condition: for any event a,b if ab then c(a)<c(b). Requires physical clocks.

Two ways to avoid such anomalous:

Outline

Introduction Event Ordering

Partial Ordering Total Ordering Model Overview The Basic Theorem External Synchronization

Clock Synchronization

Clock Synchronization

Goal: Ensure that physically dispersed processors will acquire a common notion of time using:

Local physical clocks (whose rates may vary). Message exchange over a communication network (with uncertain transmission delay).

Question: Can we set all the clocks in a distributed system to have the same time?

Physical Clocks

Every computer contains a physical clock. A clock is an electronic device, which counts oscillations in a crystal at a particular frequency.

Example: Oscillator frequency is 10MHz increment clock by one second after counting 10M cycles.

Physical Clocks in Distributed System

Can we synchronize all the clocks to some known high degree of accuracy? Well, there are some problems Its difficult to synchronize the clocks. Crystal based clocks tend to drift over time.

Physical variations in the crystals, temperature variations, aging, etc. Drift is small, but adds up over time. For quartz crystal time, maximum drift rate is 50 seconds every 106 seconds=11.6days. Best atomic clocks have drift rate of one second in 1013 seconds = 300,000 years.

Real Time

Exact time was computed by astronomers. Take noon for two days, divide by 24*60*60 mean solar second. At present, the real time is taken as the average of some 50 cesium-clocks around the world. Introduces a leap second from time to time to compensate that days are getting longer. This time called Universal Coordinated Time (UTC). UTC is broadcast through short wave radio and satellite.

Does This Solve All Our Problems?

Dont we now have some global timing mechanism?
Not every machine has UTC receiver. Suppose we have a distributed system with a UTC receiver somewhere in it, we still have to distribute its time to each machine.

Clock Synchronization Variants

External Synchronization Get tight estimate on some source clock which shows real time. Internal Synchronization Real time is not available. Keep all clocks as close as possible.

Main Difficulties

Synchronization tends to deteriorate over time and space. When the local clocks are not drift free, the synchronization loosens timing information should be refreshed periodically. Communication information to distant processors timing uncertainty due to unpredictable message delays. Usually there exist some a-priory bounds on these uncertainties.

Model Overview

Local Clock The hardware clock of the processor. Network Connects processors to their neighbors. Send Module When and where to send a message. Clock Synchronization Algorithm (CSA) Computes a correctness and an error margin.

Local Clock

Hardware clock, Physical clock. Encodes at all states s of processor v a real number called local time denoted LT(s). Drift free or bounded drift clock.

Drift free Run at the rate of real time, but does not necessary shows real time. Bounded drift Known lower and upper bounds on the rate of its progress relative to real time, denoted +, -

Drift Bounds - Example

Typical workstation - clock accuracy of 50 parts per million (ppm). If it shows that 1000000 time units have passed between event p and q, we are guaranteed that RT(p)-RT(q) [999950,1000050].

Network

Connects processors to their neighbors. May lose, duplicate and deliver messages out of order. The real time delay of each message m is within some known bounds

0 L(m) H(m) .

Bounds may vary from link to link and from message to message.

Clock Synchronization Module (CSA)

Uses:

Local time Received Messages Specification of the system

Correctness term D Error margin e

To compute

The logical time is (T,e) where T=LT+D. sticks extra bytes to an outgoing message and strips the corresponding bytes of incoming message.

Before we start
Lets take a simple example: Suppose u,v are drift free clocks. When the clock of node u shows 12, u can safety deduce that the clock at v shows something in [18,20].

System Execution

A sequence of events, where each event p, has:

Real time of occurrence RT(p). Local time of occurrence LT(p).

We shall think of an execution as a graph, whose nodes are the set of events, with an edge (p,q) if either:

q is the receive event whose send event is p. p,q occur at the same processor and q is the first event following p.

Execution as a Graph

Nodes:

p
RT(p) LT(p)

{p,q,r,s,t}

r RT(r) LT(r)

Edges:

{(p,q),(q,s),(r,s),(s,t)}

q RT(q) LT(q)

s RT(s) LT(s)

t RT(t) LT(t)

View

Given an execution, its view is obtained by omitting the real time mapping. View graph Gb

V - The set of points E - Two anti-symmetrical directed edges joining each pair
of adjacent points.

An event of an execution or a view is called a point. The local view of a point is the view of all points which happened before the point.

View Graph - Example

p
LT(p) r LT(r)

Time
q LT(q) s LT(s)

t LT(t)

The Basic Theorem

Lets start with some notations:

act_del(p,q) = RT(p)-RT(q) virt_del(p,q) = LT(p)-LT(q) Gb = (V,E) is a view graph, with nodes V and edges E. Points are called adjacent if they are
connected with an edge.

Real Time Specifications

Bounds on the difference of real times between pairs of events. Use the following real time specifications: Message transmit bounds. Clock drift bounds. The bounds will be modeled uniformly no difference between the bounds on communication delay and the bounds on drifting clocks.

Bounds Mapping

A function B that maps every pair p,q of adjacent points to a number B(p,q)>-. The interpretation of bounds mapping is only upper bounds. For every pair of adjacent points p,q we have that RT(p)-RT(q) B(p,q).

Bounds Mapping - Formulation

State the bounds mapping using quantities available to the processor:

L(m), H(m), +, -, virt_del

For a message m with send point p, receive point q and delay in the range [L(m),H(m)]:

B(q,p)=H(m) and B(p,q)=(-L(m)). For two adjacent points p,q at a processor having drift bounds +, B(q,p)= virt_del(q,p)/B(p,q)= virt_del(p,q)/+

Bounds Mapping - Example

Given: Processors clock may drift up to 100ppm. Message delivery time is completely arbitrary (between 0 to ). The bounds mapping: If p is the send event and q is the receive event of that message B(p,q)=0. If p occurs before q at the same processor, B(p,q)=(LT(p)-LT(q))/1.0001 and

B(q,p)=(LT(q)-LT(p))/0.9999. For all other cases B(p,q)= .

Relative Offset

Ties together actual and virtual delays. Absolute offset of a point p:

d(p)=RT(p)-LT(p). The offset of p relative to a point q: d(p,q)=d(p)-d(q). d(p,q)=-d(q,p).

Lemma: For any two points p,q of a pattern,

Lemma: For any three points p,q,r of a pattern, d(p,q)=d(p,r)+d(r,q).

Pause

What information do we have till now?

A view graph, containing the local time LT of each event. bounds mapping in terms of quantities available to the processor L(m), H(m), +, -,
virt_del

Offset definition which ties together actual and virtual delay.

Using this information we can now define a synchronization graph.

Synchronization Graph
Given: A view b =(V,E). Bounds mapping B.

The synchronization graph GbB=(V,E,w) is: The view graph (V,E) where for each (p,q) E, w(p,q)=B(p,q)-virt_del(p,q). Lemma: For every pair (p,q) E, d(p,q) w(p,q). Explanation:

act_del(p,q)-virt_del(p,q) B(p,q)-virt_del(p,q).

Synchronization Graph Example

Nodes, points. Two anti-symmetrical directed edges joining each pair of adjacent points. Weighted edges.

Synchronization Distance

Strategy - Reduce synchronization problem to distance computations in the synchronization graph. synchronization distance d(p,q): Length of the shortest directed path from p to q in the synchronization graph. Lemma: If B is satisfied, then for every two points p,qGbB, d(p,q) d(p,q)

Synchronization Distance Example

d(p,q) = 10 d(q,p) = -8

The Main Theorem

Let p,q be to points in a view b. In any execution:

RT(p)-RT(q) [VD(p,q)-d(q,p),VD(p,q)+d(p,q)]

Furthermore, there exist executions which meets the bounds.

Interpretation: The synchronization distances determine precisely the uncertainty bounds.

Main Theorem - Example

Given

Node u

Node v
w=8 q LT=15 w=0 r LT=17 w=0

p is an event at node u p LT=7 q,r are events at node v The clocks are drift free w=(-6) The delay of the message from p to q is in the range [0,2].
drift free w(q,r)=w(r,q)=0 B(q,p)=2 w(q,p)=2-(15-7)=(-6) B(p,q)=0 w(p,q)=0-(7-15)=8 d(r,p)=(-6), d(p,r)=8

Solution:

RT(r)-RT(p) [(17-7)-8, (17-7)+(-6)] RT(r)-RT(p) [2,4]

Main Theorem - Example Cont

RT(p)=5, RT(q)=7, RT(r)=9 Real time specifications are satisfied. RT(r)-RT(p)=4 RT(p)=5, RT(q)=5, RT(r)=7 Real time specifications are satisfied. RT(r)-RT(p)=2 RT(p)=5, RT(q)=10, RT(r)=12 Real time specifications are not satisfied. X RT(r)-RT(p)=7

p LT=7 q LT=15

r LT=17

External Synchronization
Each processor has a local clock. Each clock has drift bounds. Each message has latency bounds. One of the clocks, Called source, shows real time. Task Processors should provide the best estimate of the source clock. An estimate is an interval which is guaranteed to contain the source time.

Synchronization Algorithms

Efficient - If its complexity is independent of the execution length. Optimal If it always outputs the tightest bounds. General If it works for all possible systems.

Space Lower Bound

Any optimal and general algorithm for external synchronization has high space complexity is not efficient. There is no algorithm which is general, efficient and optimal.

One of the three should be sacrificed.

Drift Free Clocks

Lemma The distance between any two points that occur at a drift free clock is 0. Meaning All points associated with a processor with a perfect clock may be collapsed into a single supper point for synchronization distance purposes.

Number of nodes in the graph

Number of processors

Drift Free Clocks - Algorithm

Idea Use distribution Bellman-Ford. Algorithm message from u to v contains:

timestampu, wu(v,u), wu(u,v), du(s,u), du(u,s)

When v receives a message with delay bounds [L,H]: vd = timestampv - timestampu wv(u,v) = min{H-vd, wu(u,v), wv(u,v)} wv(v,u) = min{L+vd, wu(v,u), wv(v,u)} dv(v,s) = min{du(u,s)+wu(v,u), dv(v,s)} dv(s,v) = min{du(s,u)+wu(u,v), dv(s,v)} D= (dv(v,s) - dv(s,v)) ev = (dv(v,s) + dv(s,v))

References

A Theory of Clock Synchronization, by PattShamir and Rajsbaum (STOC 94). Optimal and Efficient Clock Synchronization Under Drifting Clocks, by Ostrovsky and PattShamir (PODC 99). Time, Clocks, and the Ordering of Events in a Distributed System, by Leslie Lamport.