BFS State System

Introduction
 BFS State System
Purpose
Indicates which version of maps are currently active
Indicates when new data has arrived and when new maps should be built
Serves as historical record for data feed updates

Components
BFSStateValue table
Holds state value entries for each data type. Periodically trimmed when it gets too big.

File server stores data in version subdirectories

Directories correspond to state value entries

Manager services running on Preprocs, Q Servers, Distributor.

Interrogates state values to determine what to copy, what to build, and what to load on
a per datatype basis
Advances state to next version as new data arrives and maps are built and copied to
their destinations (i.e. file server, Q servers)

Query processes (bfsqproc.exe)

Relies on state values to determine what map versions to load for pricing
 Data Feeds
Data Providers
ATPCo
OAG
IATA
Travelport (formerly WSPN)
Sabre

Data Types BFS map type designator

Domestic fares 001, 003, 005, (006 All-Adds)
Intl fares 051, 053, 055, 057, 059, (060 All-Adds)
OAG Schedules 026
MCT 009
AVS 007
CCF 036, (037 Init)
PFC 010
WSPN TAX 032
EXPE TAX 074
WSPN CVR 027
SABRE CVR 075
..., etc.
 BFSStateValue Table Definition
Part of BFSConfiguration Database
Data versioning accomplished by associating state names with state
values that reference BFS Map Versions (BMV)
Table Columns
BFSStateValueId surrogate PK
TravelProductID TPID (not used)
BFSServerFarmName BFS farm
BFSMapFileVersionNbrBegin beginning of range (used with chaining)
BFSMapFileVersionNbrEnd end of range (used with chaining)
BFSStateName identifies data type
BFSStateValue data type version (bmv)
CreateUser entity that added row
CreateDate date/time when row was added

Stored Procedures:
BFSStateValueAdd - adds new row for a state name / value
BFSStateValueGetCur - retrieves most recent row for a state name
BFSStateValue Table
 State Names
State names are associated with individual data sources. Naming conventions
follow a general pattern but are not always consistent. Data type, location, and
GDS (when applicable) are encoded in the name.

States updated by Preprocessor

Global source data version ex. GlobalCCFVersion
Current GDS version ex. GlobalWSPNCurrentMapVersion
Local Map version ex. BFSPLocalBFSPCCFVersion
Global Map version ex. BFSPGlobalBFSPCCFVersion

States updated by Query Server

Latest maps copied from file server ex. BFSQLocalBFSQCCFVersion
Current pricing maps ex. BFSQLocalCurrentCCFVersion

States updated by Distributor

Begin with "BFSD" ex. BFSDGlobalBFSQAVSVersion
 State Values

Form: YYYYMMDD_MMM_NNN
Referred to as BMV (BFS Map Version)
Identify data files present on servers (file server, Q server, etc.)
YYYYMMDD - date
MMM - identifies the data type. (ex. 001, 003, 007, 026, etc.)
NNN - timestamp (#of minutes past midnight / 2). Set to "000 when
not applicable.
Allows for easy comparison between versions
i.e. If GlobalxxxxVersion > BFSPGlobalxxxxBFSPVersion
Build new maps
Update state values

Beginning of time: 19000101_000_000

End of time: 20790606_999_999
 Global vs Local States
Global states (names beginning with Global)
Global data type versions refer to raw source data version on file server.
GlobalWSPNZipVersion domestic fares
GlobalOAGVersion - schedules

GlobalGDSCurrent versions refer to active GDS pricing data.

GlobalWSPNCurrentMapVersion
Global1SCurrentMapVersion

BFSP states
BFSPGlobalBFSP refers to maps on file server available for building
new maps.
BFSPGlobalBFSQ refers to maps on file server that are globally
available for distribution to Q servers.
BFSPLocalBFSP refers to maps residing on a specific Preprocessor
after they are built ready for copying to file server.

BFSQ states
Always Local. Specific to individual Q Server
 Work Flow
Data download and processing sequence and corresponding state updates

1) New source data on FTP site is downloaded to File Server and matching
GlobalVersion state updated with latest version number.

2) Data copied to Preprocessor. Maps built. BFSPLocalBFSP updated

3) Built maps are copied to File Server and BFSPGlobalBFSP updated.

BFSPGlobalBFSQ usually updated as well. (exception: Partial All-Adds)

4) Q Servers copy new maps. BFSQLocalBFSQ updated. BFSQLocalCurrent

may also be set depending on data type.

Note: For some data (i.e. fares), "swap" messages are sent from GDS. BFS
updates GlobalCurrentGDS and BFSQLocalCurrent states when they
arrive.
 Work Flow Diagram
FTP Server BFS Query
Servers

BFS File Server

1) Raw data downloaded
Global state Updated

5) New maps copied to

Q Servers. BFSQLocal states
Updated.
2) Raw data copied to
Preprocessor

BFS Preprocessor
BFS Distributor
4) New maps copied to
File server. BFSPGlobal
States updated
3) Maps built
BFSPLocal state
updated New maps (AVS) copied to Messages retrieved
File server. BFSDGlobal state From Local Receiver
Updated. Message store.
 Workflow -BFSP Managers
Each data feed has manager that runs in continuous loop as follows.

Begin Loop:
If BFSPLocalBFSPVersion > BFSPGlobalBFSPVersion
Copy local maps from preprocessor to file server
Update BFSPGlobalBFS?Version state(s) *
* Note: Two states BFSPGlobalBFSP and BFSPGlobalBFSQ

if GlobalVersion > BFSPGlobalBFSPVersion

Build new maps on preprocessor
Update BFSPLocalBFSPVersion

Log into FTP Server

If FTP file version > GlobalVersion

Download new source data
Update GlobalVersion
End Loop:
 Workflow - BFSQ Managers
Each map type has BFSQ manager that runs in continuous loop.
We always keep the latest two versions of each map type on the Q server:
BFSQLocalCurrentxxxVersion points to map version loaded by pricing engine for each map type

Begin Loop:
if NOT fares
if BFSPGlobalBFSQVersion > BFSQLocalBFSQVersion
Copy new maps from file server
Update BFSQLocalBFSQVersion
Update BFSQLocalCurrentVersion (Not GDS specific, no swap notification)
endif
else
// Fare map download
More complicated logic for setting BFSQLocalBFSQ and BFSQLocalCurrent[GDS]Version
Need to synch with GDS based on having received swap notifications.
endif

Cleanup. Delete old maps. Keep two most recent versions

End Loop:

Notes: 1) BFSQLocalBFSQVersion refers to the latest maps copied down to Q server

2) BFSQLocalCurrent[GDS]Version refers to maps currently used in pricing
3) BFSQLocalBFSQVersion >= BFSQLocalCurrent[GDS]Version where GDS = 1P or 1S for GDS-specific data.
BFSStateValue Table
 Init vs Update
Init - replaces all previous data and establishes a new baseline
Update - delta applied on top of existing data.

Some data always delivered as an init.

Ex. OAG schedules

Some data is always an update.

Ex. Sabre AVS (Wspn provided both inits and updates)

Some data can be either.

Ex. CCF, ATPCo Fares

Fares are always delivered as an update except by default. Fare Inits (a.k.a.
All-Adds) are delivered by request and cost ~$30K
 Fare Maps
Domestic fares
3 updates daily: 001, 003, 005

International fares
5 updates daily: 051, 053, 055, 057,059

Base map versions - identified by "120" in the timestamp field.

Ex. 20080212_005_120

Update versions contain updated records since the last base maps were built.

Pricing queries load current Base maps + current Update maps.

GDS can differ as to which update is current in their system. Base + update
architecture allows BFS to simultaneously load different update maps to match
multiple GDSs using less memory.

New base maps are generally built each night ("flattening") provided that all GDSs
have swapped to the final daily update. (005 - domestic, 059 - international)
 Fare Swap
Each GDS notifies BFS when they have switched to the next fare update. Allows
BFS to stay in synch.
WCSWAP service on Preprocessor. Lets us know when TravelPort (WSPN) has swapped.
Sabre swap msgs delivered via message queue. Distributor updates BFS state

Global States
GlobalWSPNCurrentMapVersion: current Travelport domestic fare version
Global1SCurrentMapVersion: current Sabre domestic version

BFSQ States
BFSQLocalCurrentMapVersion - Domestic maps used when pricing against Travelport.
BFSQLocalSabreCurrentMapVersion Domestic maps used version when pricing against
Sabre

Note: International fare maps have analogous state counterparts.

Currently Sabre does not send swap notification for international fares
 Fare Map Flattening
Consolidates all the fare updates and most recent base maps into a new set
of base maps.

New Base = Old Base + update1 + update2 +....+ updateN

(Note: updateN maps actually contain all previous updates since last base
was built. Previous updates are carried forward with each new map ver.)

Usually happens nightly but only if all GDSs have swapped to pricing with
the last daily fare update.

Domestic Base Map version

YYYYMMDD_005_120

International Base Map version

YYYYMMDD_059_120
 All-Adds
Full All-Adds
Complete replacement of fares and rules
Version numbers
Domestic 006
International 060

Partial All-Adds
Partial replacement
Treated more like a fare update
Shows up as timestamp in version number
Ex. 20080207_059_230
BFSPGlobalBFSP version updated but not BFSPGlobalBFSQ
Copied to file server but not Q servers
Folded in with next scheduled fare update i.e. 001, 051
 Map Version Chaining
Relevant to data sources delivered as an update (vs init).

Makes use of the BFSMapFileVerNbrBegin/End fields of the

BFSStateValue table records to ensure updates get applied in
proper sequence when building new maps.

MapFileVerNbrBegin indicates map version just prior to version

specified in StateValue field in the same record.

BFSStateValueCurGet sproc takes version number argument to

facilitate this. Searches for most recent entry in table where arg ver
num is between MapNbrBegin and End inclusively.

Bfsstatevaluecurget name, 0, 20080210_036_000, farm, server, 0

Map Chaining (cont.)
 Sabre Feeds
Data Types (feeds) are all Inits
BankersSellingRate (BSR)
MultiTransportTable (MTT)
Mileage exception data (MET)
Industry Fares Translator (YYT)
Cabin Mapping Data (CAB)
AVS Summary (AVS)

Each type consists of one or more SQL table exports from Sabres database (ex.
BSR = 1 table. MET =12 tables)

BFS imports raw data into matching SQL tables. Maps generated by extracting data
from tables.*

*Note: Really large files (i.e. Mileage and AVS Summary) may not get loaded into SQL
tables. Instead we may choose to build maps directly from the raw source.

Currently only processing BSR and MTT

 BFSSabre Database
BFSSabre database
One table for every data file (not data type feed) received
Allows for easy searching and analysis
May facilitate map linking through the use of SQL joins (MET)
Allows manual modification of data and map regeneration (when
necessary)

Sabre farm
Farm name used in BFSStateValue table to indicate current version of
data stored in tables.

BFSPGlobalBFSPxxxxVersion updated for each Sabre data source

when all tables for that data source have been loaded.

Farm-wide lease used to synchronize access to the tables. (In

production, preprocessors from multiple real farms share these tables.)
Questions?