18.02.

09
Pgpool-II for beginners
Table of Contents
prerequisite...........................................................................................................................................2
introduction...........................................................................................................................................2
installation............................................................................................................................................3
configuration.........................................................................................................................................3
start/stop pgpool...................................................................................................................................4
initial replication setup.........................................................................................................................4
copy database ..................................................................................................................................4
starting up........................................................................................................................................4
check if replication is working.............................................................................................................
!nitial situation.................................................................................................................................
create a test database and insert so"e data......................................................................................
check data in each node...................................................................................................................
failure e#a"ple.....................................................................................................................................
$nline reco%ery....................................................................................................................................&
enable online reco%ery.....................................................................................................................&
creating scripts used by online reco%ery..........................................................................................&
copy'base'backup......................................................................................................................8
pgpool'reco%ery'pitr..................................................................................................................8
pgpool're"ote'start....................................................................................................................8
initiate online reco%ery....................................................................................................................9
(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((
)opyright *c+ 2009 ,g,ool -lobal .e%elop"ent -roup
$riginal te#t written by -erd /oenig
please send co""ents to0 gk'ul" at web.de
((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((
-erd /oenig 1/12 pgpool(!! for beginners
18.02.09
prerequisite
postgresql headerfiles *postgresql(de%el(8.3.(2.1.#81'14.rp"+
libpq.so *check /usr/lib14/libpq2+
"ake3gcc
introduction
4here are 2 ser%ers3 identical hardware3 shared nothing with installed $pen5use10.3 *14bit+.
e#isting ssh(key(e#change fro" node1 6 node2
,ostgre5788.3.3 ,-.949 : /opt/postgres/data
!n this guide pgpool will be installed under directory /opt/pgpool!!3 ad;ust the para"eter <((prefi#=
if you want to install to a different location.
4his tutorial co%ers the usage of replication "ode only >
! strongly reco""end to read the docu"entation at http0//pgpool.pro;ects.postgresql.org/ to get as
"uch infor"ation as possible about pgpool
,lease keep in "ind that the pgpool instance is a single(point(of(failur *5,$?+ and should be "ade
high a%ailable with an appropriate tool like heartbeat.
-erd /oenig 2/12 pgpool(!! for beginners
18.02.09
installation
)heck if you ha%e proper rights to call <"ake install=3 or ;ust perfor" it as root.
cd /incoming/
tar -xzf pgpool-II-2.1.tar.gz
cd pgpool-II-2.1/
./configure \
--prefix=/opt/pgpoolII \
--with-pgsql-libdir=/usr/lib!/ \
--with-pgsql-includedir=/usr/include/pgsql/
ma"e ## ma"e install
@sudo chown (A postgres /opt/pgpool!!B
configuration
cd /opt/pgpoolII/
cp etc/pcp.conf.sample etc/pcp.conf
cp etc/pgpool.conf.sample etc/pgpool.conf
$i etc/pcp.conf
set user0pw3 password in C. for"at
create password in C. for"at0
/opt/pgpoolII/bin/pg%md& -p
password'(t)pe password*
e+a!+&,+&1e2+c-d.&.&.+fb2/fc&
$i etc/pgpool.conf
Dere you can define ports for pgpool3pgpool co""unication "anager3 listen addresses and a lot of
other things. 9s "entioned earlier !Ell focus on enabling replication "ode and define both database
backends.
0
listen%addresses =121
...
replication%mode = true
0
bac"end%hostname. = node.11
bac"end%port. = &!,2
bac"end%weight. = 1
bac"end%data%director). = 1/opt/postgres/data1
bac"end%hostname1 = 1node.21
bac"end%port1 = &!,2
bac"end%weight1 = 1
bac"end%data%director)1 = 1/opt/postgres/data1
-erd /oenig 3/12 pgpool(!! for beginners
18.02.09
start/stop pgpool
5tart pgpool by calling the binary and3 dependent on your needs3 add so"e para"eters0
/opt/pgpoolII/bin/pgpool -d -n * /opt/pgpoolII/log/pgpool.log 2*#1 #
,ara"eters0
@-d :F %erbose debugB
@-n :F detach ter"inalB
@-f (configfile* :F if the config file isnEt inside the default path G,AH?!I/etcB
5top pgpool by calling the binary with subsequent co""and <stop=
/opt/pgpoolII/bin/pgpool stop
initial replication setup
copy database
4he following steps requires an established session to node1 as user postgres. ,ostgre578 is
running on node1 and configured with enabled wal archi%ing. !f you donEt ha%e an e#isting database
you want to replicate3 ;ust skip this section.
psql -3 postgres -d testdb
-* select pg%start%bac"up41initial%bac"up156
-* \q
cd /opt/postgres
rs)nc -a$z ./data/2 postgres7node2'/opt/postgres/data/
psql -3 postgres -d testdb
-* select pg%stop%bac"up456
-* \q
starting up
?inally we can start ,ostgre578 on each node and afterwards pgpool(!!.
4herefore login to e%ery backend and call0
as user root0 /etc/init.d/postgres start
or as user postgres0 pg%ctl -8 /opt/postgres/data start
on node1 as user postgres0
/opt/pgpoolII/bin/pgpool -d -n * /opt/pgpoolII/log/pgpool.log 2*#1 #
:F check log file if pgpool started correctly and all the defined backends are enabled. ! strongly
reco""end to enable debug output *(d switch+ at this stage.
-erd /oenig 4/12 pgpool(!! for beginners
18.02.09
check if replication is working
Initial situation
postgres is running on all nodes
pgpool is running on port 9999 on node 1
shell session on node1 established
create a test database and insert some data
,erfor" the following steps to create database <bench'replication=3 insert so"e base data and insert
a bunch of rows to table <history=.
createdb -p ---- bench%replication
pgbench -i -p ---- bench%replication
psql -p ---- bench%replication
bench%replication=9 insert into histor) 4tid: bid:aid:mtime:filler5 4select 1:1:1:now45:i''text from
4select generate%series41:1......5 as i5 as q56
check data in each node
Jou can check if the databases are in sync with a si"ple shell script3 which connects to each node
and fires a select query there.
e.g.0
9;/bin/bash
for host in node1 node26 do
for table%name in accounts histor)6 do
echo <host' <table%name
psql -c =>?@?AB count425 CDEF <table%name= -h <host -p &!,2 bench%replication
done
done
-erd /oenig /12 pgpool(!! for beginners
18.02.09
failure example
4o si"ulate a failure scenario3 ! ;ust killed postgres processes on node2 while an update state"ent is
running *!E%e chosen an update on table history because we inserted a lot of data in the pre%ious step
there+.
Jou need at least two shell sessions3 one at node1 to fire the update/insert state"ent3 and one
session on node2 to kill the processes.
Kode10
psql -p ---- bench%replication
update histor) set tid=26
Kode20
p"ill postgres
LL log entries in pgpool.log
@EG' pid 2.++' statement' update histor) set tid=26
8?H3G' pid 2.++' do%command' Iuer)' H?GIJ
8?H3G' pid 2.++' command tag' H?GIJ
8?H3G' pid 2.++' Dead)CorIuer)' transaction state' B
8?H3G' pid 2.++' do%command' Iuer)' H?GIJ
8?H3G' pid 2.++' command tag' H?GIJ
8?H3G' pid 2.++' Dead)CorIuer)' transaction state' B
8?H3G' pid 2.++' waiting for bac"end . completing the quer)
8?H3G' pid 2.++' waiting for bac"end 1 completing the quer)
8?H3G' pid 2.++' read%"ind%from%bac"end' read "ind from . th bac"end A J3F%HKAL?J8>' 2
8?H3G' pid 2.++' read%"ind%from%bac"end' read "ind from 1 th bac"end ? J3F%HKAL?J8>' 2
?DDED' pid 2.++' pool%process%quer)' 1 th "ind ? does not match with master connection "ind A
@EG' pid 2.++' do%child' exits with status 1 due to error
8?H3G' pid 2.+-' I am 2.+- accept fd .
@EG' pid 2.+-' connection recei$ed' host=MlocalN
8?H3G' pid 2.+-' Orotocol FaPor' , Finor' . database' bench%replication user' postgres
@EG' pid 2.+-' connection closed. retr) to create new connection pool.
8?H3G' pid 2.+-' new%connection' connecting . bac"end
8?H3G' pid 2.+-' new%connection' connecting 1 bac"end
8?H3G' pid 2./-' reap%handler called
8?H3G' pid 2./-' reap%handler' call wait,
8?H3G' pid 2./-' child 2.++ exits with status 2& b) signal .
?DDED' pid 2.+-' connect%inet%domain%soc"et' connect45 failed' Aonnection refused
?DDED' pid 2.+-' connection to node24&!,25 failed
?DDED' pid 2.+-' new%connection' create%cp45 failed
@EG' pid 2.+-' notice%bac"end%error' 1 fail o$er request from pid 2.+-
8?H3G' pid 2,!.' I am 2,!.
8?H3G' pid 2./-' for" a new child pid 2,!.
8?H3G' pid 2./-' child 2.+- exits with status 2& b) signal .
8?H3G' pid 2,!1' I am 2,!1
-erd /oenig 1/12 pgpool(!! for beginners
18.02.09
8?H3G' pid 2./-' for" a new child pid 2,!1
8?H3G' pid 2./-' reap%handler' normall) exited
8?H3G' pid 2./-' failo$er%handler called
8?H3G' pid 2./-' failo$er%handler' starting to select new master node
@EG' pid 2./-' starting degeneration. shutdown host node24&!,25
@EG' pid 2./-' failo$er%handler' do not restart pgpool. same master node . was selected
@EG' pid 2./-' failo$er done. shutdown host node24&!,25
8?H3G' pid 2./-' reap%handler called
8?H3G' pid 2./-' reap%handler' call wait,
8?H3G' pid 2./-' reap%handler' normall) exited
Kow we can ha%e a closer look how to handle a failo%er and how to get back to replication "ode
with databases in sync. 4o reduce the "anual steps pgpool offers <$K8!KH AH)$MHAJ=
"echanis"3 see chapter $nline reco%ery
-erd /oenig &/12 pgpool(!! for beginners
18.02.09
Online recovery
4his "echanis" can be used for both attaching a node after failo%er and attaching a new node.
9 node has been detached auto"atically after failo%er3 and a co"plete new node is in detached state
also after defining the backend'IJN para"eters in pgpool.conf and reloading pgpool.
.atabase copy and reco%ery will be co%ered by ,ostgre578Es ,!4A functionality. Marious scripts
are needed to co"plete the reco%ery successfully and should be located under directory G,-.949.
4he sa"ple scripts pro%ided in the source tarball are a good starting point and !Ell use the" later on
and adapt the" to this tutorial en%iron"ent.
9n o%er%iew o%er the different steps included in online reco%ery is shown in drawing
online'reco%ery'theory.
enable online recovery
4o be able to use online reco%ery an additional function for te"plate1 is needed3 therefore the
following steps should be perfor"ed on e%ery database node0
cd /incoming/pgpool-II-2.1/sql/pgpool-reco$er)/
ma"e install
psql -f pgpool-reco$er).sql template1
!n the ne#t step we ha%e to set the para"eters in pgpool.conf to enable the reco%ery co""ands and
the health check. !n this tutorial ! use the failo%er(/failback co""ands ;ust for creating a te#t file
and log a "essage there. Jou can define whate%er co""and/script you want instead. 4he %alues of
the reco%ery'IJN'co""and para"eters are na"es of scripts3 weEll ha%e a look at the" in the ne#t
step.
Cy settings are as follows0
0
health%chec"%period = ,. 9 chec" e$er) ,.s
0
failo$er%command = 1echo host'Qh: new master id'Qm: old master id'QF *
/opt/pgpoolII/log/failo$er.log1
failbac"%command = 1echo host'Qh: new master id'Qm: old master id'QF *
/opt/pgpoolII/log/failbac".log1
0
reco$er)%user = 1postgres1
reco$er)%1st%stage%command = 1cop)%base%bac"up1
reco$er)%2nd%stage%command = 1pgpool%reco$er)%pitr1
JouEll ha%e to reload pgpool to reflect the changes.
-erd /oenig 8/12 pgpool(!! for beginners
18.02.09
creating scripts used by online recovery
!n this step we ha%e to create the scripts which will be called at the two reco%ery stages3 and weE%e
defined in the pre%ious step. )opy the sa"ple scripts fro" /incoming/pgpool-II-2.1/sample to
G,-.949 */opt/postgres/data in this case+
copy_base_backup
4his script creates a checkpoint on node1 and copies the directory G,-.949 fro" node1 to node2.
9dditionally the file reco%ery.conf will be created *and copied+ to enable ,ostgre578 reco%ering to
the latest le%el at startup on node2.
Hnabled wal archi%ing on node10
archi$e%command = 1rs)nc Qp postgres7node2'/exchange/wal/Qf (/de$/null1
source of script copy'base'backup0
9; /bin/sh
psql -c =select pg%start%bac"up41pgpool-reco$er)15= postgres
echo =restore%command = 1cp /exchange/wal/Qf Qp1= * /opt/postgres/data/reco$er).conf
tar -A /opt/postgres/data -zcf pgsql.tar.gz base global pg%clog pg%multixact pg%subtrans pg%tblspc
pg%twophase pg%xlog reco$er).conf
psql -c 1select pg%stop%bac"up451 postgres
scp pgsql.tar.gz node2'/opt/postgres/data
pgpool_recovery_pitr
4his file perfor"s a switch'#log co""and on node1 to flush the latest transactions fro" buffer to
disk. !tEs si"ple source is0
9; /bin/sh
psql -c 1select pg%switch%xlog451 postgres
pgpool_remote_start
9fter copying the database files to node2 the database on this node has to be started. 5ince thereEs
the file reco%ery.conf in the G,-.949 directory post"aster will reco%er the database to the latest
known state and start it afterwards.
Dere we go0
9; /bin/sh
if M <9 -ne 2 N
then
echo =pgpool%remote%start remote%host remote%datadir=
exit 1
fi
8?>B=<1
8?>B8ID=<2
-erd /oenig 9/12 pgpool(!! for beginners
18.02.09
OGAB@=/usr/bin/pg%ctl
9 ?xpand a base bac"up
ssh -B <8?>B 1cd /opt/postgres/data6 tar zxf pgsql.tar.gz1 2*/de$/null 1*/de$/null ( /de$/null
9 >tartup Oostgre>I@ ser$er
ssh -B <8?>B <OGAB@ -w -8 <8?>B8ID start 2*/de$/null 1*/de$/null ( /de$/null #
initiate online recovery
8etEs see how we can start the online reco%ery auto"atis" and how the scripts fro" the pre%ious
step will be used.
4o re(attach a failed node you si"ply ha%e to call pcp%reco$er)%node co""and. !t is located in the
bin directory of the pgpool installation.
/opt/pgpoolII/bin/pcp%reco$er)%node 2. node1 -+-+ postgres postgres 1
,ara"eters are0
<20= 6 ti"eout in seconds
<node1= 6 hostna"e where pgpool is running
<9898= 6 port at which pgpool co""unication "anager listens
<postgres= <postgres= 6 userna"e password
<1= 6 !. of the node we want to attach *refers to the backend nu"ber in pgpool.conf+
4he state"ents produces a lot of log entries3 like0
8?H3G' pid !!11' pcp%child' start online reco$er)
@EG' pid !!11' starting reco$ering node 1
8?H3G' pid !!11' exec%chec"point' start chec"point
8?H3G' pid !!11' exec%chec"point' finish chec"point
@EG' pid !!11' AR?ALOEIJB in the 1st stage done
@EG' pid !!11' starting reco$er) command' =>?@?AB pgpool%reco$er)41cop)%base%bac"up1: 1infra.21: 1/opt/postgres/
data15=
8?H3G' pid !!11' exec%reco$er)' start reco$er)
8?H3G' pid 2-&+' starting health chec"ing
8?H3G' pid 2-&+' health%chec"' . the 8H node status' 2
8?H3G' pid 2-&+' health%chec"' 1 the 8H node status' ,
8?H3G' pid !!11' exec%reco$er)' finish reco$er)
@EG' pid !!11' 1st stage is done
@EG' pid !!11' starting 2nd stage
@EG' pid !!11' all connections from clients ha$e been closed
8?H3G' pid !!11' exec%chec"point' start chec"point
8?H3G' pid !!11' exec%chec"point' finish chec"point
@EG' pid !!11' AR?ALOEIJB in the 2nd stage done
@EG' pid !!11' starting reco$er) command' =>?@?AB pgpool%reco$er)41pgpool%reco$er)%pitr1: 1infra.21:
1/opt/postgres/data15=
8?H3G' pid !!11' exec%reco$er)' start reco$er)
8?H3G' pid !!11' exec%reco$er)' finish reco$er)
8?H3G' pid !!11' exec%remote%start' start pgpool%remote%start
-erd /oenig 10/12 pgpool(!! for beginners
18.02.09
8?H3G' pid 2-&+' starting health chec"ing
8?H3G' pid !!11' exec%remote%start' finish pgpool%remote%start
8?H3G' pid 2-&+' starting health chec"ing
@EG' pid !!11' 1 node restarted
@EG' pid !!11' send%failbac"%request' fail bac" 1 th node request from pid !!11
@EG' pid !!11' reco$er) done
8?H3G' pid 2-&+' failo$er%handler called
8?H3G' pid 2-&+' failo$er%handler' starting to select new master node
@EG' pid 2-&+' starting fail bac". reconnect host infra.24&!,25
@EG' pid 2-&+' execute command' echo nodeid'1: host'infra.2: port'&!,2: db-cluster path'/opt/postgres/data: new
master id'.: old master id'. * /opt/pgpoolII/log/failbac"1.log
8?H3G' pid !!11' pcp%child' recei$ed OAO pac"et t)pe of ser$ice 1S1
8?H3G' pid !!11' pcp%child' client disconnecting. close connection
@EG' pid 2-&+' failo$er%handler' do not restart pgpool. same master node . was selected
@EG' pid 2-&+' failbac" done. reconnect host infra.24&!,25
4he drawing ( online'reco%ery'theory( on the ne#t page will show you the <workflow= of online
reco%ery "echanis"0
-erd /oenig 11/12 pgpool(!! for beginners
18.02.09
-erd /oenig 12/12 pgpool(!! for beginners
8rawing 1' online%reco$er)%theor)