TroubleshootingPuppet

Enterprise
Celia Cottle
Support Engineer | Puppet Labs
celia@puppetlabs.com
@celiaPDX
TheStack
Console
The console is Puppet Enterprises web GUI.
Mcollective/Live Management
LM is an interface to PEs orchestration engine (Mcollective).
PuppetDB
PuppetDB collects data generated by Puppet.
Master/Agent
The central puppet server/ Retrieves the client conguration
from the puppet master and applies it to the local host
TheConsole
Console
Logs
/var/log/pe-httpd/puppetdashboard.error.log
/var/log/pe-httpd/puppetdashboard.access.log
/var/log/pe-httpd/puppetmaster.error.log
Conguration
/etc/puppetlabs/puppet/puppet.conf
No nodes are reporting
Console
CommonProblems
Stop the pe-puppet-dashboard-workers
Check opt/puppet/share/puppet-dashboard/tmp/pids for les ending in .pid.
Restart the pe-puppet-dashboard-workers.
Run ps aux | grep delayed_job and see if entries like dashboard/delayed_job.1 and
delayed_job.1_monitor appear. If they are, that means the dashboard has started
up properly again.

Console
CommonProblems
Theres No Facts Listed For Nodes
/Node Manager Wont Display
/var/log/pe-httpd/puppetmaster.error.log
!"#$ &'( )* ++,-.,+/ +/)01 !2##3#1 !45$267 )8+9)*9/9+1
:2#7$;$4<72 =2#$;$4<7$36, >##3# ?+0@, 42#7$;$4<72 #2A3B2C
ConsoleAuthentication
Logs
/var/log/pe-httpd/access.log
/var/log/pe-httpd/error.log /var/log/pe-console-auth/
cas.log
Conguration Files
/etc/puppetlabs/console-auth/cas_client_cong.yml
/etc/puppetlabs/rubycas-server/cong.yml
ConsoleAuth
CommonProblems
Cant Log In
/var/log/pe-console-auth/cas.log:
Invalid credentials given for user 'console@puppetlabs.test'
Possible Cause: Bad Credentials/Lost Credentials
$ cd /opt/puppet/share/console-auth
$ sudo /opt/puppet/bin/rake db:create_user
USERNAME="adminuser@example.com"
PASSWORD="<password>" ROLE="Admin
Alternatively, if using 3rd Party Auth:
/var/log/pe-httpd/access.log
PuppetDB
PuppetDB

Log Files:
/var/log/messages
/var/log/pe-puppetdb/puppetdb.log
Cong Files:
/etc/puppetlabs/puppet/puppetdb.conf
PuppetDB
CommonProblems

SSL Errors
* /var/log/messages
>##3#, :3'5C 637 #27#$2A2 4<7<53( ;#3D #2D372
E2#A2#, >##3# -// 36 F>G=>G, "<$52C 73 E'HD$7
I#2J5<42 ;<47EI 43DD<6C ;3# <(267)9AD 73
K'JJ27LM <7 D<E72#/9AD,N/N), F2#A2# O3E76<D2
ID<E72#/9ADI C$C 637 D<74O E2#A2# 42#7$;$4<72P
2QJ2472C 362 3; D<E72#)9AD
Puppetdb
CommonProblems
PuppetDB Wont Start, Fails Silently
/var/log/pe-puppetdb/puppetdb.log
***/var/log/pe-puppetdb/puppetdb-oom.hprof
R<A<95<6(9S'7S;T2D3#U>##3#, V<A< O2<J EJ<42
Fix:
Edit the defaults in /etc/default/pe-puppetdb or /etc/
syscong/pe-puppetdb, and change the 256m to 1024m
JAVA_ARGS="-Xmx256m -XX:
+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/
var/log/pe-puppetdb/puppetdb-oom.hprof -Xms256m"
LiveManagement
LiveManagement
/Mcollective

Logs:
/var/log/pe-activemq/activemq.log
/var/log/pe-mcollective/mcollective.log
/var/log/pe-httpd/error.log
Conguration:
/etc/puppetlabs/mcollective/server.cfg
Mcollective
CommonProblems
* None of the Nodes Show Up In Live Management
/var/log/pe-httpd/error.log
W3 T:355247$A2 E2#A2#E #2EJ36C2C9 >$7O2#
T:355247$A2 $E 637 U27 436;$('#2C <6C
3J2#<7$36<5 3# <55 T:355247$A2 E2#A2#E <#2
3;;X5$629 :O24B 7O<7 U3' 4<6 #2<4O U3'#
E2#A2#E Y$7O ZD43 J$6(Z9 [7 D<U <5E3 O25J 73
$64#2<E2 7O2 \T]L[F:S=>G^]_[T>S`_ 3#
\T][W=>W_SG^]G>_G[>F A<#$<H52E $6 U3'# &J<4O2
436;$('#<7$369
Live Management
Common Problems And What They Look Like
* None of the Nodes Show Up In Live Management
/var/log/pe-activemq/activemq.log
a b&GW a _#<6EJ3#7 :366247$36 73, 74J,cc///9//9///9//,////
;<$52C, R<A<95<6(9F24'#$7U>Q42J7$36, `E2# 6<D2 !D4355247$A21
3# J<EEY3#C $E $6A<5$C9
Mcollective
Common Problems

* The Number of Nodes reporting from
MCollective commands, or Live Management,
varies
/var/log/pe-activemq/activemq.log
R<A<Q96279EE59FF\d<6CEO<B2>Q42J7$36, G2D372 O3E7
453E2C 4366247$36 C'#$6( O<6CEO<B2
Solution:
On the master, edit:
c3J7cJ'JJ27cEO<#2cJ'JJ27cD3C'52EcJ2]D4355247$A2cE2#A2#94;(92#H
<6C 2C$7 7O2 5$62 #2($E72#$672#A<5 e
LiveManagement
Common Problems And What They Look Like
* Nothing displays but a 500 error
Master/Agent

Logs:
* /var/log/messages
* /var/log/pe-httpd/error.log
Conguration:
/etc/puppetlabs/puppet/puppet.conf
Master/Agent
Common Problems And What They Look Like

* Nodes are failing runs
/var/log/messages
err: /File[/var/opt/lib/pe-puppet/lib]: Failed to generate
additional resources using 'eval_generate: Connection
timed out - connect(2)
err: Could not retrieve plugin: execution expired
Solution:
Splay:
http://docs.puppetlabs.com/references/latest/
conguration.html#splay
Master/Agent
Common Problems And What They Look Like

* Nodes are failing runs
var/log/messages
>##3#, :3'5C 637 #2f'2E7 42#7$;$4<72, _O2 42#7$;$4<72 #27#$2A2C
;#3D 7O2 D<E72# C32E 637 D<74O 7O2 <(267IE J#$A<72 B2U9
_3 ;$Q 7O$Eg #2D3A2 7O2 42#7$;$4<72 ;#3D H37O 7O2 D<E72# <6C 7O2
<(267 <6C 7O26 E7<#7 < J'JJ27 #'6g YO$4O Y$55 <'73D<7$4<55U
#2(262#<72 < 42#7;$4<729
S6 7O2 D<E72#,
J'JJ27 42#7 452<6 <(2676<D2
G2E7<#7 J2XO77JC
S6 7O2 <(267,
#D X; c274cJ'JJ275<HEcJ'JJ27cEE5c42#7Ec<(2676<D2
J'JJ27 <(267 X7

Master/Agent
Common Problems And What They Look Like
* Nodes cant reach the master
>##3#, :3'5C 637 #2f'2E7 42#7$;$4<72, (27<CC#$6;3,
W<D2 3# E2#A$42 637 B63Y6
Troubleshooting
1. telnet master 8140
2. Check /etc/hosts or DNS
3. ping master
RedHerrings
/var/log/pe-httpd/error.log
436;$(9#',., Y<#6$6(, <5#2<CU $6$7$<5$h2C
436E7<67 <#(A
var/log/pe-httpd/puppetdashboard.error.log
!Y<#61 GF& E2#A2# 42#7$;$4<72 :3DD36W<D2 ?:W@
ZJ2X$672#6<5XC<EOH3<#CI C32E WS_ D<74O E2#A2#
6<D2ij
/var/log/pe-console-auth/auth.log
[W"S +/)0X/NX+/ /),/8 `_:, `E2# ?<636UD3'E@
<442EE2C #2<CXY#$72 '#5 c#2J3#7Ec'J53<C

SSL Errors
Where your certs (mostly) live:
/etc/puppetlabs/puppet/ssl
/opt/puppet/share/puppet-dashboard/certs
/etc/puppetlabs/puppetdb/ssl
RegeneratingTheCAAndTheMaster
1. Delete the contents of /etc/puppetlabs/puppet/ssl directory on the
master.
2. Run `puppet cert list` to regenerate the CA.
3. Stop pe-httpd.
4. Run `puppet master --no-daemonize --verbose` to regenerate the
master cert and create a cert request.
5. Check that puppet cert list -a returned the master cert.
6. Restart pe-httpd.
RegeneratingthePuppetDBCerts
1. Stop the PuppetDB service
2. Remove agent certs from/etc/puppetlabs/puppet/ssl/ if on a separate
server and the PuppetDB ones from /etc/puppetlabs/puppetdb/ssl/
3. Run `puppet cert clean puppetdbhost.yourdomain` on the master (if not
cleaned already and on a separate host)
4. Regenerate the Puppet Agent certs by performing a Puppet run on the
PuppetDB, signing them on the master if necessary.
5. Run /opt/puppet/sbin/puppetdb-ssl-setup -f on thePuppetDB host.
6. Restart the PuppetDB service on its host, and the pe-httpd service on
your master.
RegeneratingTheConsoles
Certificate
1. cd /opt/puppet/share/puppet-dashboard/certs, and
remove any existing contents.
2. sudo /opt/puppet/bin/rake RAILS_ENV=production
cert:create_key_pair
3. sudo /opt/puppet/bin/rake RAILS_ENV=production
cert:request
4. sudo puppet cert sign pe-internal-dashboard
5. sudo /opt/puppet/bin/rake RAILS_ENV=production
cert:retrieve
6. sudo chown -R puppet-dashboard:puppet-dashboard
certs/
7. /etc/init.d/pe-httpd restart
RegeneratingTheAgents
Certificate
On the master:
1. puppet cert clean agenthostname
2. Restart pe-httpd
On the agent:
1.rm -rf /etc/puppetlabs/puppet/ssl
2. puppet agent -t
On the master:
1. puppet cert sign agenthostname
RegeneratingYourMasters
Certificate
1. Edit your puppet.conf to update any changes
to the hostname or alt names.
2. `puppet cert clean mastername`
3. Stop pe-httpd?c274c$6$79CcJ2XO77JC
E73J@.
4. Run `puppet master --no-daemonize --
verbose.
CertsthatPuppetcanRegenerate
pe-internal-broker
pe-internal-mcollective-servers
pe-internal-peadmin-mcollective-client
pe-internal-puppet-console-mcollective-client
RegeneratingAllTheCertificates
http://showterm.io/f41a4b7bb5b0b006d8a80
Q&A
Resources
Ask.Puppetlabs.com
Irc.freenode.net
#puppet
PE-Users Mailing List:
https://groups.google.com/a/puppetlabs.com/
group/pe-users/topics