CONTENT DISTRIBUTION

TROUBLESHOOTING
Kerwin Medina Kerim Hanif
Senior Software Engineer Senior Program Manager
Microsoft Corporation Microsoft Corporation
kerwinm@microsoft.com Kerim.hanif@microsoft.com
Kerwin Medina Kerim Hanif

@TweetKerwin @kerimhanif

Surviving 1 kid Surviving 2 kids 

17 years in SCCM 20 years industry, 14 years MS

Loves eating Love cooking, travelling, scuba

WHAT MAKES THE WHEELS TURN?
Problem:
I have the engine running but my car won't move.

Engineer:
Did you check the starter?
SCENARIO: CREATE/UPDATE/DELETE PACKAGE
Problem:
I distributed the package to the DP, but my content status is not green.

Engineer:
Do you have a Network Access Account?
CONFIGURATION MANAGER CONTENT DISTRIBUTION
 ConfigMgr in general is a very complex product
 Content distribution is one of the most complex areas of the product
 90% of the ConfigMgr infrastructure are distribution points
 There can be many things that can affect to an outcome
 We want to share with you what Microsoft Support and Product team uses to troubleshoot
 Here are general guidelines that you should follow prior to troubleshooting
1. Understand issue and history
 Ask the right questions
2. Collect data
 Understand environment and document
FOLLOWING GUIDELINES
 Before starting to troubleshoot anything make sure you that you have the correct context
 Ask the right questions and document the issue
 What isn’t working? Has this ever worked? If so, when did it stop? Was anything changed in the
environment right before it stopped working?
 What is the frequency or pattern for the issue, is it still happening?
 Can you repro this issue at will?
 How did you become aware of the problem?
 What have you done so far (if anything) to fix it?
 How is the hierarchy configured? Note site codes, site server names.
 How many packages/DPs are affected? Note DP names, package IDs
 What is the current software distribution thread configuration?
LIKELY AREAS FOR CONTENT DISTRIBUTION ISSUES
 Distribution is in failed or progress state
 Content Status Issues
 Related to Distribution Points
 Related to Pull Distribution Points
 Content Library
 Content Removal
 Content Validation
 Distribution Point Installation, Upgrade and Configuration
 Other
 Distribution Point Groups
 Distribution Point Migration
 Pre-staged Content
 Cloud-Based Distribution Points
MOST IMPORTANT LOG FILES
 Hman.log
 DistMgr.log
 PkgXferMgr.log
 Pulldp_install.log
 SMSDPProv.log
 PullDP.log
MOST IMPORTANT COMPONENTS

Name Component Name Friendly Description

Name
Distribution Manager SMS_DISTRIBUTION_MANAGER DistMgr Manages content and creates jobs for PkgXferMgr
Package Transfer Manager SMS_PACKAGE_TRANSFER_MANAGER PkgXferMgr Transfers packages to distribution points
Hierarchy Manager SMS_HIERARCHY_MANAGER Hman Processes and replicates changes to the site hierarchy
Sender SMS_SENDER Sender Initiates inter-site communications across TCP/IP networks
Despooler SMS_DESPOOLER Despooler Processes incoming replication files from parent or child sites
Scheduler SMS_SCHEDULER Scheduler Creates sender jobs
Database Notification SMS_DATABASE_NOTIFICATION_MONITOR SmsDbMon Watches the database for changes to certain tables and creates files in the inboxes of
Monitor components responsible for processing those changes

SMS Provider SMS Provider SMSProv Windows Management Instrumentation (WMI) Provider that assigns read and write access to
the Configuration Manager database at a site

SMS DP Provider SMS DP Provider SMSDPProv Windows Management Instrumentation (WMI) Provider that manages Content Library
operations on the DP
UNDERSTANDING THREADS – SOFTWARE DISTRIBUTION
 Right-click the site under Sites node, and select Configure
Site Components > Software Distribution.
 Maximum number of packages
 Max number of packages can be processed by the DP
simultaneously
 Max number of DPs a package can be sent simultaneously

 Retry settings
 Max number of retries the site server makes in case of a
failure
 Delay between retries

 Multicast retry settings (OSD setting)

 Max number of retries the client makes in case of a failure
 Delay between retries
UNDERSTANDING THREADS – SOFTWARE DISTRIBUTION
UNDERSTANDING THREADS – SOFTWARE DISTRIBUTION
 Example:
 Central Administration Site: CS1
 Primary Site: PS1
 Number of regular Distribution Points reporting to PS1: 200
 Total number of packages: 1000

 With default values (3x5) PkgXferMgr can’t work on more than 15 jobs simultaneously
 Find a balance…!
 Important questions: How many packages do you foresee getting distributed simultaneously on an average? What is the
network bandwidth between site and DPs?
 If the answer to first question above is avg 5 pkgs to 200 DPs, you can theoretically set these values to 5x200 but that would mean
1000 threads
 More threads can slow down things instead of improving them recommendation is to stay <750 threads
 Possible config options for this example would be: 5x50, 5x75, or 5x100

 If setting up bunch of new DPs (means they will get 1000 pkgs transferred)
 Temporarily set 50x10, and go back to 5x50 etc.. when DPs are done
UNDERSTANDING THREADS – SITE TO SITE
 Navigate to Administration > Site Configuration > Sites
 Select the site and go to its properties
 Maximum concurrent settings
 All sites: The maximum number of simultaneous
communications allowed for this sender for all sites.
 Per site: The maximum number of simultaneous
communications allowed to any single destination site

 Retry settings
 Max number of retries the site server makes in case of a
failure
 Delay between retries
THE BIG (SHRUNK) PICTURE
 Source Source Source

P3 P1 P2 P4

P1 P3 P1 P1 P2 P4 P1 P2 P1 P4 P2
PullDP1 PRI1 CAS PRI2 DP2

DP2
(Source)
P1

SEC1 CLI2(P2P)

P1 P2 P4

DP1
CLI1(P2P)

P1 P3
P1P3
 Source Source Source

P3 P1 P2 P4

P1 P3 P1 P1 P2 P4 P1 P2 P1 P4 P2
PullDP1 PRI1 CAS PRI2 DP2

DP2
(Source)
P1

SEC1 CLI2(P2P)

P1 P2 P4

DP1
CLI1(P2P)

P1 P3
P1P3
MAIN SCENARIOS
1. Create/Update/Delete content
2. Distribute content to a Distribution Point (DP)
1. Site to Site
2. Site to DP
3. DP to Client and/or Client to Client
 Source Source Source

P3 P1 P2 P4

P1 P3 P1 P1 P2 P4 P1 P2 P1 P4 P2
PullDP1 PRI1 CAS PRI2 DP2

DP2
(Source)
P1
SEC1 CLI2(P2P)

P1 P2 P4

Source Processing flow DP1

CLI1(P2P)

P1 P3
P1P3
 CONTENT LIBRARY FLOW

PkgLib

User Package Source

(\\server\share\folder)
DataLib

Staging FileLib
CONTENT LIBRARY PROCESSING
 Create the package (PkgLib)
 For each source file
 Copy to temporary staging
 Hash
 Create entry on DataLib
 Move from temporary staging to FileLib
CONTENT SOURCE PROCESSING FLOW

Distribution SQL Tables:

Component site
Manager SMSPackages
control file settings
PkgStatus
ContentDPMap

User Package Source

Content Library
(\\server\share\folder)
SOURCE PROCESSING
Settings Status and Logs:
 Thread limits.  Distmgr.log
 Content Library Size/Location  Status Message
 Temp folders. Delay:
Distribution Manager  Insufficient threads.
 Follow thread limits.  Network and disk performance.
 Add each source file to Content Library.
Common Failure Areas:
 Update SMSPackages SQL Table with package
 Source access.
information (size, version, etc.).
 Network errors.
 Update PkgStatus SQL Table for local DP
with state pending.  Disk errors.
 Create sender jobs for each destination child
site, and update PkgStatus SQL Table with state
sent.
 Source Source Source

P3 P1 P2 P4

P1 P3 P1 P1 P2 P4 P1 P2 P1 P4 P2
PullDP1 PRI1 CAS PRI2 DP2

DP2
(Source)
P1

SEC1 CLI2(P2P)

P1 P2 P4

Site to Site flow DP1

CLI1(P2P)

P1 P3
P1P3
SITE TO SITE FLOW

SQL Object:
SMSPackages,
PkgStatus
DistMgr Scheduler Inbox

Component site
control file settings

Scheduler/ Despooler Inbox

Content Library Sender
SITE TO SITE PROCESSING
Sender (on file replication properties, site sender Delay:
properties)  Thread limits.
 Check for thread limits.  Available send window
 Check for send time windows.  Bandwidth limits
 Honor bandwidth limits.
Common Failure Areas:
 Send package and instruction.
 Network connections
Status and Logs:  Disk writes
 Sender.log
 Status Messages (sending from Site to Site)
 Source Source Source

P3 P1 P2 P4

P1 P3 P1 P1 P2 P4 P1 P2 P1 P4 P2
PullDP1 PRI1 CAS PRI2 DP2

DP2
(Source)
P1

SEC1 CLI2(P2P)

P1 P2 P4

Destination Site Processing DP1

CLI1(P2P)

P1 P3
P1P3
DESTINATION SITE PROCESSING FLOW

Despooler Content Library

Despooler Inbox

SQL Tables:
SMSPackages
PkgStatus Distribution Distribution Manager
ContentDPMap Manager Inbox
DESTINATION SITE PROCESSING
Despooler: Delay:
 Receive package payload and instruction  Disk performance.
from inbox.
 Slow SQL replication.
 Waits for SQL Replication of package
metadata (SMSPackages table). Common Failure Areas:

 Extract payload to Content Library.  Disk writes.

 Update the PkgStatus SQL Table with state  Corrupt package payload.
received.  SQL Replication Failure.
 Update stored version in the SMSPackages Admin Actions:
SQL Table.
 Do redistribute action.
Logs:
 Despool.log
 Status Messages
 Source Source Source

P3 P1 P2 P4

P1 P3 P1 P1 P2 P4 P1 P2 P1 P4 P2
PullDP1 PRI1 CAS PRI2 DP2

DP2
(Source)
P1
SEC1 CLI2(P2P)

P1 P2 P4

Site to DP DP1
CLI1(P2P)

P1 P3
P1P3
SITE TO DP FLOW

Content Library
DP Staging Folder
Package
Transfer
Manager

DP WMI
SQL Tables: Provider
SMSPackages
PkgStatus
ContentDPMap
SITE TO DP PROCESSING
Package Transfer Manager: Common Failure Areas:
 Subject thread limits.  Missing files in Content Library.

 Subject to Package and DP priorities.  Connection errors.

 Remote write errors.
 Subject to transfer rate limits.
 Hash mismatch.
 Copies files to DP staging folder.
Admin Actions:
 Do RDC delta.
 Update package (package action)
 Invokes DP WMI Provider methods
 Redistribute package (DP action)
Logs:
 Extract pre-stage file.
 PkgXferMgr.log  Move the DP FileLib folder
Delays:  Move the DP staging folder share (set permissions
 Thread limits, priorities and bandwidth limits. and re-register SMSDP.DLL)
 Use the content library transfer tool
 Source Source Source

P3 P1 P2 P4

P1 P3 P1 P1 P2 P4 P1 P2 P1 P4 P2
PullDP1 PRI1 CAS PRI2 DP2

DP2
(Source)
P1
SEC1 CLI2(P2P)

P1 P2 P4

Site to DP processing DP1

CLI1(P2P)

P1 P3
P1P3
DP PROCESSING

DP WMI
DP Staging Folder Provider Content Library
DP PROCESSING
DP WMI Provider:
 Update Content Library
Logs:
 SMSDPProv.log
Delays:
 Disk performance.
Common Failure Areas:
 Disk write errors.
Admin Actions:
 Move the DP FileLib folder
 Move the DP staging folder share (set permissions and re-register SMSDP.DLL)
 Redistribute package (DP action)
 Source Source Source

P3 P1 P2 P4

P1 P3 P1 P1 P2 P4 P1 P2 P1 P4 P2
PullDP1 PRI1 CAS PRI2 DP2

DP2
(Source)
P1
SEC1 CLI2(P2P)

P1 P2 P4

Pull DP Processing DP1

CLI1(P2P)

P1 P3
P1P3
PULL DP PROCESSING

DP WMI
Provider

Pull DP Management
Manager Point
Source DP
(in CCMexec) (State System
Endpoint)

Content Library
PULL DP PROCESSING
Pull DP Manager: Failure Areas:
 Download package from source DPs.  Disk write errors.
 Update Content Library  Network errors.
 Update Package Status with State Message via  Co-location with CCMExec
MP.
Admin Action:
Logs:  Move the DP FileLib folder
 PullDP.log  Move the DP staging folder share (set
 SMSDPProv.log permissions and re-register SMSDP.DLL)
Delays:
 DP network connection.
 Disk performance.
 MP network connection and MP load.
 Source Source Source

P3 P1 P2 P4

P1 P3 P1 P1 P2 P4 P1 P2 P1 P4 P2
PullDP1 PRI1 CAS PRI2 DP2

DP2
(Source)
P1
SEC1 CLI2(P2P)

P1 P2 P4

DP to Client DP1
CLI1(P2P)

P1 P3
P1P3
DP TO CLIENT FLOW

CAS (Content
Access Service) CCM Cache

CTM (Content WMI

Transfer (SoftmMgmtAgent -
Manager) CacheInfoEx)

DTS
(Data Transfer
Service)

IIS DP ISAPI BITS Temp

DP TO CLIENT PROCESSING
Component: Failure Areas:
 Content Access Service
 Authentication (Network Access Account)
 Content Transfer Manager
 Data Transfer Service  Network connectivity (resolution,
connection, firewall)
Logs:  Disk space
 CAS.log  WMI corruption
 ContentTransferManager.log
Admin Action:
 DataTransferService.log
 Relocate CCM cache
Delays:
 Reinstall client (WMI corruption)
 BITS throttling
 Network performance
 Verification (hash check)