The Definitive Guide To SPSLogging

The Definitive Guide to SharePoint Logging and Auditing
The Definitive Guide to SharePoint Logging and Auditing A Technical Support and User Guide for SharePoint Portal Server and Windows SharePoint Services
2006 David M. Sterling All Rights Reserved
Created by David M. Sterling Sterling International Consulting Group February 2006
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group
Page 1 of 25 February 2006
Table of Contents Event Logging in SharePoint ...........................................................................................3 SharePoint Log Timing............................................................................................3 Enabling SharePoint Logging ..................................................................................4 Parsing SharePoint Logs..........................................................................................4 IIS Log information and SharePoint Logs ................................................................5 SharePoint Transactions ..............................................................................................6 Noise Transactions in SPS Logs ...........................................................................6 Document Transactions ...........................................................................................6 List Transactions .....................................................................................................6 IIS Logging .................................................................................................................7 Logging Elements (W3C extended format) ..............................................................7 Setting up Logging ..................................................................................................7 Logging with ODBC..............................................................................................10 Performance Considerations ..................................................................................13 Understanding IIS Logs for SharePoint......................................................................15 Noise Transactions .............................................................................................15 Detecting when a User has selected a New Document............................................15 Opening a file in READ ONLY mode ...................................................................17 Opening a File for Editing .....................................................................................17 Uploading a File ....................................................................................................19 List Operations ......................................................................................................19 Cross Referencing Logs.........................................................................................19 Calculating SharePoint Statistics ...................................................................................20 Determining number of Downloads a user has Performed..........................................20 Using IIS Logs ......................................................................................................20 Using SharePoint Logs ..........................................................................................20 Calculating Storage Usage.........................................................................................21 Calculating Document Library Storage ..................................................................21 Calculating Attachments Storage ...........................................................................22 Calculating Usage For a Site..................................................................................22 Counting the Number of Immediate Alerts Active in the Portal..............................22 Counting the Number of Scheduled Alerts Active in the Portal..............................22 Obtaining Sites a User Belongs To ........................................................................23 Obtaining Site Members ........................................................................................23 Obtaining User Profile Information........................................................................23 Appendix A...................................................................................................................24 Source listing for SharePoint Log Parser....................................................................25 SPSLogParser XML Settings File..............................................................................25 SPS Log Parser Table Definition ...............................................................................25 SPS Log Parse Table Bulk Insert Command ..............................................................25
Event Logging in SharePoint

Logging within SharePoint is minimal to reduce overhead and increase performance. In a secured intranet environment, nearly all logs, including IIS can be turned off. For most use, the most important aspect is usage information which is available through SharePoints object model. However, even Microsoft has commented that the Object Model functions such as GetUsageData and GetUsageBlob, part of the SPWeb Class. However Microsoft warns about using these: GetUsageData for example returns different formation depending on the type of report or period and has a restriction of returning only 2000 rows making it of limited use in monitoring site usage. GetUsageBlob returns the same data and does not have a limit, but this is very difficult to parse. Both WSS and SPS generate Usage Event logs on a daily basis (per virtual server) when logging is enabled. When enabled, SharePoint will create log files automatically to the default location this is usually: \%windir%\system32\LogFiles\STS\<virtual server GUID> Under this folder, a sub-folder is created for each day of logging (the folder name is the date). Technical Tip: While most systems will have only one or two directories under the STS Directory, if a multiple use system (SPS, WSS, etc.) there may be several. To determine the correct GUID that belongs to the site, you can easily go to SQL Server Enterprise Manager, open the SPS Configuration Database, right click on the VirutalServers and select Open > Return All Rows. You can easily find the name of the Portal instance and the ID associated with it. If you intend to do a considerable amount of logging for a period of time you should relocate the Logging files to a different device other that the Windows directory. Please see the Performance Considerations section.
SharePoint Log Timing

When using SharePoint Logging, be aware that it can take several seconds for transactions to appear. Unlike IIS which writes directly to the Log, there can be a delay in SharePoint from several seconds to an indefinite period. In some cases, SharePoint may not post the log until after the user has disconnected.
Enabling SharePoint Logging

Logging in SharePoint is NOT enabled by default. Logging is done on a Portal or Site wide basis this means you cannot enable logging for a single Virtual Server. You should take this into account when you consider storage requirements. To enable logging, you must go to SharePoint Central Administration and under Component Configuration, select Configure usage analysis processing. Check the Enable logging check box and set the Log File Location. By default this will point to the Windows LogFiles folder on the default system drive. You should change this to point to a more appropriate location as shown:
As mentioned under Performance Considerations, the directory/folder you specify should have adequate rights granted to the IIS_WPG, STS_WPG, SharePoint Services and SharePoint Administration accounts.
Parsing SharePoint Logs

Once created, you can create a Parser to parse the logs and turn them into database table information. By default, there is a parser (unsupported) available in the Microsoft Article here: http://msdn.microsoft.com/library/default.asp?url=/library/enus/odc_SP2003_ta/html/ODC_WSSUsageEventLogging.asp. A downside of this parser is that it only outputs the bare minimum information in fact, not much more to go on than using IIS itself to include the following:
Time Stamp Site GUID Site Name Web Name Document Name (of the request, ASPX page, Document, etc.) User Login The Date of the log must be determined by the Folder name. Key elements missing include the previous indicator and the Flags indicating the type of hit. In Appendix A you will find a rewritten version that supports automatically bulk loading or inserting directly into the database. In addition, the output is more useful as shown: [siteGuid] GUID of the site (cross reference for the VirtualServers table in the SPS Configuration Database) [time] Time Stamp of the event (in SPS, the Folder Name contains the Date) [siteUrl] Site URL of the request [web] Web Name of the current sub web [doc] Document requested an ASPX document or actual document name (with full HTTP path) [username] User Login name (Domain\Username) [useragent] User Agent information same as IIS [referrer] Referrer information same as IIS [querystring] Query string passed same as IIS [bitFlags] Type of hit 0 = Regular hit, 1 = Used by Office Front Page, 2 = List Update, 4 = List Operations or 8 = Discussion request made through OSE (Office Server Extensions) Discussion button in IE
IIS Log information and SharePoint Logs

IIS Logs (discussed in the next section) can be used to supplement the information provided by SharePoint, specifically Server IP Address, Port Number, etc. More on this following the IIS Logging Section.
SharePoint Transactions
Noise Transactions in SPS Logs
Like IIS below, you can usually omit any entries that are based on the SharePoint Services or Administration account. In addition, you can eliminate default Search entries by checking the Agent for Search the agent is listed as:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Document Transactions
Transaction Type refers to the Bit Flags set in the log file (0, 1, 2, 4 or 8) When a document is opened for editing of the meta data: A Type 1 transaction with Doc = Document Name A Type 1 transaction with Doc = edtiform.aspx A Type 2 transaction with Doc = Library Name When a document is opened for editing: A Type 1 transaction with Doc = Document Name A Type 1 transaction with Doc = owssvr.dll When a document is saved from MS Office to SharePoint, four type 1 transactions occur: The first to owssrvr.dll The second to owssvr.dll with User Agent set to 'Test for Web Form Existence' The third to owssvr.dll The fourth with the 'doc' column set to the document name being saved When a document is Uploaded to the site: A Type 1 transaction with Doc = Upload.aspx A Type 1 transaction with Doc = Document Name uploaded When a document is Deleted: A Type 4 transaction with Doc = Document Library URL (without the file name)
List Transactions
List transactions are easily determined by the Type of transaction: Type 2 indicates the list was updated Type 4 indicates List Operations (save and delete)
SPS/WSS Logging and Auditing Guide 2006 SICG ALL RIGHTS RESERVED Created by David M. Sterling, Sterling International Consulting Group Page 6 of 25 February 2006
IIS Logging
To support SharePoint auditing, you will need to incorporate information from IIS Logs. There are two ways in which to process these, either by bulk loading the logs into SQL Server or by integrating IIS ODBC logging to add the data immediately. Note that while the better method is to use Bulk Load due to processing time, the ODBC method is preferred for immediate logging and monitoring. Subsequent analysis can use the bulk load method to get a more complete picture. IIS log files are delimited text files that follow the specification RFC2616, Hypertext Transfer Protocol HTTP/1.1 (http://www.rfc-editor.org/rfc/rfc2626.txt).
Logging Elements (W3C extended format)

IIS can log nearly any part of an HTTP request but for monitoring purposes, the following logging elements (some of these are advanced settings) are most typically needed: date date of access time time of access c-ip client IP cs-username client user name cs-method the HTTP method for the request that was met cs-uri-stem the document being requested cs-uri-query the query string sent as part of the request sc-status status code returned by the Server processing request sc-bytes bytes sent back to the client for the request time-taken time to process in milliseconds cs(User-Agent) - agent cs(Cookie) cookie or persistent data in the request cs(Referrer) URL of the previous site visited by the user Note: some of the fields are quite large and can take up significant space; use only the columns needed.
Setting up Logging
You must enable IIS logging and setup the fields desired using Internet Information Services Manager. Within IIS, right click on the web site to be monitored and select a Logging Option as shown:
The two options of interest are the W3C Extended Log File Format or ODBC Logging. After you have selected the method, you must select Properties to set which fields will be logged.
By default, these fields will be WRONG you must set them to match what you want to log and at the same time, you must also setup the database to hold the items you select. Based on the base list shown here, the basic SQL Script to create a capture table is as follows:

DROP TABLE [IISLoggingFormatAdv] go CREATE TABLE [dbo].[IISLoggingFormatAdv] ( [date] [datetime] NULL, [time] [datetime] NULL, [cs-method] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [cs-uri-stem] [varchar] (2048) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [cs-uri-query] [varchar] (2048) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [cs-username] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [c-ip] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [cs(User-Agent)] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [cs(Cookie)] [varchar] (2048) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [cs(Referrer)] [varchar] (2048) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [sc-status] [int] NULL, [sc-win32-status] [float] NULL, [sc-bytes] [int] NULL, [time-taken] [int] NULL ) ON [PRIMARY] GO
NOTE: You may encounter an error in SQL Query Analyzer running this script this is because the record is longer than recommended. If this occurs, you must create the table using SQL Enterprise Manager. You can map these one to one to the fields selected in IIS Logging:
NOTE: the order of the fields will depend on your installation to be sure of the order in which your fields are being saved, you must check the Log file itself and locate the #Fields line:
#Fields: date time cs-method cs-uri-stem cs-uri-query cs-username c-ip cs(User-Agent) cs(Cookie) cs(Referer) sc-status sc-win32-status sc-bytes time-taken
To load the Log file into SQL Server, you must prepare the Log file for import. The first four lines of the Log file contains # as the starting character (for example, the fields line above). These must be removed or the Bulk Insert will fail. These lines can be edited out manually or you can use the Microsoft Pre-built tool for this located at http://support.microsoft.com/default.aspx?kbid=296093. NOTE: The field SC-Win32-Status data type must be either Float or BigInt to accommodate the status fields. Once the file has been prepared, it must be bulk loaded into SQL Server via the Query Analyzer as:
BULK INSERT [dbo].[IISLoggingFormatAdv] FROM 'C:\mystore\PreppedLog.log' WITH ( FIELDTERMINATOR = ' ', ROWTERMINATOR = '\n' )
WARNING: LOG TIMES USE GMT TIME (i.e. 9am EST = 2pm GMT).
Logging with ODBC

If Using ODBC, you must setup the ODBC data source first. Using SQL, you must first create the database as above and the ODBC table format is slightly different. The following table describes the fields and data types for an ODBC transaction log file: Field name ClientHost Username LogTime Service Machine ServerIP BytesRecvd BytesSent ServiceStatus Win32Status Operation Data type varchar(255) Client IP address varchar(255) Client domain name datetime Connection date and time varchar(255) Internet Information Services (IIS) service varchar(255) Computer name varchar(50) Server IP address Processing time in milliseconds Bytes received by server Bytes sent by server Simple Mail Transfer Protocol (SMTP) protocol reply code Windows Server 2003 status or error code (a 0 value indicates success) integer integer integer integer Description
ProcessingTime integer
varchar(255) SMTP protocol command

Target Parameters
varchar(255) Recipient varchar(255)
The script for this is as follows:

if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[IISODBCLogFile]') and OBJECTPROPERTY(id, N'IsUserTable') = 1) drop table [dbo].[IISODBCLogFile] GO CREATE TABLE [dbo].[IISODBCLogFile] ( [ClientHost] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL , [UserName] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL , [LogTime] [datetime] NULL , [Service] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL , [Machine] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL , [ServerIP] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL , [ProcessingTime] [int] NULL , [BytesRecvd] [int] NULL , [BytesSent] [int] NULL , [ServiceStatus] [int] NULL , [Win32Status] [int] NULL , [Operation] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL , [Target] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL , [Parameters] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ) ON [PRIMARY] GO
NOTE: You may encounter an error in SQL Query Analyzer running this script this is because the record is longer than recommended. If this occurs, you must create the table using SQL Enterprise Manager. Next, you need to create a data source within Windows - this is done via the Administration tool called Data Sources (accessible via Start > Administrative Tools > Data Sources (ODBC). Define a new System DSN that points to your Database including the Login credentials needed. Once defined, you must tell IIS to log via ODBC. Via the Internet Information Services Manager, right click on the Web Site and from Web Site Tab, select ODBC Logging. You will be prompted to enter the ODBC Data Set Name (DSN) as shown:
For example, our DSN is called WebLog, has a table name called IISLoggingFormatAdvForODBC and will use the account SPSAdmin:
Note: the account you use should have either DBO access to the database or must have at least db_ddladmin, db_datareader and db_datawriter privledges. After you have setup ODBC, you must test it to be sure it is operating correctly. If you do not see entries recorded, check the Event Viewer under the System log and look for any IIS errors.
Most common problems: User login is incorrect in the DSN definition in Data Sources User does not have sufficient privileges to write to the DB DSN is setup to point to the MASTER database (change to point to the correct default database where the logs are to be written
Be aware that ODBC logging does NOT PROVIDE you with all of the information needed for tracking SharePoint activities two fields, cs(Agent) and cs-uri-query are needed to see file information. See Cross Referencing Logs.
Performance Considerations
There are two sticking points with logging in both IIS and SharePoint first, by default, IIS will create logs in the default Windows folder this is not a good idea for long term logging. This saps precious OS resources and most importantly, disk space. This is also true of SharePoint which uses the LogFiles directory as well. Because of this, it is highly suggested that you setup a separate disk area to contain log files. This should be big enough to accommodate the large files and most important: all IIS/SPS accounts must have FULL access to the folder. This includes process accounts: IIS_WPG, SPS_WPG and if available STS_WPG, Administrative accounts: Administrators, Administrator and SharePoint Accounts: SPS Services and SPS Administrator. In general, logging will always add overhead to IIS operations and naturally, using offloaded files and bulk importing into SQL Server can be much more efficient than
ODBC as it will use less system resources and can be offloaded to perform the work without impacting users and is best if you are logging primarily for reporting. ODBC has the advantage of being instantaneous; if you intend to do near real-time, you must use ODBC. Be aware however, that IIS ODBC is not always a perfect connection; in practice, it is common for IIS to skip recording then resume for unknown reasons. If logging is a critical factor, ODBC should be avoided.
Understanding IIS Logs for SharePoint

Whether you use IIS Logging via SQL or another tool such as Log Parser, the following guide shows you how to detect when an action in SharePoint has occurred. For simplicity, we reference the ODBC Logging table form however the requests and tests are the same regardless of the log format. One major point not all SharePoint activities are traceable through IIS Logs there are some
Noise Transactions in IIS

In general, you can omit all references made by the SharePoint Administration account as it is usually performing background crawls and other maintenance type activities. This is also true for the SharePoint Services account if a separate one is used (as it should be). You can also eliminate a number of entries automatically that call the SPS Crawl Web Service (/_vti_bin/spscrawl.asmx); this type of transaction kicks off on the Content Search schedules and produces 10-20 transactions; by default this is every 10 minutes on a Portal Server. A trigger on the database table or filtered select will help eliminate these. Another call that can be eliminated is /_vti_bin/help/1033/sps/html/helpresources.htm.
Detecting when a User has selected a New Document

Note: The order of these may change, however they are always executed in groups so you can correctly identify the transaction by selecting the entries based on Log Time and checking that all steps (in whatever order) have been detected. There are a total of 18 entries made for creating a new document; the order varies however the entire scope of the transaction always covers a time span of 2 seconds. This is easily accomplished by selecting any group of 18 transactions that have a 2 second time span and includes the Author.DLL call. For example:
SELECT Count(*),LogTime FROM [WebLog].[dbo].[IISODBCLogFile] WHERE ([UserName] = '-' OR [UserName] = 'DEMO\dsterling') GROUP BY LogTime
IIS Log entries when a new document opened:

1) A POST request with UserName set as '-' to author.dll (/_vti_bin/_vti_aut/author.dll) 2) A GET request with UserName set as '-' with a Win32Status of -2146893042 to owssvr.dll (/_vti_bin/owssvr.dll)

3) A GET request with UserName filled to owssvr.dll (/_vti_bin/owssvr.dll) 4) A POST request with UserName set as '-' to shtml.dll (/_vti_bin/shtml.dll/_vti_rpc) 5) A POST request with UserName filled to shtml.dll (/_vti_bin/shtml.dll/_vti_rpc) 6) A GET request with UserName set as '-' with a Win32Status of 0 to owssvr.dll (/_vti_bin/owssvr.dll) 7) A POST request with UserName filled to shtml.dll (/_vti_bin/shtml.dll/_vti_rpc) 8) A POST request with UserName set as '-' to shtml.dll (/_vti_bin/shtml.dll/_vti_rpc) 9) A POST request with UserName set as '-' to author.dll (/_vti_bin/_vti_aut/author.dll) 10) A POST request with UserName filled to author.dll (/_vti_bin/_vti_aut/author.dll) 11) A POST request with UserName filled to author.dll (/_vti_bin/_vti_aut/author.dll) 12) A POST request with UserName set as '-' to author.dll (/_vti_bin/_vti_aut/author.dll) 13) A POST request with UserName filled to author.dll (/_vti_bin/_vti_aut/author.dll) 14) A GET request with UserName set as '-' with a Win32Status of -2146893042 to owssvr.dll (/_vti_bin/owssvr.dll) 15) A GET request with UserName set as '-' with a Win32Status of 0 to owssvr.dll (/_vti_bin/owssvr.dll) 16) A GET request with UserName filled to owssvr.dll (/_vti_bin/owssvr.dll) 17) A POST request with UserName set as '-' to author.dll (/_vti_bin/_vti_aut/author.dll) 18) A POST request with UserName filled to author.dll (/_vti_bin/_vti_aut/author.dll)
If User does NOT save the document, this will be followed by:
1) A GET request with the UserName filled for the original page user started on (for example if in Area Auto's Document Library, the URL would be /Auto/Document Library/Forms/AllItems.aspx) 2) A GET request with the exact same time as #1, UserName set as - to the same page from #1 returning a 401 and Win32Status of 0
3) A GET request with the exact same time as #1, UserName set as - to the same page
from #1 returning a 401 and Win32Status of -2146893042
NOTE: THERE MAY ALSO BE A REQUEST for /_layouts/images/headcornerp.gif - this is corner image of the Search Box. If the User SAVES the document, this will be followed by:

1) A HEAD request with the UserName set as '-', a Win32Status of -2146893042 and Parameters set (i.e. location=<doc lib location>) to owssvr.dll 2) A HEAD request with the UserName set as '-', a Win32Status of 0 and Parameters set (i.e. location=<doc lib location>) to owssvr.dll
3) A HEAD request with the UserName filled and Parameters set (i.e. location=<doc lib
location>) to owssvr.dll
Detected when a User has opened a document in Read Only mode
Opening a file in READ ONLY mode

NOTE that the open for read transaction is a time span of 2 seconds:
1) A POST request with the UserName set as '-' to author.dll (/_vti_bin/_vti_aut/author.dll) 2) A POST request with the UserName filled to author.dll (/_vti_bin/_vti_aut/author.dll) 3) A POST request with the UserName set as '-' to author.dll (/_vti_bin/_vti_aut/author.dll) 4) A POST request with the UserName filled to author.dll (/_vti_bin/_vti_aut/author.dll) 5) A POST request with the UserName set as '-' to author.dll (/_vti_bin/_vti_aut/author.dll) 6) A POST request with the UserName filled to author.dll (/_vti_bin/_vti_aut/author.dll) 7) A POST request with the UserName set as '-' to webs.asmx (/_vti_bin/webs.asmx) 8) A POST request with the UserName filled to webs.asmx (/_vti_bin/webs.asmx) 9) A POST request with the UserName set as '-' to dws.asmx (/_vti_bin/dws.asmx) 10) A POST request with the UserName filled to dws.asmx (/_vti_bin/dws.asmx)
NOTE: YOU MUST CROSS REFERENCE TO DETERMINE WHICH FILE WAS OPENED.
Opening a File for Editing

NOTE that the edit transaction is a time span of 3 seconds: Transactions usually set only once when the user first edits a document on the page:
1) A GET with UserName set as '-' and Win32Status of -2146893042 to Menu.htc (/_layouts/1033/Menu.htc)

2) A GET with UserName set as '-' and Win32Status of 0 to Menu.htc (/_layouts/1033/Menu.htc) 3) A GET with UserName filled to Menu.htc (/_layouts/1033/Menu.htc) 4) A GET with UserName filled to Checkout.gif (/_layouts/images/checkout.gif) 5) A GET with UserName set as '-' and Win32Status of -2146893042 to versions.gif (/_layouts/images/versions.gif) 6) A GET with UserName set as '-' and Win32Status of 0 to versions.gif (/_layouts/images/versions.gif) 7) GET with UserName set as '-' and Win32Status of -icdisc to icdisc.gif (/_layouts/images/icdisc.gif) 8) GET with UserName set as '-' and Win32Status of 0 to icdisc.gif (/_layouts/images/icdisc.gif) 9) GET with UserName filled to versions.gif (/_layouts/images/versions.gif) 10) GET with UserName filed to icdisc.gif (/_layouts/images/icdisc.gif)
The following occur just as with Read mode:

11) A POST request with the UserName set as '-' to author.dll (/_vti_bin/_vti_aut/author.dll) 12) A POST request with the UserName filled to author.dll (/_vti_bin/_vti_aut/author.dll) 13) A POST request with the UserName set as '-' to author.dll (/_vti_bin/_vti_aut/author.dll) 14) A POST request with the UserName filled to author.dll (/_vti_bin/_vti_aut/author.dll) 15) A POST request with the UserName set as '-' to webs.asmx (/_vti_bin/webs.asmx) 16) A POST request with the UserName filled to webs.asmx (/_vti_bin/webs.asmx) 17) A POST request with the UserName set as '-' to dws.asmx (/_vti_bin/dws.asmx) 18) A POST request with the UserName filled to dws.asmx (/_vti_bin/dws.asmx) Note: when using Advanced logging, the CS(Agent) file will indicate what the file was opened with on the Webs.asmx call if Word for example, the value is: Microsoft+Office/11.0+(Windows+NT+5.2;+Microsoft+Office+Word+11.0.6568;+Pro)
NOTE: YOU MUST CROSS REFERENCE TO DETERMINE WHICH FILE WAS OPENED. If User does NOT save the document, this will be followed by:
1) A POST request with the UserName set as '-' to author.dll (/_vti_bin/_vti_aut/author.dll)

2) A POST request with the UserName filled to author.dll (/_vti_bin/_vti_aut/author.dll) 3) A GET request with the UserName filled for the original page user started on (for example if in Area Auto's Document Library, the URL would be /Auto/Document Library/Forms/AllItems.aspx) 4) A GET request with the exact same time as #1, UserName set as - to the same page from #1 returning a 401 and Win32Status of 0 5) A GET request with the exact same time as #1, UserName set as - to the same page from #1 returning a 401 and Win32Status of -2146893042
NOTE: THERE MAY ALSO BE A REQUEST for /_layouts/images/headcornerp.gif - this is corner image of the Search Box.
Uploading a File
No IIS Log entries are created when a file is uploaded this is only recorded in the SharePoint Logs.
List Operations
IIS Log Entries for List Operations is not sufficient to determine the actual process; the SharePoint Logs should be used instead.
Cross Referencing Logs

The advanced logging mode is required to determine the File Name that may have been opened or saved. This information is contained in the field cs-uri-query. For example, saving the file Document Library/ASDASD.doc looks like this:
location=Document%20Library/ASDASD.doc&dialogview=SaveForm
If a User opened the file and did NOT make any changes, cs-uri-query will be set to -.
Calculating SharePoint Statistics

There are several internal reporting mechanisms within SharePoint both through Site Administration and through Front Page. Reporting and calculating usage is therefore to taste to be oriented more towards a specific goal vs. general reporting. Once converted to SQL Tables, virtually any report can be generated using Pattern Matching, Filtering and Grouping. However most reporting will need additional supplemental information as covered in the following.
Determining number of Downloads a user has Performed

Using IIS Logs
You must use pattern matching within the IIS logs to determine the number of documents a user has opened for read or for editing. As mentioned in the IIS Section, this means that the combination of 18 transactions on the 2-second transaction rule must be calculated based on the Period you wish to review. This method is ideal for determining limits as IIS can use ODBC logging and trigger an event. However, this does NOT provide you with the name of the file; you must use SharePoint logs to determine specifics. This can be done by cross-referencing the exact date and time stamps between the two log formats.
Using SharePoint Logs

You must use pattern searching within the SPS logs to determine the number of documents a user has opened for read or for editing. This can be based on File Extensions (.doc, .xls, etc.) or you can search for a call to owssvr.dll then check for a file name in the Query String. A simple count based on period From / To will indicate the number of items opened.
Calculating Storage Usage

A major area of importance to SharePoint auditing includes the ability to determine usage across the Portal or Site. Fortunately this is already provided via the SharePoint Administration but is easily calculated using the SharePoint Database Tables. You can perform a roll up at any level using these tables (each has either a foreign or local key that can be used for the Join). This involves the following tables: Configuration Database Table Name: VirtualServers This table contains the GUID of all sites defined in SharePoint Site Database: Table Name: Sites This table contains the GUID and Site URL Table Name: Webs (indexed by Site GUID) This table contains the GUID for each Site and Web underneath it Table Name: Lists This table contains the GUID of all lists indexed by Web GUID Table Name: Docs This table contains all of the documents indexed by List GUID Table Name: UserData* This table contains all Metadata for lists (indexed by Web/List GUID) * When calculating storage use, be sure to accommodate for attachments to regular lists.
Calculating Document Library Storage

To calculate document storage is a simple SQL Statement for example:
SELECT FROM WHERE SUM([Size]) AS TotalDocumentsStored, SUM(MetaInfoSize) AS TotalMetaInfoStored Docs (ListId = '{D933A17A-36E2-464A-B564-37112B0CB9FA}')
Calculating Attachments Storage

To calculate document storage of attachments is almost the same as a Document Library using a simple SQL Statement; however, to determine that it is an attachment, you must check the column DirName for Attachments as shown:
SELECT FROM WHERE AND SUM([Size]) AS TotalDocumentsStored, SUM(MetaInfoSize) AS TotalMetaInfoStored Docs (ListId = '{D933A17A-36E2-464A-B564-37112B0CB9FA}') [DirName] LIKE '%/Attachments%'
Calculating Usage For a Site

Calculating current storage within a site is easily done by using the Site (Web) GUID instead of the List:
SELECT FROM WHERE SUM([Size]) AS TotalDocumentsStored, SUM(MetaInfoSize) AS TotalMetaInfoStored Docs (SiteId = '{9CB53913-3802-4C2E-B25A-48F0E99C08F3}')
Counting the Number of Immediate Alerts Active in the Portal

Immediate subscriptions are those that are set to fire immediately upon a change or event. The number of these can be used when trouble shooting issues with performance and alerts.
SELECT FROM Count(*) ImmedSubscriptions
Counting the Number of Scheduled Alerts Active in the Portal

Scheduled alerts are all other alerts within SharePoint (Daily, Weekly, etc.).
SELECT FROM Count(*) SchedSubscriptions
Obtaining Sites a User Belongs To

SELECT DISTINCT tp_SiteID FROM UserInfo WHERE tp_Login = 'DEMO\dsterling'
Where demo\dsterling is the Domain\User Login.
Obtaining Site Members

SELECT DISTINCT UserId FROM WebMembers WHERE WebId = '{299B6909-45A1-494C-A158-B06804BC5C05}'
Where the GUID is the GUID of the web/site.
Obtaining User Profile Information

SELECT [RecordID], [DocID], [UserID], [NTName], [PreferredName], [Email], [SID], [Manager], [LastUpdate], [bDeleted] FROM [UserProfile] WHERE [NTName] = 'DEMO\dsterling' AND [bDeleted] = 0
Note: Users MAY OR MAY NOT HAVE A PROFILE; such users are orphaned within SharePoint security until a profile is created (if at all). Examples of this: Local system administrator, domain admins, etc.
Appendix A
Creating the SharePoint Log Parser
Creating this program requires Visual Studio with C# installed.
Executable Creation
Step 1: Create a new C# Windows Console Application project called SPSLogParser Step 2: Copy the Source Listing below into the default class generated Step 3: Rename the class file to be SPSUsageParser.cs Step 4: Add references to System.xml and System.Data namespaces Step 5: Compile
Database Preparation
NOTE: You can use any database for this, but a separate database (stand alone) is suggested. Step 1: Create a new Database called WebLogs (optional) be sure that the SharePoint Administration and SharePoint Services account both have the same access to this database as they do to the SharePoint database. Step 2: Using the Query Analyzer, connect to the database Step 3: Run the Table Creation script shown in SPS Log Parser Table Definition
Xml File Preparation

Step 1: Create a new empty file in a File folder that is accessible to the SharePoint Administration account Step 2: Paste in the XML from the SPSLogParser XML Settings File below Step 3: Modify the XML settings to your installation and save the file
SharePoint Log Parser

You can download SICGs SPS Log Parser from here: http://www.sterling-consulting.com/SICGFWebParts.htm#SPSLogParser
SPS Log Parser Table Definition

DROP TABLE [SPSLogParse] GO CREATE TABLE [SPSLogParse] ( [siteGuid] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [time] [datetime] NULL, [siteUrl] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [web] [nvarchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [doc] [nvarchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [username] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [useragent] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [referrer] [varchar] (2048) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [querystring] [varchar] (2048) COLLATE SQL_Latin1_General_CP1_CI_AS NULL, [bitFlags] [int] NULL ) ON [PRIMARY] GO
SPSLogParser XML Settings File

<?xml version="1.0" encoding="utf-8" ?> <SPSLogParserSettings> <InputFilePath>D:\00.log</InputFilePath> <OutputFilePath>D:\00Converted.txt</OutputFilePath> <DBConnString>Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=WebLog;Data Source=SICGA190;</DBConnString> <DBTableName>SPSLogParse</DBTableName> <AutoLoadAfterParse>true</AutoLoadAfterParse> <BulkLoadOrImmediate>Immediate</BulkLoadOrImmediate> </SPSLogParserSettings>
SPS Log Parse Table Bulk Insert Command

BULK INSERT [dbo].[SPSLogParse] FROM 'D:\00Converted.txt' WITH ( FIELDTERMINATOR = ' ', ROWTERMINATOR = '\n' )

The Definitive Guide To SPSLogging

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

The Definitive Guide To SPSLogging

Hochgeladen von

Copyright:

Verfügbare Formate

The Definitive Guide to SharePoint Logging and Auditing

Created by David M. Sterling Sterling International Consulting Group February 2006

Page 1 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Page 2 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Event Logging in SharePoint

SharePoint Log Timing

Page 3 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Enabling SharePoint Logging

Parsing SharePoint Logs

Page 4 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

IIS Log information and SharePoint Logs

Page 5 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

The Definitive Guide to SharePoint Logging and Auditing

Logging Elements (W3C extended format)

Page 7 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Page 8 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Page 9 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Logging with ODBC

varchar(255) SMTP protocol command

The Definitive Guide to SharePoint Logging and Auditing

varchar(255) Recipient varchar(255)

The script for this is as follows:

Page 11 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Page 12 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Page 13 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Page 14 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Understanding IIS Logs for SharePoint

Noise Transactions in IIS

Detecting when a User has selected a New Document

IIS Log entries when a new document opened:

Page 15 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Page 16 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Detected when a User has opened a document in Read Only mode

Opening a file in READ ONLY mode

Opening a File for Editing

Page 17 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

The following occur just as with Read mode:

Page 18 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Cross Referencing Logs

Page 19 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing

Calculating SharePoint Statistics

Determining number of Downloads a user has Performed

Using SharePoint Logs

Page 20 of 25 February 2006

The Definitive Guide to SharePoint Logging and Auditing