Logo SharePoint Thoughts   Downloads   About   Up to Bluedog Limited
Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
Posted on 2/21/2005 2:11 PM by pBoy
Hi everyone - I'm Steve and this is my first foray into blogging on BlueDog.  During the week I play a Lead Architect in the Portals Center of Excellence team at Microsoft.  In the evenings I craft my trade in the disguise of Master Chief, as one of many used-to-be-young-and-good-at-pong, now-getting-my-butt-kicked-by-twelve-year-olds Halo2 players.
 
Many of the big companies I work with have a pretty common and reasonable request - how do I track usage of my portal implementation?  This is a fairly tepid story as you learn about SharePoint and read through some of our Help files and KB articles.  In short, our standard story is:
  • You can use Usage Analysis for Windows SharePoint Services
  • It won't work for SharePoint Portal Server (and we took it out of the admin UI after beta 1 or 2)
  • KB838702 says "No usage analysis data is collected for portal sites in Microsoft Office SharePoint Portal Server 2003."
  • If you look in the IIS logs, you'll see that every time you download a document from SharePoint, the individual document is NOT recorded in the logs.  Instead, the DLL that streamed the file back to you is - owssrv.dll.

What's a poor user to do?  Fortunately all is not as grim as things might appear.  Here are some things to help steer you in the right direction.

  1. You CAN get usage statistics for both WSS and SPS sites, including individual document downloads.
  2. To enable this, you must turn on Usage Analysis log gathering in WSS central administration.  You don't have to turn on usage analysis processing - that's up to you (more on this later).
  3. Usage information is associated with SPWeb and SPSite objects.  The real problem in SharePoint Portal Server is that the information associated with the underlying SPSite of a Portal Area is not useful.  What I mean is that to SharePoint, an entire portal looks like one big Site Collection.  As a result, if you look at the usage statistics associated with the underlying SPSite of every single Area in the Portal, the numbers are exactly the same.  However, the statistics for the underlying SPWeb associated with Areas are correct.  Hey, I can live with that.
  4. Finally, there are two parts to Usage Analysis - capturing the data, and then processing it.  When you choose to turn on usage processing, a scheduled job will parse the data and write it to the SQL database.  Unfortunately, it's stored in a format that only its mother could love.  That's right folks, you won't be doing any cool JOINs or SELECTs off this data - it's such great information that we decided to store it as a bitmap in an Image column in the database.  And for the heckler in the back, no, you can't open it in Paintbrush either...it's not THAT kind of bitmap.

When you turn on logging, all hits to a Portal Area or WSS site will be captured as long as they go through the standard WSS DLLs (owssvr.dll stswel.dll) - this covers about 99% of the traffic in most cases.  That includes documents downloaded from the browser as well as opened directly using Microsoft Office products.  However, if you create a Network Shortcut to a document library in Windows XP or Server 2003 and then drag and drop files to your local machine, they will not be captured.

Now that you are capturing this information, you have a few options for retrieving it:
  • Parse the WSS gatherer logs - the log format is documented at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odc_SP2003_ta/html/ODC_WSSUsageEventLogging.asp.  The log format is not guaranteed to remain static through the next full version release of SharePoint though.  However you are good for SharePoint 2003, and it will ensure that you get all of the data that's been captured.  You'll need to write your own parser to interpret that data however, and then will likely want to store it in some persistent store such as SQL Server.
  • Use the Object Model - you can use the SharePoint Object Model to get at each SPWeb in site collections, as well as each SPWeb associated with each Area in your portal.  The GetUsageData method of the SPWeb class is your friend here.  Beware though - this method only returns 2000 rows maximum.  If you have more than 2000 unique doc downloads or visitors to an SPWeb for example, you won't see them all using the Object Model.
  • Use the usage analysis pages in the _layouts directory - it’s not in the UI for a few reasons, but you can still get usage information for any Area.  Simply navigate to that Area and in your browser address bar, back up over default.aspx and replace it with _layouts/1033/usageDetails.aspx, where "1033" is the language code for your particular implementation (1033 is US English).  It's not pretty, it's not much, but it's free.  'Nuff said about that option.
  • Use FrontPage RPCs - you can use the GetUsageBlob function in FrontPage RPCs to retrieve the usage statistics, and it is not bound to the 2000 row limitation of the Object Model.  You can find out more about this method at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/spptsdk/html/tsrpcmGetUsageBlob_SV01107409.asp.  However, this is not the preferred method because you will, again, have to write a parser to work your way through the results from that call.  FrontPage RPC's are not documented terribly well which is another concern.  We do have a parser sample in C++ posted however (yes C++, no managed code here my friends); you can find that sample at http://www.microsoft.com/downloads/details.aspx?FamilyID=8ed64398-4992-47cb-9075-afec32060986&displaylang=en.  At the end of the day though, it has the same exact information that is in the WSS logs so if you can get access to those logs you'll save yourself a lot of time.

Here are a few other gotchas to remember:

  • The Object Model only stores the last 31 days of statistics.  This means that for any meaningful information you need to have a tool that will scrape the statistical information out using one of the methods described above, and you will need to write it to a persistent store like SQL Server.
  • The logs will not capture hits to the _layouts directory.  This may be a good thing (do you really want your boss to know how much time you spend tooling around the admin pages??).
  • It will not capture requests that return 302 "Not Modified" or error codes.  “Yes Sir, as far as I can tell, everything's working fine.  Do I get a raise now?”  It's worth a shot.

There you have it - the good, bad and ugly of usage analysis processing in SharePoint Products and Technologies.  Hopefully you found this first post useful.  I continue to harbor hope that future topics won't require such arcane digging around, but if they do I'll be up for it!  Just leave a light on, and a Cherry Coke on my desk.

re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
One option is to simply use the information that is available within your IIS logs.  Such an approach is discussed in http://www.bluedoglimited.com/SharePointThoughts/ViewPost.aspx?ID=8.
 
Maurice Prather @ 2/22/2005 8:01 AM
re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
You could always use FrontPage to view Usage Data on SharePoint.
 
Read about it here:
Nick Porter @ 2/22/2005 1:37 PM
re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
The problem with IIS logs is that the don't contain individual document downloads; they are all represented by owssrv.dll.
 
FrontPage may be suitable for department level statistic gathering, but is really not a comprehensive enterprise level solution.
Anonymous User @ 2/22/2005 4:04 PM
re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
You're right... that was mentioned in Maurice's post. 
 
It's really a matter of balancing what type of data you need...  for some folks the abstraction of owssrv-served files is ok.  If that's the case, then IIS logs are suitable.
 
Anonymous User @ 3/1/2005 11:54 AM
re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
I read the post and I'm wondering if this will help me get to the info I need.
I would like a report that can show me details for each user, such as the sites, pages, and documents viewed.  The information is in the text usage logs, but there doesn't seem to be an easy way to pull this information. You would think this would be one of their canned reports.
 
Thanks
Court @ 3/2/2005 1:53 PM
re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
Nice idee to talk about the log, many compagny don´t do this ananlysis...
 
EROL MVP SPS
EROL @ 3/11/2005 2:21 AM
re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
Hi,
 
I am interested in getting document level usage statistics. For each document the history of who all downloaded it and the other way round too(for each user the list of documents he/she viewed/downloaded).
Can this information be got by the idea discussed in this thread?
 
Regards,
Ram
Ram @ 5/10/2005 11:36 AM
re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
Does anyone know how to truncate the usage logs before a certain date? We went live a few days ago, and we have some usage data from a month ago during development that is skewing results for the last few days.
 
Thank you,
Wes
Wes Duck @ 8/3/2005 8:33 AM
re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
We have done all the above and more using a cool tool called Cardiolog from a company called Intlock. It saved us a lot of time plus gave us info we could not obtain otherwise. They also have a sexy report dashboard. They give extra metrics such as the types of actions performed in the Sharepoint, navigation patterns and usage analysis, search details, and more. I think they pump info from all these logs plus track data from other SPS and WSS sources by themselves. Saved us a lot of headache.
 
Their website is http://www.intlock.com
 
Enjoy...
Eric @ 9/23/2005 11:54 AM
re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
I looked at the website of cardiolog. Looks impressive, do you know of other tools of this level? Any ideas for monitoring document statistics and audit trailing the sharepoint? Tnx.
Robert @ 9/30/2005 1:14 AM
re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
If you're looking for a tool, check out WebObs (www.webobs.com). It provides friendly readable reports for SharePoint usage. The product architecture is extremely scalable and there is almost no impact on performance. It doesn't use IIS Logs either and works for both SPS and WSS.
Anil @ 10/4/2005 12:43 AM
Free Version - CardioLog Lite, SharePoint Usage and reports
following this post - http://www.bluedoglimited.com/SharePointThoughts/Lists/Comments/DispForm.aspx?ID=381 I wanted to bring our product to your knowledge - CardioLog Lite for SharePoint We'll be releasing a FREE version of the product on August 10th. hope you care to take a look at - http://www.intlock.com/intlocksite/ProductsAndServices/CardioLog/CardioLog-Lite-for-SharePoint.asp Thanks, Uri
Uri - Intlock @ 7/11/2008 1:41 PM
re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
This CardioLog thing looks good but doesn't come for 64bit.  Hopefully it will come out soon... 
 
Any others that are freebies you would recommend?
Stephanie @ 9/24/2008 10:53 AM
re: Your Top 10 List...Or How To Gather Usage Statistics with SharePoint
Not exactly so, browsing through their FAQ, i found that Cardiolog for sharepoint does support x64 platforms.
 
see for yourself:
 
worked for me alright...
Scott Bradsdale @ 5/1/2009 4:44 PM
RSS feed
Microsoft Certified Master
MVP Logo
Follow me on Twitter!
Keyword Search
 
View by category
 

Disclaimer:
The contents of this site represent thoughts and opinions of the authors , not those of anyone else - such as past, present and future employers.  This a forum of the exchange of ideas centered on SharePoint technologies.  It is not a support channel.  :)

Copyright © 2004-2010 Maurice Prather, Inc. All rights reserved.