HTTP SNIFFING FOR WEB ANALYSTS - WHY?
As web analysts we need to be able to see and test the technical details of tags, so that we know exactly how the data is being collected. For Coremetrics, however, one thing that can be difficult about viewing and debugging tags is that the official Coremetrics tag viewer, TagBar, only works for Internet Explorer and some early versions of Firefox. So this post will walk you through how to use an HTTP sniffer to view tags, audit, and debug Coremetrics implementations. Don’t worry if you’re not very familiar with HTTP sniffers – they are just software that lets you see all the traffic being sent from your browser to the internet. There are several benefits to this technique:
- it will work for browsers like FF and Chrome, not just whatever old version of IE TagBar currently works with.
- it can validate what you're seeing in TagBar if you need a second opinion 🙂
- the output can be easier to read than TagBar since it both stores the calls and displays them separately, rather than listing them all one after the other
- there might be some cases where you don’t have access to the official Coremetrics TagBar since their tool can only be downloaded while logged into a Coremetrics account.
- the method here can actually be easily adapted for other web analytics tools. I’m just focusing on Coremetrics since Omniture and Google Analytics have quite a good ecosystem of tools that are even better designed for this purpose, while Coremetrics does not -- for example I previously wrote about tools for debugging Google Analytics, but none of those tools work very well (or at all) for Coremetrics.
So, here we go.
HTTP SNIFFING FOR WEB ANALYSTS - TUTORIAL
1) Get an http sniffer (also known or available as web debuggers, packet sniffers, http analyzers, etc, but we don't need anything too complicated here)
I’m going to recommend HttpFox. It is a free, simple plug-in for Firefox that works great for this purpose. Download it here: https://addons.mozilla.org/en-us/firefox/addon/httpfox/
(Chrome has a built-in sniffer that you can access by navigating to the "network" panel of Chrome's developer tools and clicking on the "Headers" tab. However, this tutorial will focus on HttpFox for Firefox because it's easier to use).
2) Install the plugin and restart Firefox. Activate it by navigating to Tools > Web Developer > HttpFox > Toggle HttpFox (or just hit Shift + F2)
3) Click the “play” icon and make sure Start is greyed out.
4) Now the fun starts. Navigate to the site you’re checking out. For the purposes of this tutorial, I’m going to Costco.com.
5) Oh no! There are hundreds of calls.
6) We need to filter these so we just see what we’re looking for: the calls to the Coremetrics servers. Type in “ci=” into the search filter box. This will limit it to just calls with the text “ci=” in them, which shows up in all Coremetrics tags (more on that in a minute). You could probably choose some other text but I’ve found this to be reliable across any site running Coremetrics, and excellent for filtering out the non-Coremetrics calls as well. Now we'll see just the Coremetrics calls.
7) You can click around the site and these calls will persist in the top section, with the most recent calls appearing on the bottom. Click on the Query String tab to see a parsed list of what is being sent to Coremetrics.
8) Now let's look at an example result and dig into it. Here is the first call that gets sent from Costco.com to Coremetrics.
You can see a long list of query strings. Let's go through one by one:
Parameter | Description |
ci | Client ID |
st | Session start time |
vn1 | Coremetrics library version |
ec | Character set encoding |
vn2 | Coremetrics library version #2 |
pi | Page ID (page name) |
ul | Destination URL. The URL of the page. |
tid | Tag ID/type |
tid 1 = page view tag | |
tid 2 = registration tag | |
tid 3 = order info tag | |
tid 4 = shop tag (combine with action 5 for shopping cart tag, combine with action 9 for order receipt tag) | |
tid 5 = product view tag | |
tid 6 = technical properties tag | |
tid 7 = custom details tag | |
tid 8 = link click tag (for real estate/ site promotions click tracking) | |
tid 9 = link impressions tag (for cm_sp impression tagging) | |
tid 10 = form action tag (auto-generated for forms) | |
tid 14 = conversion event tag | |
tid 15 = element tag | |
tid 404 = error tag | |
cg | category. This needs to match an entry in the Category Definition File (CDF) |
rnd | Random # (used for cache-busting) |
pc | Boolean answering “is this a page view?” |
jv | Javascript version |
np0...np11 | Technical browser properties |
je | Boolean answering "is Java enabled?" |
sw | Screen width |
sh | Screen height |
pd | Color depth |
tz | Time zone |
cvdone | Not sure |
As you can see, this was a Technical Properties Tag, triggered since it was the first page of the visit. The standard Page View tag and Product View tags would look very similar.
If we next add a product to cart, a Shopping Cart Tag (aka Shop Action 5 tag) fires and HttpFox displays the following:
Now we can see some additional query parameters, described as follows:
Parameter | Description |
pr | product number |
pm | product name |
qt | quantity |
bp | base price |
cg | product category |
at | action tag ID tag |
at 5 = Shop Action 5 | |
at 9 = Shop Action 9 | |
rf | referring URL |
Besides these, you will come across Order Tags, Registration Tags, Element Tags, Conversion Event Tags, and Error Tags (deprecated since 2010, but you may see them in older implementations). Each of these will contain slightly different parameters (and a lot of overlap) but the basic method of viewing them is the same.
Just for completeness, here are the remaining parameters you're most likely to come across:
Parameter | Description |
eid | element ID |
ecat | element category |
hr | href (look here to check cm_sp site promotions and cm_re real estate tagging) |
cid | conversion event ID |
ccid | conversion event category ID |
pr_a1...pr_a5 | extra product attributes |
pv_a1...pv_a5 | extra page view attributes for use in Explore (will not show up in regular reports) |
Again, HttpFox will persist all the calls in the top, so you can click around on the pages you're interested in, and then review what was sent.
I hope this tutorial helps you understand exactly how to debug Coremetrics when you aren't able to use TagBar.
**9/19/12 Update: This post contains the complete list of query strings for reference.
the 'at' parameter you have listed as 'not sure' is the Action Tag. The values are 5 for a Shop Action 5 Tag or 9 for a Shop Action 9 Tag.
Thank you, Shawn! You've solved a mystery... I'll update the post.
Sure. That was one I actually knew. Your article hepled a lot. I've been looking for a breakdown of all of the CM parameters. I even tried to get more info from IBM and they wouldn't tell me what everything meant.
Hey, here's another one for you... In the Technical Properties Tag the 'jv' parameter is the JavaScript Version, I believe. In this case, version 1.6.
Yeah, not sure if they don't want us to know or just don't remember the definitions anymore 😉 Thank you for the new definition as well, this is great! Do you find it useful to see the actual parameters instead of what gets displayed in TagBar?
It's hard to say really. The Tag Bar does show the error messages, which can be helpful if there actually is an error (which is suspect). It think using another tool and looking for the parameters is a bit easier, once you get a list of what the parameters mean. Some of them do get displayed in the TagBar as well, so those are easy to find in the image requests. All the other ones that you have here are the ones that it's helpful to have this list for. I'm guessing IBM wants to force you to use the TagBar...
Very userful article. thanks a lot
Fab article for beginner like me ... many thanks.
I hope to learn something from this...
Totally agree, Ana always give us awesome Coremetrics tips, thanks for sharing all this information with us Ana
Hello,
The rnd (Random #) parameter is used to have a different url whatever the user is. Thus it could not be cached by a CDN (Akamai for instance...) Thanks for the article
Thanks for the explanation, Damien! That makes sense. I'll update the article.
I created a chrome extension that pretty much duplicates the functionality of the Coremetrics tagbar in Chrome: https://chrome.google.com/webstore/detail/coremetrics-bar-for-chrom/llegcghmokaemodgdddnchiijfdbfnlg?hl=en
Chris, that's awesome - you are going to make many people very happy! Is there any documentation on how to get it running? I installed it and navigated to some sites running Coremetrics but it was still coming up blank, do I need to turn something on first (or am I just missing something obvious...)
Ana, You only need to install the extension in Chrome and restart Chrome. Once you restart Chrome, you will find and external window that will show most of the Coremetrics tags once they are fired up. This extension is great for Google Chrome lovers, Thanks to Chris Dial for sharing the extension and source code to make changes to the extension
-Oscar
Ah, you're right, it's working now. Very useful extension!
Chris, please could you check the pull request which I have sent to you on GitHub - it contains some new features for yours otherwise great extension.
Hi Ana Kravitz,
I have configured coremetrics for my client site. while loading home page and category page response time is more. we observed that this is happening because of impression tags(i.e., automatic impression tags). Now we have decided to disable Automatic Impression Tags. I have tried some scenarios it is not working. Can you please help me how to disable or turn off Impression tags for the site. In home page we are loading eluminate.js and cmcustom.js
Hey Harikrishna, are you using site promotions tracking? Automatic impression tags are often set because you're using site promotions tracking that automatically enables impression tracking. If this is the case you'll need to switch to using Manual Link Click tags using the cmCreateManualLinkClickTag() function. I suspect this will work, but please let me know if it doesn't or send me a link so I can check it outl
Thank you for your reply Ana. In cmdatatagutils.js I have changed a line from
var cm_TrackImpressions = "RS"; to var cm_TrackImpressions = "";. Now I am not getting automatic impression tags.
Super!