W3Counter 4.0: New Referrers and New Searches

I try to decide what reports W3Counter should include and how they should be presented by formulating questions a website owner might pose about their visitors. W3Counter now answers two new questions I often ask about my own sites — “Who’s started linking to me today?” and “What new searches am I showing up for today?”. As with all the other reports in version 4, these questions are answered over any date or date range.

W3Counter 1.12.2007

The two new reports are called “New Referrers” and “New Searches” and look for referring URLs or searches that appeared in the date range you specify but not any time before that. It’s an easy way to find out who’s just placed a link to your site, what blogs are talking about you, and what new searches your pages are turning up in.

Posted on January 12, 2007 in Programming, W3Counter

More from this category

  • http://www.serze.com Gersitz

    Hey dan, can you hook me up with a free stat account at w3counter? I would like to check it out.. thanks.

  • http://www.ryansgoblog.com/ Ryan Williams

    This is one hell of a late comment, but what happened to this feature?

    When looking at these two reports I was trying to figure out why one can only see referrers/search from the current day, and in fact ended up searching to see if you’d documented this feature at some point — hence finding this post.

    It seems like it’d be more useful if this could be seen on a several-day basis as I sometimes might not check my stats for a few days but still want to get an update on unique referrers (the alternative being an epic trawl through the normal referrer reports).

    Am I morbidly blind or did you choose to withdraw this functionality? If the latter, any chance of us seeing it again? :D

  • http://www.dangrossman.info Dan

    Version 4 was originally meant to be a downloaded web stats program you’d run on your own server. When the obstacles to that happening became apparent, the code was repurposed as the base of the next version of the hosted solution, which several months after this post, became what is available now.

    The ability to select date ranges was a feature dropped as storing and querying arbitrarily large amounts of data for each site to do so is not viable under the economic constraints of the hosted service with the current subscriber base. Depending on the account plan and usage patterns, the average site has between 25,000 and 50,000 page views of history stored that can be analyzed in the reports. That represents vastly different date ranges for a site like this blog and one which pushes its page view limits every day. I have months of data in my 50,000 page view log, while the most active sites only have 2-3 days.

    Therefore adding date ranges would be not only inconsistent between accounts but between websites within accounts, offering those that pay to upgrade the least benefits, and that’s not something I want to do.

    Archiving off data by date for analyzing over arbitrary periods would be a fantastic upgrade. Half the code for it remains mostly in place. Most of the actions and model class methods take a bunch of arguments that are never offered as options to the user — date ranges, number of results per page, sorting… because they were part of the feature set when I designed it, but aren’t viable to offer right now. Do me a favor and poke Yahoo! or someone else with a big wallet, get ‘em to buy up W3Counter, and I’ll turn it all on then :)

  • http://www.ryansgoblog.com/ Ryan Williams

    Well, okay. That all seems reasonable and makes sense.

    Was my concern about missing hours of data valid, though? *Does* it just give you referrers/searches from the last 24 hours, or does it work on a date basis (thus if you checked it at 1am you’d have just data from the last hour)?

    If this is the case, perhaps it’d be worth extending the report to two days instead of one by default, or 48 hours if it works like that. Otherwise as I exampled above, if you were to check your reports daily at 8pm you’d routinely have four hours of missing data.

    I’m guessing that the way it currently works is something that’ll bother some people and not bother others at all, depending on their stats-viewing habits. Either way, I can’t see how just extending it to a couple of days would adversely affect anything or anyone — but I could be wrong. D:

  • http://www.dangrossman.info Dan

    It’s by date, and extending it to a couple days could well make it include the entire log for high traffic sites, where it’d degenerate into the referrers report sorted by timestamp instead of by count.

  • http://www.ryansgoblog.com/ Ryan WIlliams

    Hmm, I’m not sure ‘degenerate’ is the word I’d use to be honest.

    ‘New Referrers’ as a title implies that the report counts all referrers, and only adds new ones to the top if they are indeed newly referring sites. Sites that have already appeared in the logs as referrers, even if it was a few days ago, logically shouldn’t be included as they’re not new referrers. They’re just sites that linked a few days ago and are presumably still in your living memory.

    The way it currently works, a site could be counted as a ‘new referrer’ every single day for years after the original link was placed, assuming people keep using it to get to your site. But in reality, it clearly isn’t a new referrer; it’s just a repeat referrer.

    I think this is why I got confused when I saw the reports, because with the current titles and lack of date options they just doesn’t make much sense. This was made worse because Mint has an almost identical report, except it stretches over its entire five-week date range (thus literally only counts new referrers for as long as it’s capable of remembering).

    It definitely goes against your very own description of as “easy way to find out who’s just placed a link to your site”, because you could quite conceivably end up with the same few dozen referrers appearing in there day after day, all of whom certainly wouldn’t have “just placed a link”. If the report instead showed a longer date range by default, this wouldn’t happen and the report would serve its implied purpose more effectively.

    If it did work this way, the only downside to having a high-traffic site (and thus being a subscriber) would be that you’d get sites that’ve already been reported as a new referrer reappearing sooner than they would on a low-traffic site. But then, is that worse than them reappearing every single day regardless of whether or not they need to?

    This’d also eliminate the lost hour syndrome that occurs if you check your stats at 8pm one day and then return the next day, to find you’ve no way of viewing the ‘New Referrers’ data for those four hours at the end of the previous day.

    Food for your thoughts, anyway. You can disregard everything I say if you like; I don’t have my own stats software to play with so I can’t help but think of ways to make yours rock that bit more. :D

  • http://www.dangrossman.info Dan

    I don’t understand your response. The report does exactly what your second paragraph says it should. A site that linked to you at any point before the current date, if that past time is within the range of the log, will not show up again… especially not every day. In SQL-PSEUDO-CODE…

    SELECT referrer, MIN(time)
    FROM somelog
    WHERE uniqueVisit = 1 AND referrer isn’t own site
    GROUP BY referrer
    HAVING DATE(MIN(time)) = CURDATE()
    ORDER BY MIN(time) DESC LIMIT x,y

    I think what you’re really asking is that I change the HAVING condition to be 24 hours or so instead of an exact date… and that’s doable.

  • http://www.ryansgoblog.com/ Ryan WIlliams

    Well, in that case I think it transpires that I’m a bit of a dick.

    I think my confusion pretty much centred around the date-based nature of the report. I assumed that because the view of the data was clipped at a day, the data itself was too.

    My main concern was as you rightly pointed out the potential to miss out hours due to the date-based nature of the report. Just changing that to 24 hours would be a great change (Although clearly 48 hours is the sweet spot! ;) ).

  • http://www.dangrossman.info Dan

    I’ll play with it, probably end up with what you want, when I can… next 24 hours are dedicated to implementing RSA encryption, the chinese remainder theorem, and a three-part Karatsuba integer multiplication algorithm… followed by two days of midterms. Sorry, already in the zone~

    You’re the only person I know that has Mint. Have you taken a look at the source code? I don’t have a copy, so I always wonder… is it open, unencrypted code? Can you easily make changes to it? Nothing stops people from selling it as their own?

  • http://www.ryansgoblog.com/ Ryan WIlliams

    It’s open and unencrypted, yeah. I wouldn’t know about making changes to it, but I’m guessing one could theoretically scrape out a lot of what makes Mint tick and use it elsewhere.

    Theoretically.

  • Pingback: Reseller Program Available