10 Years of W3Counter

W3Counter is turning 10 years old.

Of the dozens of sites I put together in 2004 and 2005 from my college dorm, W3Counter’s the only one still around aside from this blog. It’s rather popular still — with around 70,000 users acquired completely organically and billions of page views being tracked — despite the explosion of competitors in the now enormous analytics industry. I’m sometimes asked why Google Analytics didn’t kill it — I think that has a lot to do with the fact that “easy” and “fun” are rarely used to describe GA, but I hear them used to describe W3Counter all the time. There’s a high school teacher that has his entire class sign up for accounts every year after creating their own webpages — they get a kick out of seeing the counters, and it’s easy enough to set up that it takes most people just a minute or two.

Every 2-3 years I’ve given the site some needed attention to keep up with the times. The architecture’s gone through several major revisions — from running on a single VPS to several servers, to a short stint running in Amazon’s Elastic Compute Cloud, then back to bare metal servers at SoftLayer. 2014′s users demand two things, at least if I care to keep some portion of them upgrading for the paid perks — real-time dashboards, and a mobile site.

Tracking Active Visitors in Real-Time

pulse

Handling even a few thousand concurrent visitors pinging the service every few seconds with their current activity to run these dashboards necessitates something a little more efficient than the way W3Counter was originally programmed — a traditional PHP MVC application, with the Symfony framework on one or more frontend servers, talking to one or more backend database servers that store data about page loads from which the reports are generated. All that framework overhead and database communication can take 40-50ms to finish.

The code that collects data for the dashboards cuts that down to under one millisecond. It’s a node.js application, which bypasses all the overhead of the PHP framework. Redis, an in-memory key-value store is the first thing that code talks to. Redis caches information I’d otherwise need a database lookup to get. While MySQL has its own query cache, using it would still require a round-trip to the remote database server, and having the spare RAM on the database servers to dedicate to that.

The information on what all those active visitors are doing still sits in a MySQL database, but an entirely in-memory one on the front-end servers using the MEMORY storage engine. Unlike the database servers, my front-end web servers typically use less than 25% of their available RAM — they’re CPU-limited rather than memory-limited in their capacity — so keeping the ephemeral data for these dashboards there is just using otherwise unused resources.

By keeping the code efficient and only storing data in memory, one relatively low-end server can serve several thousand concurrent active visitors with negligible load. The only problem I actually ran into was the raw number of TCP connections. They came in faster than nginx was accepting them — I needed to increase its worker_rlimit_nofile and worker_connections parameters to handle more. There were still more than the OS was happy to handle, complaining through the messages log that I was under a SYN flood attack — no, it’s not an attack — until I increased the available local port range and SYN backlog size. I quickly became friends with the sysctl command.

I’m a fan of dashboards myself. W3Counter’s Pulse Dashboard for Improvely runs in a rotation on a screen in my living room along with dashboards from Geckoboard, New Relic and my self-built home automation system for controlling lights, locks and temperature in my home. I’ll have to write about building that some day.

Mobile Web Design

I was also, a bit reluctantly, forced by increasing mobile use to finally learn responsive web design. I still haven’t got it quite perfect, but W3Counter’s now usable on any screen size between your 4″ smart phone and 30″ monitor without half the screen being whitespace, like some vertical equivalent of the black bars on a 4:3 TV watching a letterboxed movie.

Screenshot_2014-02-08-22-05-18

Various reports switch from multi-column to single-column layouts at smaller screen sizes, and menus disappear into a touch-friendly button-activated dropdown at the top of the page.

Most of the credit goes to Bootstrap 3′s responsive grid and “collapsible” plugin, along with a smattering of hand-written media queries that even work on IE8 thanks to Respond.js. Nobody’s running IE8 on their phones, but without that library, it shows the mobile version of the site which doesn’t look so great on a 20″ monitor.

A Minimal Website

I also redesigned the marketing site for W3Counter to match the new look — responsive, mostly flat aesthetic, using the same two shades of blue (#c1deef and #e1f3fd) as I’ve used with W3Counter since around 2006.

I found that most pages of the old site were rarely if ever viewed, so why have them at all? I trimmed it down to just 3 — a homepage with a giant image showing off the reports and signup button, the monthly browser & OS market share report that gets quoted in news at least a few times a month, and a signup form.

So far so good — I published the new design three days ago. Only one user wrote in with a note that they didn’t like something I changed. The daily signup rate has doubled. Several of them upgraded to paid accounts within minutes of signing up. 2014 should be another good year for W3Counter.

Excuses for Lazy Coders

When Walker put the objects, along with their accompanying stories, up for sale on eBay, the results were astonishing. On average, the value of the objects rose 2,700%. That’s not a typo: 2,700%. A miniature jar of mayonnaise he had purchased for less than a dollar sold for $51.00. A cracked ceramic horse head purchased for $1.29 sold for $46.00. The value of these formerly abandoned or forsaken objects suddenly and mysteriously skyrocketed when they were accompanied by a story.

If You Want to Raise Prices, Tell a Better Story

A frontend UI for the Spreedly Core API

Spreedly is one of my favorite companies because they’ve solved the problem of payment processor lock-in for my business. Their service is a very affordable, independent payment information “vault” — a secure, PCI-DSS compliant place to store customers’ data, and later charge it using whatever payment processor I want.

Spreedly is built for developers — it is a set of APIs that my software can communicate with to store payment information or tell Spreedly to do things with it, like charging some amount or deleting some information when a customer cancels their subscription. There’s no way to look at what’s been stored in an account other than writing code. That’s why I created Vault Viewer, a small PHP application that lets you look into your account and see, search and delete your stored payment gateways, payment methods and transactions.

Vault Viewer

The code is open source and free to use, and installation is typically just a matter of dropping the whole thing into a web server document root configured to serve PHP. There’s a Live Demo Here if you’d like to try before installing. I recommend only using the demo if your account only contains test data, as it’s not on an SSL-secured domain.

Download the code at GitHub

Date & Time Range Picker for Twitter Bootstrap

My date range picker component for Twitter Bootstrap has been updated with the much-requested ability to choose times as well. You could use it in an app to schedule meetings, choose flight departure/arrival times, etc. Enjoy.

P.S. Yes, it works with the upcoming Bootstrap 3 as well.

The ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the right solely on its quality.

His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the “quantity” group: 50 pounds of pots rated an “A”, 40 pounds a “B”, and so on.

Those being graded on “quality”, however, needed to produce only one pot — albeit a perfect one — to get an “A”.

Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity.

It seems that while the “quantity” group was busily churning out piles of work-and learning from their mistakes — the “quality” group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.

Art & Fear: Observations On the Perils (and Rewards) of Artmaking via Derek Sivers

What’s the shrewdest, smartest maneuver you’ve ever seen in business?

Unhappy Truckers and Other Algorithmic Problems

Parsing User Agents

For many, many years, Gary Keith maintained a set of files (“the files”) as part of the Browser Capabilities Project. These files contained thousands of entries mapping regular expressions to browser families, versions, platforms and capabilities, and these files were used by parsers written in a dozen languages to turn website visitors’ user-agent strings into structured data about their system. I’m truly thankful for the hard work Gary put in maintaining those files single-handedly for so many years.

At the end of 2012, after many months of warnings about his health and declining ability to run the project, it was closed with no new maintainer. Nobody’s stepped up to provide the same level of support he provided, and it’s no longer a reliable source of information about browsers and their capabilities.

If, like me, you need the ability to parse user agents for browser and platform information, I recommend the ua-parser project. It’s up-to-date, with a team of contributors, takes pull requests on github and provides libraries in 10 languages. It’s also much simpler to maintain, with only 1000 lines of YAML rather than 28,000 lines of INI files.

I’ll be using ua-parser as the basis of W3Counter’s browser and platform reports going forward.

Hi, I'm Dan Grossman, the Philly-area creator of Improvely, W3Counter, and some open source stuff. Here's a couple other places you can follow me online —