W3Counter Grows Again

July 17, 2008 // W3Counter

It’s been about 7 months since W3Counter’s last hardware upgrade, adding a beefy dual-processor dual-core server to the mix. Now, 6,000 new active accounts later, the site’s slowing down during the day again, which means there’s not enough memory and IO bandwidth to go around. I just ordered two more powerful servers from SoftLayer. One will replace a current database server in order to move everything to a single physical location (less latency between the web and database servers), and another to expand the database server count to 3. Since I rearchitected the software to a database partitioned over multiple servers during the last upgrade, all I need to do is plug the new ones into a config file once they’re built and online.

I was tempted to add some resources from Amazon EC2, but it didn’t make sense financially. I have already built and tested custom AMIs I built with W3Counter ready to go — a click of a button and W3Counter is running in the cloud — but without persistent storage for the database, and the “small” instance not having enough compute resources to run even the web server alone, it’d cost more to run the service there than with a set of physical servers. Once persistent storage is available, and you don’t need replication to avoid data loss if an instance crashes, I can evaluate EC2 again. It is a lot of fun to use.

AddThis Social Bookmark Button

1 Trackbacks to “W3Counter Grows Again”

  1. Trackback from Bookmarks about Storage on August 24th, 2008 at 12:00 am:

    […] - bookmarked by 5 members originally found by dperkel on 2008-08-03 W3Counter Grows Again http://www.dangrossman.info/2008/07/17/w3counter-grows-again/ - bookmarked by 4 members originally […]

3 Responses to “W3Counter Grows Again”

  1. Brendon Kozlowski
    July 18th, 2008

    Dan, glad to see you getting back in to the swing of things with your blogging. I remember when you discussed extending the database to multiple servers via code awhile back using a configuration “DB lookup” file. I know Symfony uses YAML for most of its configuration, but… Can you explain (in a quick summarized manner) how you went about doing this? I think this might be beneficial to others, if you have the time.
  2. Dan Grossman
    July 18th, 2008

    There are two types of requests the W3Counter web server gets — hits to tracker.php to serve the counter image and log a visit to a website, and requests for a report on a website. In either case, the website ID to act on is always part of the URL.

    The first thing that’s done in either case is to query a lookup table that has (at least) two columns: website id, and server id. This is the table that keeps track of what websites are on what database servers. The server id’s correspond to connection names in the databases.yml file, so the script can connect to the correct database to record a visit or generate a report.

    It’s basic horizontal partitioning of the database. The table structures on each DB server are identical, so it’s just a matter of knowing which server has the rows you’re looking for. A few functions have to connect to every database to generate their report, like the summary screen shown after login that lists all websites in an account and some basic stats for each, as a user’s websites may be spread across multiple servers. In that case it’s just a loop over all the databases in the config file.

Leave a Reply

You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Network Activity

Visitor Boost and Targeted Visitors have received 0 order today and 0 order yesterday.

W3Counter is currently processing 0 queries per second for 13,676 websites.

Website Goodies is hosting 94,379 guestbooks, 12,586 counters and 7,427 polls.

Award Winning Hosts has collected 206 customer reviews of web hosts.

Join this Community