Unicode is the preferred method of representing text outside the ASCII range, which includes text from virtually all non-English languages. Unicode maps characters to integers, and includes a large range of characters, many more than Windows-1252 or ISO-8859-1, the most common character sets used for English documents. Luckily there’s another character set, UTF-8, which can represent Unicode and has wide operating system and browser support.

Handling UTF-8 in HTML

The first step in capturing and displaying non-English text is to deliver your webpages with the UTF-8 encoding. This tells the browser to interpret the text of the webpage as UTF-8 sequences, allowing it to display characters UTF-8 can encode that other character sets can’t. There are two places your page tells the browser what encoding to use — the Content-Type HTTP header, and the Content-Type meta tag.

On an Apache 1.3.12 or later server, you can set what content-type header will be sent by default with the AddCharset, AddType, or AddDefaultCharset directives. These can be set in a .htaccess file if you’re on shared hosting and don’t have access to the server’s main configuration file. You can also specify the character set in a meta tag:

<meta http-equiv=”Content-Type” content=”text/html; charset=utf-8″ />

If you’re using IIS, you can find the content-type setting for each file type under the “Headers” menu in the properties of your web site.

Handling UTF-8 in JavaScript

JavaScript internally works with all text in Unicode, so it’s going to handle UTF-8 encoded text properly without any extra care. However, in the context of web application development, JavaScript is often used to pass off data to server-side scripts. Whether it’s done through rendering HTML (such as constructing an iframe URL) or through AJAX calls, you may need to send text as a parameter in a URL’s query string.

You’ll often see escape() used to prepare the string for use in a URL; it escapes characters like the ampersand that would otherwise result in a malformed URL. However, escape() doesn’t handle characters outside the ASCII range correctly, so the receiving script won’t be able to interpret them. You simply can’t use escape() on Unicode text.

Luckily, all recent browsers support two new JavaScript functions, encodeURIComponent() and encodeURI(). These functions are safe for UTF-8 text, encoding them with the proper escape sequence, as well as everything escape() did to make sure the text is usable in a URL. The encodeURI() function encodes entire URIs — so it leaves characters such as & intact. encodeURIComponent() encodes strings to be individual parameters of a URI, so it encodes all characters except ~!*()’.

In short, if you’re using escape(), use encodeURIComponent() instead. If you’re worried about breaking compatibility with very old browsers, you can test for the existence of the function before using it:

if (encodeURIComponent) {
    string = encodeURIComponent(string);
} else {
    string = escape(string);
}

Handling UTF-8 in PHP

Internally, PHP uses ISO-8859-1/Latin-1 encoding. This character set is much smaller and incompatible with Unicode, which makes handling UTF-8 text difficult. Use of most string functions in PHP will result in the interpreter handling the text as Latin-1, and your output looking like garbled junk. PHP provides a multibyte string function library if your host has compiled it into their PHP build, although it’s sometimes difficult to use and doesn’t provide equivalents to all the string functions PHP normally provides.

PHP handles integers just fine, and Unicode is just a mapping of characters to integers. We can take advantage of that, using some handy functions Scott Reynen has written, to deal with the incoming UTF-8 text. He provides several functions that work well together, allowing you to convert strings to Unicode, convert Unicode to HTML entities for display on a webpage, and do simple string manipulation.

Storing UTF-8 in Non-UTF-8 Databases

The beauty of Scott’s unicode_to_entities_preserving_ascii() function is that it turns UTF8-encoded text into a string that is represented entirely with ASCII characters. All of the chracters outside the ASCII range are turned into their HTML escape sequences, like &#1575;. That means you don’t need your database tables to be set to the UTF-8 character set, which on shared hosting, you may not have the ability to do, and it’s not often the default.

This is useful even if your output format isn’t HTML. Now that you have a way to get the text into the database without losing non-English characters, you can convert it back after you get it out for use elsewhere in your app. PHP has a built-in function which will handle this part for you: html_entity_decode.

$original_string = html_entity_decode($string, ENT_NOQUOTES, ‘UTF-8′);

The caveat, of course, is that you can’t search or sort on those strings in the database properly. If you need those abilities, you need to ensure the database, the table, the columns, and the connection are all set to the UTF-8 character set, and that you don’t use any non-multibyte-safe functions on the strings before inserting or after retrieving them.

And there you have it: Handling non-English text in your web applications even in a shared hosting environment.

1. Create a Non-Root User

   When your server is provisioned, you’ll generally only be given an IP address and a root password. That’s all you need to SSH in to the server as the root user for the first time. It’s bad practice to log in as root for a few reasons — if you log in as a single user and only “su” to root access when needed, you’re less likely to accidentally damage your own system by deleting an important file or providing the wrong options or path to a command. It’s also harder for someone to attempt to break into your server by brute force if they can’t log in directly as root; they’ll need to find a way in as another user then additionally gain root access.

   The first thing you’ll want to do is create a user for yourself to log in as in the future. In most Linux distributions, that’s as easy as typing “useradd [username]” or “adduser [username]“. To set the password for your new user, type “passwd [username]” and you’ll be prompted to supply the new password.

2. Disable Root Logins Over SSH

   Now that you have a second user account, reconnect to your server as that user. Now you can use “su” to gain root access again and edit the SSH configuration file. To do so with the pico text editor, type “pico /etc/ssh/sshd_config”. You’re going to make two changes:

   Find the line Protocol 2, 1, uncomment it, and change it to Protocol 2. Find the line PermitRootLogin yes, uncomment it, and change it to PermitRootLogin no.

   Save the file (CTRL+X, Y) and quit your text editor. Now restart the SSH service (/etc/rc.d/init.d/sshd restart) and it’s no longer possible to log in as the root user over SSH.

3. Disable Telnet

   Telnet is another way to connect to your server, but unlike SSH, is not encrypted. As it’s less secure and just another opportunity for someone to attempt to gain access to your server, it’s best to simply disable the service. To do so, edit the telnet configuration with “pico /etc/xinetd.d/telnet”.

   Find the line that reads disable = no and change it to disable = yes.

   Now, restart the xinetd service with “/etc/rc.d/init.d/xinetd restart” and prevent telnet from starting on boot with “/sbin/chkconfig telnet off”.

4. Install APF (Advanced Policy Firewall)

   APF is a policy based firewall for Linux. It’s very simple to install and configure.

   1. Download it to your server by issuing “wget http://www.rfxnetworks.com/downloads/apf-current.tar.gz“
   2. Extract the file with “tar -xzf apf-current.tar.gz”
   3. Enter the directory that was created, for example “cd apf-0.9.6″
   4. Install APF with the provided script “./install.sh”
   5. Edit the configuration file: “pico /etc/apf/conf.apf”

      Find the line USE_DS=”0″ and change it to USE_DS=”1″ to enable the DShield.org block list. Then you’ll want to edit the ports APF will allow traffic through on your server by finding and updating the following lines as appropriate:

      # Common ingress (inbound) TCP ports -3000_3500 = passive port range for Pure FTPD
      IG_TCP_CPORTS=”20,21,22,25,26,53,80,110,143,443,8443,2222,123,3306,10000,8767,14534,51234″

      # Common ingress (inbound) UDP ports
      IG_UDP_CPORTS=”21,22,53,123,8767,14534,51234″

      # Egress filtering [0 = Disabled / 1 = Enabled]
      EGF=”1″

      # Common egress (outbound) TCP ports
      EG_TCP_CPORTS=”21,22,25,80,443,8443,43,2222,123,8767,14534,51234″

      # Common egress (outbound) UDP ports
      EG_UDP_CPORTS=”20,21,22,53,123,8767,14534,51234″

   6. Start up APF to test your settings. “/usr/local/sbin/apf -s”
   7. If everything looks right (you’re still connected, you can still access whatever ports you need to access, etc.) you can edit the configuration file again and change DEVM=”1″ to DEVM=”0″ to disable development mode.
   8. Restart APF and set it to start on reboot with “/sbin/chkconfig –level 2345 apf on”
5. Install BFD (Brute Force Protection)

   BFD is designed to work alongside APF by scanning your system’s logs for a large number of failed login attempts, and issuing the command to APF to deny that person’s IP address from connecting again. This protects you from attempts at “brute forcing” access to your system, such as repeatedly trying to log in to common account names using a dictionary of common passwords.

   To install BFD, follow the same procedure as above, using the archive at http://www.rfxnetworks.com/downloads/bfd-current.tar.gz.

   The configuration file for BFD is located at /usr/local/bfd/conf.bfd if you want to change any settings, including the ability to have a daily report of failed login attempts e-mailed to you.

6. Install mod_security

   mod_security is a module for the Apache web server that lets you filter out certain requests from being processed. This lets you stop many types of vulnerability exploits on your web server, especially those aimed at sending spam through web forms and issuing commands through known vulnerabilities in some PHP scripts.

   The download and installation process is slightly different depending on what version of Apache you’re running, but it only takes a few minutes to install in most cases. You can download and find documentation at the ModSecurity website.

7. Check Services are Up to Date

   While you just purchased your server, the software that came with it may already be out of date, and potentially vulnerable to newly discovered exploits. A good place to keep track of high risk vulnerabilities in the wild is the McAfee Threat Center.

   Many Linux distributions come with a program such as yum or up2date which you can use to check for updates to installed software on your system automatically. Make use of them on a regular basis and check with the websites of the service creators for updates and patches.

8. Tune Apache and MySQL

   For most people, Apache and MySQL will work relatively well out of the box. If you intend to put considerable load on the server, it’s worth doing a little tuning before you go live. There’s documentation for tuning the settings of both apache and mysql on their websites, and an excellent blog at mysqlperformanceblog.com.

   If you’re going to be running PHP applications, a byte code cache such as APC can significantly boost performance as well.

Follow this checklist to get up and running, ready to host your websites. Remember that managing a server is an ongoing process. You need to keep up with software updates, vulnerabilities, and performance bottlenecks on a regular basis to keep things running smoothly.

This is great news. Now I need to buy a C# book.

Using Mozilla in testing and debugging web sites
A guide to all the tools Mozilla makes available for debugging websites, especially useful for debugging web applications and complicated JavaScript code. From the DOM inspector to JavaScript debugger, you’ll probably find something you didn’t know existed.

bbPress
Forum software from the makers of WordPress. Having explored the code base a bit when writing my first plugin, I can say these guys practice good coding standards, and extending their stuff is straightforward. A whole different world from the hole-riddled mess called phpBB most sites use. Thanks Brendon for the link.

Amberjack
Another site pointed out by Brendon, Amberjack lets you create website “tours” with just JavaScript. It overlays information about webpages over the actual webpage along with navigation buttons to move page to page. A good way to show off features of a web service or product you’re trying to sell.

The Web Developer’s List of Resources
While not the most visually appealing site, I found a couple useful sites from the list, which is editable by the site users by “pinning” or “unpinning” links you like or don’t like.

iWebTools
A collection of extremely useful tools for search engine optimization. I always use their PageRank checker when an update is going on since it checks two dozen different servers at the same time.

PHP UTF-8 Cheatsheet
Writing web applications that deal with multiple languages is a messy process. You can never be sure what encoding is coming in, but you can make a decent attempt at it with UTF-8. This cheatsheet shows you what’s involved in handling UTF-8 data in PHP.

Snoopy: The Web Client Class for PHP
One of my favorite PHP classes. I don’t use it much anymore, but I used to; it makes grabbing webpages, stripping them of HTML, retrieving all the links on pages, submitting forms, sending and receiving cookies, and everything else a browser would do easy in PHP.

Netvibes
My favorite among all the personalized homepage services. It’s quick and responsive even on older computers, I’ve never encountered a bug, and you can make virtually anything into a Netvibes module if there isn’t already one made. I use it to organize my RSS feeds, my Google calendar, check the weather and check my web stats (RSS feeds from W3Counter).

AnyBrowser Screen Size Test
A simple tool which opens a URL in a specific sized browser window. The Web Developer Toolbar for Firefox handles resizing for me, but I still use this to test pages at 800×600 in Internet Explorer, as I have been for almost 7 years now.

Archive.org
Look at previous versions of any webpage going back more than 10 years. Great for checking out a domain you’re interested in buying or advertising on. Also great for verifying you don’t overestimate how many years you’ve been using some site (like AnyBrowser).

Example.com
Mysite.com and yoursite.com get far too many free incoming links from casual forum posts and software documentation. When you’re writing instructions telling someone to provide their website address, use example.com, example.net or example.org as your example. That’s what they’re specifically reserved for.

CSSTYPE v2
Pixel perfect text styles for your website. Change the appearance of the text with the selection and input fields, see the results instantly, and click the CSS link to grab the styles for your own site.

Hyperlink Cues with Favicons
CSS and JavaScript which automatically displays the favicon icon of each website a page links to next to that link.

Programming Books
A new, user-ranked directory devoted entirely to programming books. Most of the books I’ve read are recommended there, as well as quite a few that I’d like to read this year.

When you read code, imagine typing it by hand.
Tips for better understanding others’ code, especially when it’s in an unfamiliar language.

YouTube is Better Than the Superbowl
A 75-second film Dove released at no cost on YouTube that brought a bigger traffic spike than their Super Bowl commercial.

Vanilla
A relatively new open-source PHP message board package I absolutely love. It’s well written, extensible through plugins, actively maintained, and devoid of unnecessary features that could be added as needed for individual boards.

Design Snack
“Digg” for web design inspiration. A website gallery which uses user votes to decide what gets featured on the homepage.

adClustr
A site dedicated to helping you blend text advertising into your website design. CSS snippets and images to help style text ads from AdSense, YPN, and Text-LinkAds.

Sweetie
A set of 150 icons for use in web applications. Free for commercial use and provided in PSD and PNG formats.

Feel free to share anything interesting you’ve found this week in the comments; if I do too maybe it’ll end up in my post next week. I’m off to watch the live webcast of Bill Gates giving the keynote at CES.

##NOLIGHTBOX##