Recently, I was having some issues with a site that was attempting to use UTF-8 in order to support multiple languages. Basically, you could enter UTF-8 characters — for instance, characters with umlauts — but they weren't going through to the web services or database correctly. After more debugging, I discovered that when I turned off javascript on the site, and used the degradable interface to submit the form via plain old HTTP, everything worked fine — which meant the issue was with how we were sending the data via XHR.

We were using Prototype, and in particular, POSTing data back to our site — which meant that the UI designer was using Form.serialize() to encode the data for transmission. This in turn uses the javascript function encodeURIComponent() to do its dirty work.

I tried a ton of things in PHP to decode this to UTF-8, before stumbling on a solution written in Perl. Basically, the solution uses a regular expression to grab urlencoded hex values out of a string, and then does a double conversion on the value, first to decimal and then to a character. The PHP version looks like this:

$value = preg_replace('/%([0-9a-f]{2})/ie', \"chr(hexdec('\1'))\", $value);

We have a method in our code to detect if the incoming request is via XHR. In that logic, once XHR is detected, I then pass $_POST through the following function:

function utf8Urldecode($value)
{
    if (is_array($value)) {
        foreach ($key => $val) {
            $value[$key] = utf8Urldecode($val);
        }
    } else {
        $value = preg_replace('/%([0-9a-f]{2})/ie', 'chr(hexdec($1))', (string) $value);
    }

    return $value;
}

This casts all UTF-8 urlencoded values in the $_POST array back to UTF-8, and from there we can continue processing as normal.

Man, but I can't wait until PHP 6 comes out and fixes these unicode issues…