Cerebral Soup - The Archives

Japanese Moblogging + WordPress + Postie + WP-Cron - a How To

Update: Postie version 0.9.9.3.2 handles Japanese text OK! No need for the editing listed below.

The path to finding the solution to posting keitai mails to WordPress was a winding rocky one. I hope this post will save you from what I went through.

I won't say this is "easy" (mainly because it drives me crazy to read that comment when you are wrangling with something) - but if you follow the instructions carefully you should be rockin'. I can't guarantee it - but it worked for me!

Of course if you are using a Japanese cell phone to post messages only in English you probably don't need to go through this. English text messages seem to post fine just using WP-mail.

The Issue: Encoding Hell

Japanese keitai mails send text in ISO-2022-JP. This means they will post onto your WordPress blog in mojibake. This is a bad thing.

First step: Go and download this stuff -->

WordPress 2.0.1
Postie plugin 0.9.9.2
WP-Cron 1.4

Install it:

According to the instructions either on the WordPress site or in the included Read Me files.

Make sure your blog is charset utf-8.

Go to Plugins in the WP (WordPress) admin interface. Activate Postie, and WP-Cron (and Cronless Postie as well but I forget why this may be important).

Don't bother activating WP-Cron-Mail.

Go to your web host's control panel:

You need to set up a special email address just for Postie to collect the mails you wish to post.

Make sure that your mail character option is set to ISO-2022-JP.

While you are here check that your host has "iconv" enabled in the PHP build. This will probably be in the PHP information section. If it is not, or you can't find it, bug your host to enable it or tell you if it is. There are solutions if you don't, for both Windows and Unix servers (Windows: .dll files.com, Unix: gnu.org - good places for extra info on this).

What the hell is iconv? It's the magic ingredient to make this work. If you have it enabled - cool bananas - you probably don't want to know much more about it (but if you do - php.net).

If you are having problems with the encoding after following all the steps here check that you have all the files will the character mappings necessary for iconv on your server.

(We are not there yet! Keep reading!)

Code changes you need to make in the Postie plugin:

The file you need to alter is postie-functions.php. Before you open it up, copy it and rename the copied file something like postie-functions_original.php. This is so you have a back up of the original code if you botch things up.

Open the file in your preferred editor (Dreamweaver, BBEdit, Notepad etc.).

You need to change this code (line 763):

function ConvertToUTF_8($encoding,$charset,&$body) {
$charset = strtolower($charset);
$encoding = strtolower($encoding);
if (strtolower($charset) == "iso-8859-1") {
$body = utf8_encode($body);
}
}

To this:

function ConvertToUTF_8($encoding,$charset,&$body) {
$charset = strtolower($charset);
$encoding = strtolower($encoding);
if (strtolower($charset) == "iso-8859-1") {
$body = utf8_encode($body);
} elseif (strtolower($charset) == "iso-2022-jp") {
$body = iconv("ISO-2022-JP//TRANSLIT", "UTF-8", $body);
}
}

This will convert the Japanese mail content and header into utf-8 when it grabs the mail from the mail box.

But if you want to post pictures or videos with your mail (and who doesn't? After all that's why we have the snazzy camera keitais) you need to change another part of the code too. Adding an attachment means the subject will end up mojibake. This is bad.

Go to (about) line 1403.

You will change this code:


function GetSubject(&$mimeDecodedEmail,&$content) {
$config = GetConfig();
//assign the default title/subject
if ( $mimeDecodedEmail->headers['subject'] == NULL ) {
if ($config["ALLOW_SUBJECT_IN_MAIL"]) {
list($subject,$content) = ParseInMessageSubject($content);
}
else {
$subject = $config["DEFAULT_TITLE"];
}
$mimeDecodedEmail->headers['subject'] = $subject;
} else {
$subject = $mimeDecodedEmail->headers['subject'];
HandleMessageEncoding($mimeDecodedEmail->headers["content-transfer-encoding"],
$mimeDecodedEmail->ctype_parameters["charset"],
$subject);
if (!$config["ALLOW_HTML_IN_SUBJECT"]) {
$subject = htmlentities($subject);
}
}
return($subject);
}

To this:


function GetSubject(&$mimeDecodedEmail,&$content) {
$config = GetConfig();
//assign the default title/subject
if ( $mimeDecodedEmail->headers['subject'] == NULL ) {
if ($config["ALLOW_SUBJECT_IN_MAIL"]) {
list($subject,$content) = ParseInMessageSubject($content);
}
else {
$subject = $config["DEFAULT_TITLE"];
}
$mimeDecodedEmail->headers['subject'] = $subject;
} else {
$subject = $mimeDecodedEmail->headers['subject'];
HandleMessageEncoding($mimeDecodedEmail->headers["content-transfer-encoding"],
$mimeDecodedEmail->ctype_parameters["charset"],
$subject);
if (!$config["ALLOW_HTML_IN_SUBJECT"]) {
$subject = htmlentities($subject);
}
// escape sequence is 'ESC $ B' == 1b 24 42 hex.
if (strpos($subject, "\x1b\x24\x42") !== false) {
// found iso-2022-jp escape sequence in subject... convert!
$subject = iconv("ISO-2022-JP//TRANSLIT", "UTF-8", $subject);
}
}
return($subject);
}

Save. Close file. Replace current postie-functions.php on your server.

Configure Postie:

In your WordPress admin go to Options > Configure Postie. Follow the instructions there to configure it. Make sure the mail is set to our charset buddy utf-8.

Code changes you need to make in the WP-Cron plugin:

This technically has nothing to do with the encoding issue. But if you try a test post now you still end up with mojibake. Whiskey Tango Foxtrot I hear you cry! WP-Cron when activated, will use the original WP-mail to post the message. WP-mail will bugger it up. You need to tell WP-Cron to access Postie's get_mail.php file.

This little (but essential) trick was found on a German blog (my Uni degree sometimes comes in handy after all): foobla.

Go to the WP-Cron folder. Open wp-cron.php. Go to line 57.

Change this code:

function wp_cron_15_exec() {
do_action('wp_cron_15');
}


function wp_cron_hourly_exec() {
do_action('wp_cron_hourly');
}


function wp_cron_daily_exec() {
do_action('wp_cron_daily');
}

To this:

function wp_cron_15_exec() {
file_get_contents('http://www.yoursite.com/wp-content/plugins/postie/get_mail.php');
do_action('wp_cron_15');
}


function wp_cron_hourly_exec() {
file_get_contents('http://www.yoursite.com/wp-content/plugins/postie/get_mail.php');
do_action('wp_cron_hourly');
}


function wp_cron_daily_exec() {
file_get_contents('http://www.yoursite.com/wp-content/plugins/postie/get_mail.php');
do_action('wp_cron_daily');
}

Note: change "www.yoursite.com" to your domain name. Also change the path if needed to point to the Postie get_mail.php file on your server.

Let's testing!

Grab that keitai and send a test message in Japanese to that special secret email address you set up. If the Internet Deities are willing, your message should be converted from ISO-2022-JP to utf-8 and post correctly.

If so:

Jump for joy!

If not:

Weep a bit and then look at the next section.

Troubleshooting

If you still get mojibake try the following -->

- check that you've copied the code correctly
- that your database can handle Japanese
- that the files have been uploaded correctly via ftp
- permissions are correct

Also see WordPress troubleshooting guide for stacks more information.

Note: You can edit the Postie and WP-Cron files directly through the WordPress interface, Plugins > Plugin Editor. I chose to edit them in an external editor because I prefer to wade through hundreds of lines of code highlighted in pretty colours. Your choice. Do copy any files though before you edit them. It's a lot quicker to do this then go back and download the whole plugin again.

Thank you and big smoochy kisses to:

You may be surprised to learn after reading this that I am not a PHP guru and simply pull this stuff out of thin air. I would therefore like to thank the following wonderful people for their help:

zengargoyle who not only sent me code snippets to try (that worked!), but also explained where to bung them into the code, and what to watch out for

drivingmenuts for listening to me ranting on in iChat and offering solutions

Richard Northcott for taking the time to look through the code and offer food for thought

Dr Dave who tackled this problem in a different way, which unfortunately did not work with the hosting service used

JapanBloggers Group several people offered help and suggestions - ta!

MonkeyFilter I hope I never have to hurt Monkeys' brains with another techie post like that one

And to everyone who comforted me during the nightmare of trying to solve this!

Posted by mjd-s on February 25, 2006 2:58 AM