In response to my emails to the Zend Framework I18N list and my previous post, Thomas, the author of the Zend_Translate framework items mailed back to the list here:

> 2) gettext is a more expensive version of using the arrays backend.

No… it’s a less expensive version. What takes time is reading the original
source. Your processor is always faster than your harddrive.
It is better to do some computations than reading a bigger file. And mo
files are much smaller than the same sized array files.

This still seems wrong to me, so I’ve done a bit more analysis. I have now got XDebug up and running in my portable environment, so I can really see the details of the costs. Now, to caveat all this, I’m running all this from a USB key on a laptop that’s doing a number of other background tasks, so, the performance is not isolated. Due to this I’ll be looking at percentages of time in Wincachegrind, not actual execution times.

Now, to test this I have generated two files. One of which is a .po containing 1000 phrases which I have compiled to a .mo file. The other is a PHP array in a PHP file containing the same 1000 translations. I generated this with a script, the translations are a bit simple:

From the .po file:

msgid “String 0″
msgstr “String Translated 0″

From the .php file:

‘String 0′=> ‘String Translated 0′,

I have then written a simple PHP file which translates 50 of these items. A reasonable enough test I think. Firstly, to test the translation using the fast Zend_Translate gettext options:

require_once 'Zend/Translate.php';
i = new Zend_Translate('gettext', '/development/language/test.en.mo', 'en');
 
function _($s)
{
    global $i;
    $s = $i->_($s);
    echo($s."<br/>\n");
}
 
_('String 1');
_('String 2');
...

I then ran this file and loaded the cachegrind output into WinCacheGrind. 87.88% of the execution time was spent in Zend_Translate_Adapter_Gettext->_loadTranslationData. Performing translations took 1.99% of the time.

Next I used my PHP array and the Zend_Translate array backend:

require_once 'Zend/Translate.php';
require_once '/development/language/test.en.php';
$i = new Zend_Translate('array', $LANGUAGE, en);

(The rest of the file remains the same). I then ran this and checked the output. loadTranslationData took 78.36% of the time. Performing translations took 2.86% of the time.

My third test was just to use the test.en.php file and a simple translation function:

require_once '/development/language/test.en.php';
function _($s) {
  global $LANGUAGE;
   $s = (array_key_exists($s, $LANGUAGE)) ? $LANGUAGE[$s] : $s;
  return $s;
}

The first thing to note was that the Zend_Translate items took over 20ms each. This one not using Zend at all took 2.8ms. The require_once statement took 1.83% of that time. Then it was just repetition of an un-recordably-fast translation 50 times.

So what do I draw from this? I draw from this that for simple translations, you can’t beat a very, very simple system with just an array of translations. I haven’t looked in any depth at the other services offered by Zend_Translate, but it does allow you to add multiple translations and translate in multiple languages. But, do you have a use-case for that?

If your UI needs to display in a single language, but translate that language, take the simple approach. It needs a little extension to support modular languages, but look at the PHPBB3 implementation and you can’t go far wrong. That loads modular translation files (just to keep that trivial require_once cost down) each of which array_merge’s back into a single translation array which is key’d by constants.

Fast.

My cachegrind files for your reading pleasure:
Zend_Translate - Array
Zend_Translate - Gettext
Non-Zend_Translate

Popularity: 37% [?]

When you are working on a web site or web application, something that little thought is spared for at development time all too often is it’s performance. You need to know where your bottlenecks are and what the costs of each architectural and implementation choice you make are. You should routinely profile your application’s core functions to see how they behave. How do we do this? You could put little timers into the code and log timings to see how things work, or you could use XDebug.

XDebug is a PHP extension that provides a number of critical features to developers. It supports step through debugging (assuming your code editor can hook into it) and it supports profiling of your scripts. This gives you a detailed breakdown of every command executed in your application, how many times it was executed and what it cost.

This is invaluable when tracing your performance work. You can identify exactly which routines are costing you too much. It can give valuable insight into the performance landscape of your software. So I’m going to hook it up into my development environment on it’s USB key.

Firstly, pop over to the XDebug site and download the relevant version for your PHP version. I downloaded the Windows Binary of 2.0RC4 for PHP 5.2.1+. XDebug is a “Zend Extension”, it’s not a standard PHP Extension, it extends the Zend Engine that powers PHP. This changes how we configure it in php.ini and means it doesn’t have to go on the extensions folder of your PHP install. But, I keep it there for consistency. Once placed in that file you need to edit your php.ini, the commands can go anywhere in the file, but, I placed them after the PHP Extension loading commands, again for consistency:

zend_extension_ts=/development/php/ext/php_xdebug-2.0.0rc4-5.2.1.dll

Note that this is loaded with the zend_extension_ts command instead of the extension command (the ts denotes Thread Safe mode) . Also note that we specify the full path to the extension. The zend_extension_ts (and other zend_extension commands) need the full path as they don’t pay attention to our extension directory command.

Once this is done, go to your PHPInfo() test page and check, you should have XDebug information included:

Ok so far so good. If you have a page which currently throws a php error, go check it now. You’ll notice that just having XDebug installed gives you much more information. XDebug makes developing easier just by being there.

Now, we’re mainly going to use it to profile performance of our applications and third party libraries, so we need to enable profiling, this is done with a couple of new entries in php.ini. I placed them just after my command to load the extension so it’s all in one place:

xdebug.profiler_enable = 1
xdebug.profiler_output_dir=/development/

Now restart Apache and hit your PHPInfo() page. In the development folder on your USB Key you will have some files called cachegrind.out.[some number]. This is the profiling information in it’s raw form and of no use to you on it’s own.

You need a cachegrind analysis program. I use Wincachegrind as I’m on windows. You can use this to open up the cachegrind file and see what took what time. Visit yourPHPInfo() page, pick up the cachegrind output and take a look. You can see a lot of detail.

I don’t propose to detail how to use Wincachegrind and do a full analysis, a bit of poking should show you what’s going on. But, I’ll be using XDebug and WinCacheGrind to get under the covers of some third party libraries I’m considering for use in the development of Multiblog, so we’ll see more information then.

Popularity: 31% [?]

The web is global.

Lots of websites do not cope with this. They do not provide a user setting for the language and deliver their content in that language.

Clearly, this is bad. If you are producing an application, like Multiblog, then you need to make it international. It needs the UI at least (content is a more thorny issue) to work in the users preferred language. Otherwise they will experience friction trying to use the confusing foreign thing.

There are a lot of ways to achieve this. Geeklog and PHPBB3 use arrays to translate content and allow the user to pick things. Drupal uses the GNU GetText system. And there are other approaches.

Choosing the right approach and using it correctly is difficult. I’m currently experimenting with approaches for Multiblog and other projects. I’m currently looking into the very interesting Zend Framework’s Zend_Translate class. This allows a number of different approaches, including both Array and GetText.

GetText appears to be the recommended choice. There are a number of free tools that can generate your translation files, as the translation files are not human readable. It’s fast and threadsafe. The Zend Framework Manual offers some advice on how to structure your translation files. There are several suggested methods, but, there is no suggestion of how to structure your translation modules.

The question I asked was “What’s the best practice?”, and no-one seems to know, so I guess I need to figure it out for myself from basic principles.

Now, GetText was written to provide internationalisation for the GNU software. Including the core of the Linux OS. Here, the GetText file is (I assume) parsed once at start up and held in memory to translate as things go. Web applications are different. Every page view is essentially a new start up. That GetText translation source is going to be loaded hundreds and thousands of times. Not just once on boot of the web server.

So, if we want to get this right, we need to know what our best bet is. Do we want a monolithic all translations file, or do we want to modularise this file and load it as needed? Does it use the file like a database and seek things out, or does it parse the whole thing every time and process it internally?

I’ve done some simple testing. I produced a basic test catalogue with poEdit and compiled a mo file from it:

msgid ""
msgstr ""
"Project-Id-Version: Test Zend GetTextn"
"POT-Creation-Date: n"
"PO-Revision-Date: 2007-06-21 12:20-0000n"
"Last-Translator: THEMike n"
"Language-Team: n"
"MIME-Version: 1.0n"
"Content-Type: text/plain; charset=utf-8n"
"Content-Transfer-Encoding: 8bitn"
"X-Poedit-Language: Englishn"
"X-Poedit-Country: UNITED KINGDOMn"
"X-Poedit-SourceCharset: utf-8n"

msgid "This is a test."
msgstr "[Translated]This is a test.[/Translated]"

I then wrote a simple test harness PHP file which loads a Zend_Translate using gettext and translates a single line. Before performing a translation, I var_dump the Zend_Translate instance to see what’s in it:

  <?php
  /* Configuration: */
  define('PATH_TO_ENGINE', '/development/engine/');
  define('PATH_TO_LANGUAGE', '/development/language/');/* Put engine on the include path */
$curPHPIncludePath = ini_get( 'include_path' );
if (defined( 'PATH_SEPARATOR')) {
    $separator = PATH_SEPARATOR;
} else {
    // prior to PHP 4.3.0, we have to guess the correct separator ...
    $separator = ';';
    if( strpos( $curPHPIncludePath, $separator ) === false ) {
        $separator = ':';
    }
}
if (ini_set('include_path', PATH_TO_ENGINE . $separator . $curPHPIncludePath) === false){
        die('Buggered');
}
require_once 'Zend/Translate.php';
$t = new Zend_Translate('gettext', PATH_TO_LANGUAGE.'test.en.mo', 'en');
echo('<pre>');var_dump($t);echo("</pre><hr/>n");echo($t->_('This is a test.'));?>

The result of the var_dump being:

object(Zend_Translate)#1 (1) {
  ["_adapter:private"]=>
  object(Zend_Translate_Adapter_Gettext)#2 (6) {
    ["_bigEndian:private"]=>
    bool(false)
    ["_file:private"]=>
    resource(21) of type (stream)
    ["_locale:protected"]=>
    string(2) "en"
    ["_languages:protected"]=>
    array(1) {
      ["en"]=>
      string(2) "en"
    }
    ["_options:protected"]=>
    array(1) {
      ["clear"]=>
      bool(false)
   }
    ["_translate:protected"]=>
    array(1) {
      ["en"]=>
      array(2) {
        [""]=>
        string(339) "Project-Id-Version: Test Zend GetText
POT-Creation-Date:
PO-Revision-Date: 2007-06-21 12:20-0000
Last-Translator: THEMike
Language-Team:
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Poedit-Language: English
X-Poedit-Country: UNITED KINGDOM
X-Poedit-SourceCharset: utf-8
"
        ["This is a test."]=>
        string(40) "[Translated]This is a test.[/Translated]"
      }
    }
  }
}

As you can see, before I’ve even called a translate call, the entire mo translation catalogue has been loaded into memory and parsed internally to form a PHP array. Which is then used for translation.

Clearly, this indicates a very modular translation system. I would want a core.lang.mo file for “common” translations used througout the application and then a controller.lang.mo file for each controller that had that controller’s specific phrases in it which is only loaded by that controller.

However, note that the translation is done to a PHP array. Essentially, it seems the gettext translator is really a front-end loader of the array translator. So why not use the array translator?

The only downside I can see is that it’s harder to get non-programmers to generate valid PHP arrays when supplying your translation. Other than that, anything that the PHP extension does to optimise compilation and processing of PHP code will kick in and give you a significantly improved performance. Put extra things on top of that like the Zend Optimisers and so forth, and you have a compelling reason to use highly modular array based translation.

The problem then remains getting valid PHP array files back from your translators, and frankly, that can be solved by writing a simple front end for your translators so they have a GUI to use.

Popularity: 48% [?]

My internet connection at home is gone. I’m migrating ADSL ISP’s and it appears my old ISP has cut me off and my new ISP is not quite ready for me. “Due to demand, expect to wait three business days beyond your activation date”. Last night I tried to write some code.

Now, coding at home is always painful. At work I have dual monitors and a reasonably powerful box with plenty of RAM (for the coding I do, which is very little and mostly scripting, for real programming that my team do, not really powerful enough. But since I’ve wandered up to manage them, it doesn’t matter that my box sucks).

At home, however, I’ve got a 4 year old laptop that cost £300 when I bought it. With a 15″ screen. At home I’m doing the same kind of scripting, but, it’s just less productive due to raw power of machine and lack of screen real estate.

And now I have no internet.

I can’t remember when exactly I made the transition from referencing books (”In a nutshell” or “Pocket reference” books when I needed something, or checking a “Cookbook” for a recipe, or perhaps even using the MSDN CD) to relying 100% on Google and the wider internet. But it seems I’ve done so. And loosing the internet when I’m coding is like loosing a limb.

It’s easy enough to deal with not having the internet to surf and in fact distract me. But not having it to bail me out when I’m stuck is terrible. Jeff Atwood just said, entirely in passing:

I can barely program these days without an active internet connection; I feel crippled when I’m not networked into the vast hive mind of programming knowledge on the internet.

And he’s right. It’s horrible. His main point was about coding with other people so that you can bounce off each other etc, which is also important. But the lack of the internet as the solution to all your dead-ends is even worse.

Popularity: 14% [?]

I’ve been keeping an eye on all the Ruby and Ruby on Rails hype. Thinking, wow, how exciting, I wish I had time to have a dabble. I really do want to get an environment sorted out and have a play, so I opened this article with great excitement and started to read. The excitement soon waned.

Ruby removes unnecessary cruft: (){};

• Parenthesis on method calls are optional; use print "hi".
• Semicolons aren’t needed after each line (crazy, I know).
• Use “if then else end” rather than braces.
• Parens aren’t needed around the conditions in if-then statements.
• Methods automatically return the last line (call return explicitly if needed)

Ruby scraps the annoying, ubiquitous punctuation that distracts from the program logic. Why put parens ((around),(everything))? Again, if you want parens, put ‘em in there. But you’ll take off the training wheels soon enough.

The line noise (er, “punctuation”) we use in C and Java is for the compiler’s benefit, not ours. Be warned: after weeks with Ruby, other languages become a bit painful to read.

I’m sorry, but none of that is rocket science. I can code in a large number of languages, many of which don’t require a semi-colon to terminate a line. Several of which use if then else end instead of braces. Several don’t need parens around if statement clauses and I’m sure at least one or two return the last statement.

One of the languages that springs to mind is BASIC, and it’s many flavours. And yet, hard-core Java/C/C++ nuts switching to Ruby wouldn’t touch something like Basic or VBScript for exactly the same reasons they declare Ruby to be great.

Notice the points about pointless cruft and punctuation? Then notice some of the examples that follow on:

dictionary = { :cat => "Goes meow", :dog => "Barks loud."}

That strikes me as pretty crufty. Other syntax are equally illogical:

x = a || b || c || "default"

Frankly a lot of the reasons I see for Python or Ruby being so productive and great are the fact it’s just damn fast to script. Something those of us who’ve been scripting for years have known for a long, long time. I can knock something up in any of half a dozen scripting languages really fast. And have been able to since ‘96 when I first got into script languages. But, I’ve always been knocked by Java developers who think Java is the one true language. Seems that the Java guys and the other compiled language drones are finally turned on to scripting and blissfully unaware they’ve been slagging it off for years when others of us have been doing it.

Popularity: 28% [?]

So I’ve reached a point where I know what the application I will write needs to do. The next logical step is to design that application. However, before I do a design, I need to have a “big picture” idea of the software architecture and I need to know what third party code libraries I can use.

As I noted, choosing a language is choosing a platform:

This was something I was planning to move on to talk about in more detail later. When you pick a language, you do need to look around at the choices of libraries available to you. PHP is lucky. It’s mature. There are a lot of libraries out there.

But, you must be very careful when confronted by such choice. Take a look at the options available for templating. Pear (a major source of libraries) have several implementations. There’s Smarty, and numerous other “just templating” libraries. The new Zend Framework also has a templating implementation.

I need to make a very careful set of choices when choosing what libraries to build on, if you choose the wrong items, and get them fundamentally embedded into your application, at a later date you have a much bigger job to replace them. Possibly a fundamental re-write.

Now, I’m wanting to make use of standard design patterns to make sure my application architecture is maintainable, controllable, extendible and scalable. So I’m going to be following standard Object Oriented software design using standard design patterns. I’m going to utilise Model View Controller (MVC, or in Microsoft Land Model View Presenter), Database Abstraction Layers, Singleton Patterns and other things.

AS noted, PHP is very mature, there are a lot of choices. Some of those choices are more mature than others. Some are big and bloated, some a trim and limited. It’s essential I make the right choices.

I’m going to spend the next few articles examining database abstraction libraries and templating engines. Then move on to look at design pattern libraries to support an MVC implementation. But, first I need a good set of criteria to compare them on. I need to look at the following:

• Performance
• API details (will it do what I want flexibly enough, will it lend itself to integrating with other libraries and my own code?)
• Documentation (will I be able to figure out how to use it?)
• Is the project alive and moving or is it dead?

Critical to me most of all is performance, I don’t want the PHP engine to have to process thousands of kb of useless code that will never be called by my code, just to get some trivial feature. So I need to find a way to asses this. The first stop on this route is to get some profiling tools installed into my development environment, so that’s what I’ll do next.

Popularity: 24% [?]

As I noted in my last post, adverts put people off. As does paying for a service. Jeff Atwood points out that registering for a service also puts people off.

If your application requires users to log in, don’t underestimate the impact of the login barrier you’re presenting to users. Consider utilizing anonymous, cookie-based accounts to give users a complete experience that more closely resembles the experience that named users get. By removing the login barrier and blurring the line between anonymous users and named users, you’re likely to gain a lot more of the latter.

Just asking users to login or register scares them away. And he’s right. When I’m looking at new sites online, I don’t want to sign up to find out if I want to sign up. I want to poke around before I create yet another registration, yet another version of my information. Jeff’s post was very timely, I need to take this into account when building Multiblog.

The barrier to entry needs to be so low anyone can get over it. Or no-one will.

However, given it’s nature this is going to be difficult with Multiblog. I guess it’s possible to create accounts tied to an anonymous cookie and allow them to create blog accounts on that account. But it’s hardly secure. What I need to do is allow anonymous users to wander round everything, toy with configuring accounts, even draft and hit post on entries. Then I’m afraid I’m going to have to stop them and make them sign up. But, this has to be painless.

How do I do this? There are numerous options I could take. One idea is to prompt for an OpenID, the idea behind OpenID is that it’s an Open system for authentication, everyone should support OpenID and then you can allow people to post behind an identity on any site without signing up.

I think that’s great for many applications, such as posting comments on blogs (like this one), but it doesn’t work for services like Multiblog. Even the OpenID site says so:

This is not a trust system. Trust requires identity first.

There has even been spam originating from OpenID sources. So we need something else.

I’m going to have to go with an infrequently used option. I’m going to prompt for an email address. That’s it. This isn’t a site with profiles, so there is no need for usernames or anything else. When you first start using Multiblog I just need a globally unique way of referencing people. And that’s their email address. Couple that with a random password and then you have a full sign-up and authentication system. On entry of the email I’ll send an activation password. On logging in with that password, I’ll set everything up and have the user back at the “are you sure you want to post this entry?” stage. Job done.

As frictionless as it is, it’s a threat. This is predicated on at least one free post to all users. Which is an exploitable hole. People could buy a domain and have an infinite number of email addresses. So, I’ll have to make blog accounts (the accounts Multiblog posts to) be unique, monitor sign-ups (not expecting high volume) and take appropriate other anti-spam/abuse techniques.

But basically, I think it’s a good approach. It will allow users to sign up in seconds and to explore 95% of the application without an account. I hope Jeff likes it more than an vanilla textarea control.

Popularity: 56% [?]

For web based businesses, being able to accurately report the number of unique visitors to your site and the amount of time they spend on your site is essential you need to be armed with this information and the evidence to back it up in order to attract good rates from advertisers.

Back in the old days, you’d just parse your weblogs and count “hits”, every access was a hit, no matter what it was. That was fine until pages linked in Javascript, CSS, Images etc. Then we had to move to “Page Views”. We just looked at the specific “root” file for a page, and discounted the other objects that were loaded as a result.

Of course, then that doesn’t allow for pages like the request-o-meter on snakenet, which auto-refreshes to show people when they can next request. Does this really count as another page view? Not to advertisers mostly. Advertisers don’t want to see a buy ads on a site with 10,000 page views per hour, if that page view count comes from 2 users.

And so, sites and advertisers started to track with cookies, but that’s no good any more:

BBC NEWS | Technology | Web counting tools ‘need change’:

The way web audiences are measured could be ripe for an overhaul, according to two reports out this week….

In comScore’s study, an analysis of 400,000 home PCs in the US found that a hardcore minority of web users are clearing their cookies from their computers on a regular basis.

This causes servers to deposit new cookies which in turn could lead to an over-estimate of unique users to a particular website.

It found that 7% of computers accounted for 35% of all cookies, which extrapolated could mean the size of a site’s audience is being overstated by as much as 150%, said comScore.

Justly, many users don’t like cookies. And these are the most web literate users. It’s an invasion of your privacy they argue. People trying to profile you and track your habits. So they can better target their advertising to the right sites and the right people.

Perhaps unsurprisingly, comScore offers a very different approach to audience measurement - using the panel-based system favoured by the TV and radio industries which relies on using a representative sample of net users to gauge behaviour.

This works for TV, because TV is pretty limited. It takes a big chunk of capital to set up a TV channel, to make a TV show, to get someone to air it. It’s free to set up a web site. And thus, there are a lot of them. A lot of competition, lots of sudden start-ups. Things are niche. Take social networking, there’s MySpace, Facebook, LiveJournal, LinkedIn. Loads of them. People gravitate towards different ones on all sorts of random basis.

You can’t generalise traffic to one of those sites. There are new services getting tons of traffic from markets you can’t generalise. There are massively trafficked sites I’ve not heard of.

Perhaps the complexity is just in finding out how to come up with sensible representative survey groups. With the social networking it’s a bit easier. People who’ve been online a long time are more likely to use LiveJournal, which isn’t a true social networking site, but has aspects of it and has been around a long time. School kids and students will gravitate to facebook. Professionals to LinkedIn. That’s an easy one. But to music sites? You can’t judge the internet radio sites people visit that easily.

I know of several sites with no reason to choose between them that can be generalised. It’s down to the community. Some people fit in better elsewhere. So how are you going to be able to calculate their sites on that basis?

But, I guess this approach isn’t really about figuring out the traffic for every site, but just for the big sites, that everyone knows about.

So, they’re playing round with ways to get an accurate idea to estimate the number of unique people visiting bigger sites so they can sell advertising.

People expect the web to be free. They don’t want to pay to use facebook. Facebook need to make money. Even if it’s a cool idea that someone sets up because they think that would be a cool service running a big site is not free. The internet is only free to the consumer. The middle-men, the people who run the sites have to pay money to the bandwidth people, the power people. They don’t get that free. So you have to find a way to get money that doesn’t directly take it out of the pockets of your users.

And that’s advertising.

And your users hate it.

There are many things around that exist purely to take the advertising out of the web. I use AdBlock Plus in firefox to strip the vast majority of adverts out of every site I visit. I combine this with Greasemonkey and some user scripts to take other adverts out of sites I use a lot. And so do a lot of other people. Plus people use tools to block cookies that are used to track their use of sites and their browsing habits. Some people just purge all, but there are tools more like AdBlock Plus that do extend and enhance Firefox’s user permissions to blacklist and whitelist cookie providers. Spybot, Adaware and other tools rip out these tracking cookies.

So on the one hand, you’ve got the problem accurately reporting your usage and getting that user base up to get revenue from advertisers. On the other hand you have a hard-core minority of web users trying to eliminate that revenue stream because they don’t want your crappy invasive adverts and they don’t want to be profiled and tracked.

You have to find another revenue stream for your online business. Adverts alienate users and dilute your brand image. But people don’t want to pay for things. I guess that’s a bit of a problem for me isn’t it? On the one hand, I won’t attract users and keep users if I make Multiblog a pay service. I will have problems attracting advertisers to start with and producing evidence of my true user base to get them later on. And if I do get adverts, many of my users will be blocking adverts or alienated by them. Blogging to multiple-blogs in one hit is a very web literate thing to want to do. My core demographic is the advert avoiders.

So what do I do?

The answer isn’t great. I’m not going to waste my time building an advertising funded service. My users will ad-block. I’m not going to get anywhere charging for everything. I have to follow the web equivalent of the shareware model. I have to start free, then I have to introduce charges for advanced features later.

It’s essential that I’ve identified this model early, because I’m going to have to design user accounts and features around the fact that some features or sub-features will one day switch to paid user only, and I may even have tiers of paid users. Critical thing. if you miss that at the start, then you’re really not going to manage to deliver the payment extensions later on.

Popularity: 18% [?]