I don't normally use Hotmail to write messages, but I've just tried the spell checker in Hotmail and I get the same message as you do.
In Firefox, go to Tools>Options>Advanced>General tab and, under 'Browsing', make sure that 'Check my spelling as I type' has a tick by it. This should give you the red line under a misspelt word. You can then right click on it and select the right word from the list given.
If you want a separate spell checker, have a look at
SpellBound 3.0.6 which works well with Firefox 3xx, up to and including the current version, 3.5.3. I have it and use it a lot.
Either way, you need the British English Dictionary as mentioned above. You may already have it, so check your Add-ons (Extensions part) to see if it's there. If not, you can get it
here.