After spending the day trying to figure out why firefox was not caching images from my website even though chrome seems to honor the caching headers from the server correctly, it turned out that my browser settings were not correct
Somehow, caching was disabled in firefox, to enable it i did the following
visit about:config (fill it into your address bar), this will give you your browser settings
in the top box, type in browser.cache
Make sure both browser.cache.memory.enable and browser.cache.disk.enable are set to true, if they are not, you simply double click it to flip it.
Close this window and try (no need to restart firefox).