You might consider using Paul Heckbert's zoom code if you can still find it. He has nearly all of the image resampling algorithms implemented in a C callable fashion. The trick is getting your data into a format that the algorithms can work on. It's a bit of code, but it's not entirely out of the question to convert his code to a template style so that you can specialize on the pixel type.

In my experience the code performs reasonably well. I'm sure if you really need to extra performance you can optimize it.

Looks like you can still download it here:
http://www.xmission.com/~legalize/zoom.html