It had some pretty ingenious heuristics to detect the entire source image, see exactly what pixels had changed and only update the relevant regions. Of course this was originally written years ago so this kind of hack is not necessary, and I have disabled it. On my laptop, everything seems smooth as long as it is compiled with optimization. Without, it is quite jerky (the compiler must be doing something right). However I am considering adding optional access to the old source image as part of the scaler api so all those hacks would not be necessary.
So here are some comparisons. Click to see larger versions.
EDIT: I have clarified some of the optimizations I disabled in another post.