if you are using apache with mod-php the biggest positive impact could be changing to apache + phpfpm or nginx + pgpfpm
I am using phpfpm, I ran into problems in the beginning thankfully which caused me to switch to it instead and made a big improvement.