Speed is now a factor in you google ranking. In my tests I found these speed issues are not much to do with the web host but with the complexity of the design of a page. Some aspects of that complexity are inherited from the CMS (WordPress, Drupal, ExpressionEngine etc) and may take a lot of work to remedy.
Speeding up websites is important — not just to site owners, but to all Internet users. Faster sites create happy users and we’ve seen in our internal studies that when a site responds slowly, visitors spend less time there. … Like us, our users place a lot of value in speed — that’s why we’ve decided to take site speed into account in our search rankings. We use a variety of sources to determine the speed of a site relative to other sites.
via Official Google Webmaster Central Blog: Using site speed in web search ranking.