The CPU is not the cause of slowdown in GG, at least not with 2 items as stated above. I would suspect onboard video to get that vast a slowdown with a low entity count. The CPU will hold back GG when the scene gets more complicated and AI is introduced, but with a bare map and 2 items, no. It is without doubt a video display issue.
So is it a laptop? If so, is it using your laptops best GPU? Many default to the extremely basic GPU and not the Nvidia or AMD one. You should be able to force GG to use the faster GPU, there are many threads pertaining to this on these very forums and generally it is the GPU at fault (using the wrong one).
It's fairly well know that Nvidia generally performs better than AMD with GG. Still you should get better performance than what you suggest you are getting at present. My CPU only runs a little faster (2.4gig) but it runs basic GG scenes no problem. CPU's of course no longer can be judged by mere clock speed and haven't for many years, there is a lot more to a CPU than that. An AMD CPU at the same clockspeed as an Intel one will always be far slower for anything mathematically demanding. This has always been the case and unless things change dramatically will always be the case. A Xeon processor is unusual as they used to be server cpu's and therefore fairly capable bits of kit. You may have a refurbished server base or such, in which case the graphical side of things would most certainly be somewhat lacking.
As said above a screenshot of the scene would help us judge it better, but in my opinion, the GPU is your main issue here. There simply isn't enough else going on (from the info you have given) for it to be the CPU. As I said my CPU is only 2.4 gig and I get at least 150 fps on a blank map with a couple of entities on it, possibly closer to 100 on full settings. It is a common misconception that the CPU is hugely important with games. It isn't. The video card is king and someone with an all singing and dancing I7 with an old video card will run games a lot worse than say my CPU and a fast one. Obviously you will get some improvements with an I7 as well, but not to the extent you will see instantly with a video card.
SPECS: Q6600 CPU. Nvidia 660GTX. 8 Gig Memory. Win 7.