All static physics-on entities will be included in the AI path finding system so if neither the player, AI characters nor dynamic physics entities can come into contact with them then turn off physics.
Animated entities are way more of a performance issue though, any background entities which happen to be animated will be bad for performance.
Been there, done that, got all the T-Shirts!
