Thanks to Diego Wasser for pointing out the error I made in my use of Colliders in Unity3D.  He’s 100% correct – if you have a collider attached to a GameObject that is moving you should also attach a rigid body to the same GameObject.  In my case, I was moving a parent object that held a bunch of smaller objects, each of which had a collider attached.  I went back and added a rigid body to each of those smaller objects as recommended in the post linked by Diego.  I also marked them as kinematic, as I don’t need physics (gravity etc), just collisions.

I also confirmed with Unity support that this was still the case, and was told it was even more of an issue in Unity5.  I requested that they add a line to the manual in the collider section to make sure everyone is aware of this performance buster.

For more information, check out the post on the Unity Answers forum:

http://answers.unity3d.com/questions/11324/move-gameobject-with-collider-without-rigidbody-co.html