Loki stayed where he was for a solid count of twenty before going back inside as well. Long enough to believably not just be following his brother, long enough for Thor to storm entirely out of the room if that was his intent, short enough not to be particularly pointed otherwise.