Loki actually shook his head before remembering invisibility. Fortunately, because invisibility, no one saw it. Then he whispered, "No. I know another way in. It's this way." He started to move and tugged Thor's hand so he knew which way this way was.

Technically it was one of several servants' entrances, handy to the kitchens. When food was being delivered in the mornings or at night when leftovers were being handed out, it was busy. The rest of the time, the door was open but almost no one used it. Very useful for someone who wanted to come and go, and maybe get an apple at the same time.