"I would much rather have my brother in my life." Yes, his human friends were important, but he could easily see them whenever he wished with the Bifrost. That wouldn't be the case for Loki if they settled on Earth.