The File Explorer button opens a Quick Access page, which opens what it has determined Frequent Folders and Recent Files. If any of those files in that list are remote, have moved, or are otherwise not available, Windows may be spending extra time trying to find what it thinks should be there.