I may be wrong, but I believe that Outlook is actually a part of MS Office, while it is Outlook Express that is a part of Windows.
I've never tried, and don't know how, to remove a single program from MS Office. I don't use Outlook so I deleted it from the desktop only, which was good enough for me.
🙂