There's a program called
Favorg that will allow you to permanently save them. I've been using it for years with all versions of Windows. I make my own icons for sites that don't employ favicons with their own graphics and logos; and I substitute other favicons with cleaner looking better ones for sites that do (that use sh*tty looking ones). These favicons can be used to customize your folders as well. Using favicons makes it very easy to navigate your favorites visually and find the bookmark you're looking for immediately, which is especially useful, if you have thousands of bookmarks to look through. It's not just "eye candy" as it serves this very useful purpose.
When websites started using favicons, it allowed them to know that you've bookmarked them. This was a big privacy issue with some people. There is no harm that can be inflicted from this knowledge, especially with today's firewall protection and security measures. After the initial bookmarking, I backup my favicon data and reuse it upon reinstalling the OS.
You can manually rename and apply the favicons yourself without any program; however favorg is very handy and simple to use. I recommend highly.
I have a sweet tooth for eye candy!