Dual Ethernet ports are necessary for various network/server applications where the system in question is going to be used as an Internet proxy server, a network firewall, or similar type of application. Having dual Gigabit Ethernet controllers allows for easy deployment for such applications.
In other situations, workstations can still benefit from multiple Ethernet controllers. Many Ethernet switches will allow for the aggregation of connections. Hooking two Ethernet controllers from one PC or workstation will double its effective network bandwidth over the LAN.