Just to clarify, the 4 ports on the back of your routers are a switch. Most SOHO router/switch combos cannot do both from the switch ports. If you connect like you have diagrammed it's really not much different than daisy chaining regular switches together.
As was mentioned it's probably a good idea to have some device handling DHCP so you don't have to manually configure each computer, however it's not required.
Although to be honest I'd rather run out to the store and buy an extra switch or two rather than deal with the extra headache of setting it up as you proposed. It would probably take less time and would be easier to troubleshoot.