I really wasn't a fan of Samsung's constant attempts to make plastic look like something it wasn't... faux chrome, of course, but also faux leather and faux brushed metal. Then there's the GS4, where Samsung dropped the pretence and made a phone with all the cheap, featureless charm of a low-end washing machine. The Note 4 (and Galaxy Alpha right before it) was a sheepish admission that Apple was right, and that things like real metal matter in the perceived feel of a product.
As for the Note 5? Part of the problem was that it came in the GS6 era, when Samsung was both struggling sales-wise and created something of a shock with its design (and I don't just mean the absence of a microSD slot). There were probably a lot of people happy with the Note 3/4 at the time, and people were down on Samsung in general. That and people who did like the new Samsung aesthetic could spring for the S6 Edge Plus.