That's what Tinyumbrella's for, I've downgraded iPhones to unsigned versions, so no, it's not IMPOSSIBLE. All you need is to have made an SHSH Blob from your phone + Tinyumbrella and you're good to go.
http://www.iphonehacks.com/download-tinyumbrella
"Apple stops signing the firmware files (by not providing SHSH blobs) for older iOS versions to prevent users from downgrading. Tools like TinyUmbrella have allowed you to save the SHSH blobs, so you can use it to create (stitch) a pre-signed IPSW file, which could then be used to restore your device with an iOS version that Apple was no longer signing.
Apple has closed the loophole that allowed users to use the SHSH blobs in newer devices (iPhone 4S and later) and also in iOS 7 or later. So you cannot restore to 6.x using SHSH blobs except on iPhone 4, iPhone 3GS, and iPod touch 4G (and iPad 2 to iOS 5.x if you have iOS 4.x and iOS 5.x SHS blobs)."