It wasn't a matter of quickness but rather of interest - the combat controls were unimaginative (if serviceable) at best and turned combat into a drudge, a problem for a game with no shortage of it. I wanted to actively avoid encounters - hence, they made me not want to play. I thought the arbitrary damage scaling based on your 'stance' was pretty silly too (I think I recall my "AoE" attack stance dealing more damage per target to four X enemies than it does to one X enemy?) Plus the camera was all over the place - I remember being really annoyed with it's collision detection.
At no point did the story get me to care about what was happening - I could never connect with Geralt. And to make matters worse, I don't recall Geralt ever really connecting with other characters either - even your 'friends' in at the very beginning of the game felt like strangers or generic NPCs, not your companions. I guess you could attribute that to the memory loss, but most of the people you meet are treated with similar apathy (unless you can sleep with them, which moves the needle slightly). It was hard for me to get invested in the mission when it didn't seem like Geralt could be bothered to do so either.