I keep reading about how the pain is from microtears, the basis of muscle growth. If you're not getting at least a bit stiff you may not be working out enough or hard enough. You say you lift to fatigue (the point where you physically can't do another rep), which is good - but are you doing a light weight for 20 reps or a heavy weight for 6-10? Heavier is better for muscle growth. Also can try doing more sets, if you do 3 like many people. I'm focusing on my chest, arms, and quads, and do 6-8 sets of the exercises that target them. I've noticed a lot more growth and stiffness since upping the number of sets. After a good chest workout I'm stiff for about 5 days. Your body will also adjust to what you're doing after a while, so as to do the required task with the minimum amount of energy expenditure. You have to switch up what you're doing or how you're doing it generally every few months.