Downshifting does cause a lot of wear on the clutch. That's because the fibers of the clutch pad are mostly worn in one direction because the engine is driving the transmission. Downshifting causes the wear to be "against the grain". Rev matching probably will help reduce some of that kind of wear, but it will not eliminate it.