Redline is mainly limited by valvetrain ability, namely the ability of the valves to slam shut fast enough that the engine can retain compression. Also, a large bore/short strok engine will have a higher redline because the pistons have less distance to travel with each time the crankshaft revolves, which allows the mean piston speed to be lower at a given speed. With long stroke comes a higher mean piston speed, and for a higher mean piston speed, the connecting rods/pistons need to be more solidly attached. Basically higher engine speeds require more durable componants and these componants are costly. Also, the majority of people need engines with power in lower areas of the RPM band because of the types of driving normally done by a consumer. Revving an engine to the sky may be great for the track, but it can become tiring while driving around the city.
Zenmervolt
EDIT: I don't know for sure, but I suspect that the reason for the lower red line in a Diesel engine is because of the much higher compression ratio of a Diesel engine. A Diesel must work much harder to compress the air inside the cylinder in order to heat it up enough for it to spontaneously ignite the atomised fuel that is injected. Also, most Deisels are designed for maximum low end torque, and in such designs the stroke of the engine is typically very long, and a long stroke which also lowers redline in general. Basically Diesel engines have different design parameters than normal engines and as such a comparison between the two is not really fair.