There is some misconception here about ps3 and mkvs. First, mkv is a container like avi and mpg. Ps3 does not understand a mkv container due to piracy issue, but it plays the video/audio contents within most HD mkv. Pretty sure your mkv file contains h.264 video with either AAC or DTS audio muxed together. There is a setting in pms to use tsmuxer to demux mkv files and stream your contents in mpeg transport. In short what this does is the pms server will open the mkv file and demux the audio/video, then put them in mpg container to stream to your ps3 and will play in its original native format. thats what mkv2vob does, but on the fly instead, which usually very little cpu. I been using pms to stream all my mkvs fine for years and even got a receiver to decode dts audio. my file server is a old dell p4 1ghz machine that only takes a second or two to start streaming 1080p mkvs and then pretty much less then 5% cpu thur out the movie.
as for the person asking about wifi and getting a apple tv, its the same issue. look at the numbers, most 1080p movies are 8+gb, no way wifi can transfer that size within a typical 2 hour period. ps3 can do 720p via wifi easily and should be no shutter. you probably not demuxing the mkvs and instead transcoding/encoding the media to a high bitrate mpeg transport. just gotta fix your settings to play the contents in its original format and you should have no problems streaming 720p. your better off wiring your ps3 and you can watch 1080p mkvs in its original format natively no problem. look at your servers cpu usage. if its demuxing correctly, your usage should just spike a little bit then idle out. if you see your cpu usage very high for longer then a few seconds, then your trancoding/encoding, which is bad