Well they are made to measure things, so one would hope they are decently accurate within say 10%. If not, what's the point?
Yes, it IS reasonable to expect "reasonably accurate". But we also must recognize that ALL measurement systems have limits on what they can do, and how accurate they can be under various circumstances. Measuring the voltage between your two hands is NOT what a standard voltmeter is designed for, so it is outside its capabilities to be accurate.
Nevertheless, what you see is partially true. You grab the two voltmeter probe ends, one in each hand, and you get a non-zero reading but it fluctuates. First you have to recognize that the meter is reading some voltage provided by a very high-resistance source. Now, the voltmeter itself actually is measuring a CURRENT through itself across its own input impedance and using an amplifier to do it. Knowing the current and the input impedance allows the meter to present to you on its display the result in terms of voltage. But if the impedance of the source (your body and hands) also is very high, the result can be inaccurate.
Why is there ANY non-zero voltage? Several possible sources, including things like induced currents from nearby AC fields, etc. But one VERY common one is: you have just created a battery of sorts. In electrochemistry look up Galvanic Cells. If you connect two pieces of metal to an external voltmeter and them place those two metals into some conductive medium (electrolyte), you have a Galvanic Cell. The most common ones we see everyday are batteries made with two DIFFERENT metals and some thick chemical paste between them, all sealed into a case. But another version is one in which the metals are the same, but the conductive medium in contact with each of them is different, and there is still some electrical connection between the two different media. Different how? Could be different types of electrolytes. But also, you get the same result if the electrolyte is fundamentally the same all over BUT the solution's concentration of certain dissolved components is a little different in the vicinity of the two metals. This is called a "Concentration Cell", and it is the basic mechanism by which metals can corrode - like when your car body starts rusting out in spots because of "road salt". Your car's body is mostly one type of metal, and the electrolyte is just wet salty deposits, but the concentration of the salt varies from one spot to another on the car. So you have a Concentration Cell and the Galvanic Cell system causes one area to oxidize the metal to soluble ions that leave - that's corrosion.
Now, back to you holding the two voltmeter probes. The moisture on your skin is an electrolyte - it contains a few slightly conductive dissolved materials. AND those materials are a little different between your two hands. So YOU are the battery, and the voltmeter is doing its best to show you the output. Now, the concentration of dissolved materials in each hand may vary a bit over time. ALSO, the resistance of the contact between your skin and the probe will vary over time as your hands flex. For both those reasons the voltmeter will actually "see" a varying voltage, and it will be very small since the source is a Concentration Cell driven by very small differences in the electrolytes (the skin moisture) at each metal probe.