Friction is involved, I believe, in an interesting way, and there is no doubt the shaking action does work on the particles being sorted.
First of all, I see friction as the reason that the initial array of particles does NOT sort itself through the sieve(s). If it weren't for friction, they would all slide over and around each other and fall through the sieves without any shaking.
What does shaking do? It adds to the particles an Activation Energy - that is, a small amount of additional energy that allows the particles to "escape" their initial stable state and move around to do something else. The particles certainly gain Enthalpy because they begin to move - they have acquired translation energy. But they also have acquired Entropy - more disorder - because they no longer are resting securely against each other in a well-defined arrangement, but are now separated from each other slightly by a thin film of some fluid like air or a carrier solvent. The minute separations reduce the effect of friction so that the main forces inhibiting movement of particles past each other becomes short-lived impacts, and the new translational (and rotational) energy of the particles allows them to move around each other. The result is the the particles at the bottom of the array and in contact with the sieve wires can move to the holes and re-orient themselves as necessary to fit through the holes (provided the holes are larger that the particle).
Now, what happens to all that added Activation Energy? Some of it is dissipated into the surroundings as sound because particle impacts with each other and with the sieves cause vibrations. Some of it is retained in the particles as minutely increased temperature - heat, really - as the atoms and/or molecules of the particles increase their own vibrational and rotational energy states. I agree with others that, at the end, the overall system would appear to have reduced disorder - reduced Entropy - but I think it also has increased Enthalpy. However, as I said, some of that Enthalpy has been lost to the surroundings, too.