Look at the vest. If it is faded, with the straps and fabric starting to fray, then you have a lot of UV damage. Tug at the faded parts. If they start tearing easily, or the seams let go easily (the thread is usually synthetic and also subject to UV degradation) it's definitely time to replace it.
These two tests cover the two most common ways the vests degrade. One is the slow decomposition of the foam (which is invisible to you, being encased in fabric) causing loss of buoyancy, and the other is ripping/tearing and thus releasing the foam to float away (whilst you sink) during a "carnage moment."