Bigger factor is where and what type of boating you are doing. Local park-and-playspot, where your car (and other warmth) is close-at-hand? Less concern. Easy local run, roadside virtually it's whole length? Again, less concern. Run has virtually no likelihood of you or anyone else spending extended time needing a rescue (virtually never any deadfall or snags, no undercuts, no sieves, etc)? Less concern. However, if you are doing runs which get away from civilization a bit more, which have known or recurring/random likelihood for deadfall, if you are pushing your skill/comfort level up a notch (boating unknown runs or runs at or above your current skill level, or if you are with anyone else who may be at or above their skill level!), . . ., any number of things which would mean that you NEED every extra margin of safety that you can allow yourself . . . then you really need to consider the drysuit.
One of the toughest things is to really be judicious at all times . . . to do critical self-awareness, to make wise decisions about your preparedness for various runs and various conditions. There are many factors which need to be considered before deciding if you are ready for a given run on a given day, and one of them is whether you have the right gear for the conditions.