I did that mistake long ago, caring about story pointing and velocity. It was a new concept to learn and embrace, a new weapon to my increasing toolset. But after years you realise that who cares? Why should a team spend time getting more accurate at story pointing? Worrying about how many points have you delivered on a sprint?
We always have limited time to spend, especially when we care about things outside work. If you are leading a team, you could focus on following the Scrum methodology like some holy grail, turning your team into an efficient sausage factory, worrying about the impact on your precious velocity that holidays or new team members will have. Or you could spend your time ensuring your team is trusted, listened to, has meaningful work to do, they help each other and nobody is blocked.
Which approach is more likely to produce better work? I know which one I prefer, hence I don't care about velocity.