Heart rate

My heart beat is always on higher side. I started noticing it few years back when it was approximately 95/min. Now a days it remains between 100-110/min, more closer to 110/min. When I consulted doctors earlier, I was told nothing to worry about as long as I don’t see any other symptoms. Now when it is over 100 all the time (except when I’m sleeping it drops to around 85/min), I’m bit worried. I use fitness tracker and check it regularly using it. Is it something to worry about? Is it related to lifestyle etc.? My work makes me sit whole day in front of laptop with minimal mobility. I’m not into sports or workout. I do take evening walks though.