For me, I consistently worry about bears and moose. In fact, just this week I ran into a cow and calf moose while fishing in a willow swamp...and I mean really close. I have been fishing rivers and have been really focused on what I was doing, and then looked up to see a moose barreling straight at me.

That can make your adrenaline spike.
Grizzly bears are the other real fear, but honestly I think that they are less of a problem. As long as you make a point to be loud and noisy in known bear holes...they will move away. I have yet to see one that didn't run...I hope that this continues, I do carry a .44 or bear spray, sometimes both. We still frequently run into tracks.