They're supposed to. Someone decided that terror was an essential component of summoning. The idea is that you're confronting your own fear of death every time you do.

And the Evokers are probably the least of it. I'm going to guess you're not aware of the experiments they did with artificial Persona users?