Tips for Esper
Esper builds are notorious for becoming extra-levels of zany bonkers in terms of raw power. Psychic Hunters are how the game balances against that.
There are several select methods of overcoming/countering that – most of them are down to personal discovery. However, here is a quick tip that should help ease up the attacks (at least early on).
Roughly half of all Psychic Hunters will be Seekers of the Sightless Way. Increase your faction relationship with them until you are favored (250+) and even aggressive members will no longer attack you.
The OTHER half of Psychic Hunters do not follow faction alignment rules and are always hostile.