Not all these ingredients behave the same way. Some anionics are harsh when left on skin (e.g, SLS) some cationics (typically the short chain ones) are harsh when left on skin (e.g., cetrimonium chloride).
In general, non-ionic surfactants are less irritating.
And, yes, as with almost everything else, concentration matters. Even cetrimonium chloride is allowed to be used in leave on products at low levels (up to 0.1% or 0.2%, I believe.)