I haven't read the Black Swan book either, but the argument was pretty central in his book Fooled by Randomness, which I recommend strongly.
A lot of Taleb’s thinking seems to stem from his background as an options trader. For example, he vigorously bashes Jim Rogers for arguing that buying options is a losing proposition since about 80% of options turn out useless. Taleb, of course, notes that this doesn’t make buying options a bad deal – if only the rest results a sufficiently high return.
So his argument, as I understood it, is simply that if an event is deemed unlikely, and you bet on it occurring, then your return if it does occur will be high. This is the ‘impact’.
In relation to 9/11, the (psychological) impact was large simply because it ex ante seemed unlikely, which led us to be psychologically unprepared when it did happen.
The term ‘the black swan theory’ was probably chosen because it’s a catchy title. But, as you suggest, the term is somewhat misleading since we do not really associate the discoveries of black swans with great cataclysmic effects. The way to understand this, I believe, is that the impact of their discovery was large only to those who had made a bet – in some form – on their existence, or non-existence.