Not a bad list of hypotheses (and you can add 9/11, SARS, etc.). But there could also be other factors.
If lots of Americans were coming here as tourists in previous years, why were they coming?
Part of it may have been because they wanted a big city experience but found their own cities -- like New York or Chicago -- a bit too scary. Well, crime rates in big American cities have fallen dramatically. Is it a coincidence that New York tourism is at record levels?
I'm not at all trying to suggest that crime in Toronto is our tourism problem. Rather, I'm saying that in a competitive field of destinations some of the alternatives may have improved themselves faster than we can keep up.
Frankly, I always find it hard to come up with good lists of attractions for tourists coming here. I feel that the best parts of Toronto are things that have to be lived, and can't be experienced easily in just a weekend. Then again, when I visit other cities I'm just there to get a feel for the place rather than hit a list of attractions. In any case, Toronto Unlimited sure isn't going to get people here.