One reason not to give money: we work hard for the stuff!
An important part of this issue is to understand "Social Capital" which is, that by virtue of being born in a place that has good institutions and infrastructure, we are able to make a lot more money. (possibly accounting for at least 90% of our earnings).
So if you live in a third world country, it doesn't really matter if you are smart and hard working. It is pretty much impossible to get ahead if your whole day is spent walking to clean water, or if there is no electricity in your community, or if the government and court system is corrupt, if there are no banks to provide loans, no good schools etc. etc.
Warren Buffet said: If you stick me down in the middle of Bangladesh or Peru, you'll find out how much this talent is going to produce in the wrong kind of soil.
-The Life you can save by Peter Singer
The reason we make the money we do, has less to do with how smart or hard we work and more to do with where we were born.