It is considered defamation to make a false statement about someone, to a third party, if that false statement harms the person's reputation.
Yes, you can sue your brother. And the recipient of your brother's letter can sue you. And, I suppose, your brother could sue either or both of you. Anyone can sue anyone else for pretty much any reason at all.
Whether any suit would be worth the high costs involved and the time that must be invested, however, would depend on all of the facts.