Consider whether or not you care that you'd be called a social outcast anyway, considering you're not being cast out, but choosing NOT to go.

Secondly, consider all options. Do you have anything better to do? Do you have reason to go? Do you have reason NOT to go? etc.