In either case, ugly or beautiful, people derive a significant part of their identity, be it negative or positive, from their body. To be more precise, they derive their identity from the I-thought that they erroneously attach to the mental image or concept of their body.
Equating the physical sense-perceived body that is destined to grow old, wither and die with 'I' always leads to suffering sooner or later.