I know, I'm too skinny -E! Anchor Guiliana Rancic finally speaks out

For months, she's stayed mum on the subject of her body, but now,Giuliana Rancic, who says the firestorm of attention has "gotten out of hand," is finally speaking out. 
The E! News anchor's slim appearance at the Jan. 11 Golden Globes sent the Internet abuzz with rumors of eating disorders – and worse. 
"Some people were saying, 'The cancer is probably back,' " Rancic, who was diagnosed with breast cancer in 2011 and underwent a double mastectomy that year, tells PEOPLE. "And they were accusing me of every eating disorder. I thought to myself, 'God, if someone really thought I had an eating disorder, what a horrible way to approach it.' " 
Thankfully, Rancic, 40, is healthy, though she says the cancer-suppressing medication she has been taking since 2012 has had an effect on her body. 
"I started noticing that I was eating a lot, but not gaining weight at all," Rancic says. "I was concerned." 
Her oncologist, Dr. Dev Paul of Rocky Mountain Cancer Centers in Denver, Colorado, informed her that the drug, which can alter metabolism, could cause weight loss. 
"It's really hurtful," says Rancic of the attention, adding that she had hoped criticism about her appearance would quiet down on its own. "I'm sorry that some people think I'm disgustingly skinny, as they put it, but there's nothing I can do. I'm lucky that I even have the type of cancer that reacts to the medicine." 
When it comes to her appetite, "I eat more than any of my friends," she says. "I eat a very robust, healthy, balanced diet and dessert almost every night. I'm not hiding from anything." 
Though the mom of son Duke, 2½ (with husband Bill), admits that the physical change has been difficult to bear. 
"I look in the mirror and it's hard for me," she admits. "I am really thin. I want to look fit and beautiful and sexy, and I can't."