Not scary, because there's nothing abrasive in banana flesh. However, I have no idea whether it would be effective. I have heard bananas recommended for so many unlikely things over the years that I would certainly give it a try. So long as all stickiness is washed off, I can't see that it would harm.
I wonder if this means we could use bananas as soap!