Couldn't agree with you more. Been there, done that.
To answer your question, No. You need not sell it just yet.
CD5004 has a decent headphone output for its price. Borrow a good quality headphone from someone and see how different it sounds directly out of the CDP than compared to the rest of your chain. I see that you have started noticing the differences with your amp auditions lately. So start saving up for a dedicated stereo amp. Not sure about your particular avr but if it comes with some kind of direct mode, experiment with that for the time being. Also if open to it, get in touch with our fellow b'lore member hifipal4all for an interim solution via the used route.
A little snobbish, but let me just say it out loud anyways. Most of us started off with an entry level hifi system. Our expectations, however were of larger than life euphoric audiophile sonic nirvana - even when our starter systems were often basic simple baby steps to this fascinating (and expensive) hobby. Over time, we learned to differentiate and hence appreciate various levels of purity in music reproduction. This unfortunately comes at a price. While we can all debate over the merits of audiophile diminishing returns or cost vs. value matrix, good components do cost substantially more and often for good reasons as well. However, there is a wealth of knowledge out there for anyone in this hobby to tweak their system to their individual preferences, in any budget. So take your time and experiment. Listen more, learn more.
Cheers.