sounds like you should stop buying it if it hurts you and switch to reusable containers. I do agree though, that the way it is implemented isn't great. ideally every store that collects a deposit fee would also be a place to give back the item.
In Denmark, small shops must refund at least the bottles they sell. Normal or large shops must refund all deposit bottles and cans.
Most people take the empties to a supermarket. People drinking in a public place often leave them for bottle collectors (often homeless etc) to gather.