How can I get out of this circle?

Decide that you are going to refrain from for a period of time...say 6 months. When the subject of comes up, tell them that you are not having for six months. If the guy is still around for six months then you know he's serious.