I was going to say "side of your nose" but yep, topaz always feels colder, even after its been in your pocket.
Nearly all the topaz I have seen has retained some remnant of the crystal shape with at least one flat side plus an indication of the flat cleavage plane
Put my money on quartz