Use a pencil eraser to clean the connectors - Never use WD40
Would a fine sandpaper (eg emery board) help with this without damaging the connectors?
Yes it would work - but on some connectors, you are removing a protective coating on the pins to prevent corrosion. I'm not EXACTLY sure of it's properties, but it is easier to clean with a simple erasor.
