I thought I would need to use PolyFilla just to level the pf paint. If i'm reading this right I can just use primer

Try using "spot" or "knifing" putty, available from any Sprints or Supacheap auto store.
It's basically really thick auto primer (it comes in a tin sort of like car polish comes in)  and great for filling small chips and scratches.