It can be fixed, but it is a fair bit of work. Generally the way to do it seems to be to use a heat source (heat gun or hair dryer) to loosen the glue and then the insert can be removed, cleaned up and then glued back in place.

If you use the search function you should be able to find a number of threads on fixing inserts.