Hi Stuffmaker, welcome to the community. You should be able to remove the sleeves by using a flat tipped screwdriver. You will want to insert the tip of the screwdriver between the top of the caster and underneath the sleeve bottom. Now turn the screwdriver like you are tightening a screw. This should loosen the sleeve from the stem of the caster.
You do not want to hammer the entire caster & sleeve as one piece. You need to be able to insert the sleeve and then insert the caster stem into the sleeve to get a proper fit.