How to get off this black stuff around tail lights?

I would start by attempting to remove the taillights. They are not original to the trailer, and it looks like someone cut holes to fit them into and then tried to seal them with something that came in a tube at the time it was done. If you can remove without bending up the edges of the holes, then you should be able to chip the sealant away more easily. If it can't be chipped away without damaging the skin, then I would try some different solvents to soften it up, starting with something like a paint thinner or denatured alcohol. If they do not work then move to something a little more potent. I have heard of others using gasoline, but I highly recommend something less volatile like diesel fuel. Protect your hands whatever you use, if you need to get that far. If it were me, I would consider a replacement for the lights. The originals are a bit difficult to find and may not cover the holes. There are a couple options to use a recessed light, or re-use the ones that are there if they clean up to your satisfaction.
By the way, they would have to be removed anyway if you are removing the rear wall panel to get at the framing underneath. This is just one step in the many steps.