Unfortunately reversing the clipping polygon (so everything outside of it is rendered but inside is not) is technically difficult. We would like to support image based clipping, which would allow for smooth edges, but we won't be able to get to that in the short term.
If we could support inverting the clip polygon, using clipping for holes is not generally a good idea due to the performance impact. Clipping is very expensive CPU-wise and should be used judiciously with the Spine Runtimes. Performance is not a concern if you are doing image or video export.
For holes, the easiest solution is to use an image with transparent pixels. If you have an image in the background that is larger than your top image, you could use clipping with 3 or 4 vertices to prevent it from showing. If your hole is very large, you can use two meshes around the hole.
We are in the midst of updating our Spine User Guide, but here are a few relevant sections:
Clipping attachment performance:
Clipping in the Spine Runtimes is implemented using the CPU can be a very expensive operation, especially when using mesh attachments with many vertices. Always check the performance of your animations that use clipping on your target platforms.
To minimize CPU usage for clipping, reduce both the number of vertices in the clipping attachment and the total number of vertices in all the clipped region and mesh attachments. Set the start and end slots to clip the fewest number of attachments and edit those attachments to reduce the number of vertices in each one as much as possible. The size of the clipping polygon does not affect performance.
Use a clipping attachment with just 3 vertices whenever possible, which is the minimum number of vertices allowed.
Clipping attachment performance is only a concern when using the Spine Runtimes, not when exporting images or video.
Meshes that need holes:
Meshes can be concave but cannot have holes. If a hole is needed, usually it is sufficient to use transparent pixels in the mesh's image. For example, the image for a head mesh can use transparent pixels for the eye holes.
If the hole needed is large, it may be undesirable to draw a large number of transparent pixels, which can negatively affect the fill rate. Also, a large image with a lot of blank space can take up a lot of room in the texture atlas. In that case two meshes can be used, one for each half of the image around the hole.
Using two meshes is done most easily by first making one mesh that encompasses the whole image and has vertices around the inside of the hole. Next, duplicate the mesh and modify both meshes by deleting vertices until each makes up one half around the hole.
