Anisoft

What is the correct way create a sprite via mix and match, and pack normals with the character?

I have a character creator which uses mix and match on different unique sprite parts. I got all the mix and match working but what I don't fully understand how to get normal textures to work too.

I was thinking I'd need to have a way to assign both the diffuse and the normal sprite.

I'm trying to keep all my sprites as separate pieces, each with their own normal. Then assign or pack at runtime.

Edit* Is the method involved still the same as mentioned here?

http://esotericsoftware.com/forum/Runtime-Repacking-with-Normal-Map-12701
Anisoft
  • Posts: 61

Harald

Please do not implement anything yourself as described in the above posting!

The proper and easy way is described in the official spine-unity documentation pages here:
spine-unity Runtime Documentation: Combining Skins (see subsection Advanced - Runtime Repacking with Normalmaps below)
User avatar
Harald

Harri
  • Posts: 1490

Anisoft

What has me confused is that there isn't an input for the actual normalmap sprite. So I'm unsure how it creates the normalmap skin without a reference to my actual normal sprite.

My game generates all the skins at runtime via combining skin parts into skins.



So I had thought I'd need to make a sprite for the normal-map and put it in the part.
Anisoft
  • Posts: 61

Harald

Sorry for the late reply.

How are you combining the single Sprites in your PartSO script? Are you using Attachment.GetRemappedClone()? Please provide more context regarding the code used in your PartSO component.

In general, if you have created a working single Attachment that has a Material with a normalmap assigned, then it will use this assigned normalmap texture when packing a new texture-set via Skin.GetRepackedSkin(). So if the Material has the normalmap assigned beforehand, it will work after packing (when following the documentation section mentioned above and passing the respective optional parameters at GetRepackedSkin()).
User avatar
Harald

Harri
  • Posts: 1490

Anisoft

I am too sorry for the late reply.

Currently I combine everything inside my CharacterVisual script.

So this is my CharacterVisuals script Im using.
using System;
using Anisoft.Character;
using Anisoft.Game.Entity.Skin;
using Spine;
using Spine.Unity;
using Spine.Unity.AttachmentTools;
using UnityEngine;

namespace Anisoft.Behaviors.Character {
[RequireComponent(typeof(CharacterB))]
public class CharacterVisualsB: MonoBehaviour {
private SkeletonAnimation _skeletonAnimation;
private CharacterB CharacterRef { get; set; } = null;
public GameObject visuals;
public Material sourceMaterial;
public Material blankMaterial;

public Skin[] skinParts;
public Skin customSkin;
[Header("Do not assign")] public Texture2D runtimeAtlas;
public Material runtimeMaterial;

public Texture2D[] additionalOutputTextures = null;
public int[] additionalTexturePropertyIDsToCopy = new int[] { Shader.PropertyToID("_BumpMap") };
private bool _refreshTexture = true;


public void Setup(CharacterB character) {
skinParts = new Skin[Enum.GetValues(typeof(SkinType)).Length];

CharacterRef = character;
CharacterRef.UpdateVisuals += RefreshVisuals;
_skeletonAnimation = visuals.GetComponent<SkeletonAnimation>();
}

private void RefreshVisuals() {
RefreshSkin();
RefreshSkinTexture();
}

private void RefreshSkin() {
var skinPartsSlots= CharacterRef.DataActor.dataSkinPartSlots;
var skeleton = _skeletonAnimation.Skeleton;
var templateSkin = skeleton.Data.FindSkin("base").GetClone();


customSkin = customSkin ?? new Skin("Custom Skin");
customSkin.AddAttachments(templateSkin);

for (var index = 0; index < skinPartsSlots.Length; index++) {
var skinPart = skinPartsSlots[index];
if (skinPart == null || skinPart.Name == (skinParts[index]?.Name)) continue;


var skin = new Skin(skinPart.Name);
foreach (var subParts in skinPart.subParts) {
var attachKey = subParts.attachKey;
var attachSlot = subParts.attachSlot;
var slotIndex = skeleton.FindSlotIndex(attachSlot);
var attachSprite = subParts.attachSprite;

var templateAttachment = templateSkin.GetAttachment(slotIndex, attachKey);
var newAttachment = templateAttachment.GetRemappedClone(attachSprite, blankMaterial,
false, false, true);
skin.SetAttachment(slotIndex, attachKey, newAttachment);
}
customSkin.AddAttachments(skin);
skinParts[index] = skin;
_refreshTexture = true;

}
}
private void RefreshSkinTexture() {
if (!_refreshTexture) return;

var skeleton = _skeletonAnimation.Skeleton;

customSkin = customSkin.GetRepackedSkin("custom skin", sourceMaterial,
out runtimeMaterial, out runtimeAtlas,2048,
additionalTexturePropertyIDsToCopy : additionalTexturePropertyIDsToCopy, additionalOutputTextures : additionalOutputTextures);
skeleton.SetSkin(customSkin);
_refreshTexture = false;
}
}


}
Which looks like the following in the inspector



What I'm trying to do is build a custom skin + normals from a base skin (using a texture atlas and normal texture atlas)
and then on top apply separate attachments that each have their own texture/normals


One issue I found is that if I assign a material that doesn't have a normal, I get an error about missing textures.
//inspector

//base texture settings

//error


So i tested out just using the same material with the same normals for each (for testing)

And it... sorta worked but resulted in an incorrect normalmap.


I will note that it wouldn't allow me to combine the textures unless they matched formats. So I changed it to the only uncompressed format I could RGBA 32 bit

And it seems like when I check the channels... the GBA have texture but the R doesn't... so it seems like it copied the colors to the wrong channels based on what i did?


So its pretty evident I don't know what I'm doing just yet lol. I'm try to get some sleep and tackle it more later tmw. I hope I've provided some info as to what I'm trying to pull off here.

---

I tested not combining the skin and it appears the seperate attachment with a placeholder normal worked fine.|

Separate attachments


Combined skin attachments



The combined skin doesn't apply the normal correctly

Looking at the combined texture it seems the rgba aren't correct for me. Its placing the channels of the normal texture in gba instead of rgb.



The added normal is the little square of noise in the middle right.
Anisoft
  • Posts: 61

Harald

Thanks very much for the detailled writeup!
Anisoft wrote:One issue I found is that if I assign a material that doesn't have a normal, I get an error about missing textures.
Unfortunately packing would not work consistently if regions (where textures are null) would be skipped. So it is a requirement that all repacked materials have the all used maps assigned and not be null.
Anisoft wrote:Looking at the combined texture it seems the rgba aren't correct for me. Its placing the channels of the normal texture in gba instead of rgb.
You are right about the repacked normalmap looking strange in terms of RGBA order.
I would assume that the content is not normal xyz input data, but already the prepared reordered data, e.g. _y_x as is commonly used for e.g. DXT5_NM compression and other later compressed formats. Here the Y and A channels are filled with data, as they always receive the highest precision when packed using any texture compression format (since our eye is more sensitive to green than to blue). The third channel (Z, upward) is left out and deduced at runtime in the shader for greater precision. Judging from the look of the single channels data at my local test scene, this is exactly what happens when I press play:

normalmap_repacked.png


Strangely your output image seems slightly different in channel order, with up and right channels seeming flipped compared to mine.

When I test copying from a source normalmap to a repacked texture, the source format shows RGBA32 in the debugger (with Windows active as target platform). For testing, I have implemented functionality to specify RGB24 at the normalmap, which however fails at the copy operation with error message of incompatible src and dst formats (again listing src as RGBA).

What puzzles me, is that you said that the repacked normal map does not apply correctly and your channels seems flipped in X/Y order - it behaves correctly at my test setup, using the Stretchyman assets. Could you perhaps please send us a minimal reproduction Unity project to contact@esotericsoftware.com, then I can have a look at what's going wrong.
You do not have the required permissions to view the files attached to this post.
User avatar
Harald

Harri
  • Posts: 1490

Anisoft

I sent an email with the file.

I do caution it contains a nsfw base character.

It should be a very small and easy to open file.
Anisoft
  • Posts: 61

Harald

Thanks for the reproduction package. I will get back to you as soon as I've figured out what's going wrong.
User avatar
Harald

Harri
  • Posts: 1490

Anisoft

Okay thank you! I wonder if it has to do with the default material or something. Since if you notice its generating normalmaps automatically using the base material it seems, then it uses the newly one I added.

I actually liked that as it allows me to easily use a base skin/template, but maybe its the reason things are acting up?
Anisoft
  • Posts: 61

Harald

The reason why the hair normal map ended up incorrect is because of different size - the diffuse sprite image and normal map needs to match in size, otherwise different regions are copied upon the repack operation. So be sure to use a normalmap of the same size (I have sent you one as attachment), and set the texture import settings in Unity under Advanced - Non-power of 2 to None instead of To nearest to keep this size.

After this modification, the scene looks correct after repacking, including the hair which was looking incorrect before due to only having copied a smaller part of the too-large normalmap.

What I don't quite understand is why the repacked normal map channel order is different in your project (with your project that is also the case at my PC) as compared to the project I tested repacking normalmaps of Stretchyman in the past. Also the target platforms are the same, which was my first suspect. Anyway, as the lit result scele looks correctly oriented, I would declare this issue as resolved. Please let me know if it looks correct from your view now as well.
User avatar
Harald

Harri
  • Posts: 1490

Anisoft

Thanks so much for looking into it! Unfortunately after following what you said I'm still getting wrong normals I feel.

left is the original, middle is part packed into 1 texture, right is part but seperate materials


The middle packed one is off compared to the others.
Anisoft
  • Posts: 61

Harald

Now I found your problem. Your input normalmap atlas is incorrect:
wrong_input_normalmap.PNG


Here the head and hair region has green facing left and red facing up, looking as if the normalmap has just been copied and rotated 90 degrees. As a result, the face and the initial hair behave incorrectly.
How did you create these incorrect normalmap regions? In case you manually copied and rotated the regions in an image editor, please always make sure to flip red and green channels accordingly when rotating.
You do not have the required permissions to view the files attached to this post.
User avatar
Harald

Harri
  • Posts: 1490

Anisoft

Heya, thanks for pointing that out. I had the sprites set to rotate in texture packer.



However disabling that doesn't change the issue of the normals looking off still.


I noticed that my shader material settings didn't match (so will need to make sure they are copied properly in future automatically)

However the issue is that even after adjusting the material props to match in post... there still isn't parody

Here is an example with the diffuse removes and just the normals

left: original, middle is parts combine with texture repack, right is parts combine multi material



It looks like the g and b are like identical maps... and dont line up properly with how the r (horzontal) and the b( vertical) should be

The depth map (a in the repack) doesn't match up properly at all with how the depth map should be in the source

I will stress that these all share material settings that are exactly identical now. left and right share exact parody but the repack normals don't act the same.

---

I appoligize for letting this go on for so long, I just wanna know why its not resulting in the same results.

I also noticed that the channels in the normal map repack doesn't seem correct. the g and the b look like they are the same map...

Here take a look at the default base map



vs

repack map (with no added textures, just a repack)

Anisoft
  • Posts: 61

Harald

The depth map (a in the repack) doesn't match up properly at all with how the depth map should be in the source
The third channel in your input texture is not a depth map, it is the Z (where Z = direction of the vertex normal) component of the normal. This is redundant information and does not need to be stored. We do not manually copy channel content, it is left to Unity's implementation of Texture2D.PackTextures. I already answered why the repacked normalmap channel content looks like it does:
I would assume that the content is not normal xyz input data, but already the prepared reordered data, e.g. _y_x as is commonly used for e.g. DXT5_NM compression and other later compressed formats. Here the Y and A channels are filled with data, as they always receive the highest precision when packed using any texture compression format (since our eye is more sensitive to green than to blue). The third channel (Z, upward) is left out and deduced at runtime in the shader for greater precision.
It looks like the g and b are like identical maps... and dont line up properly with how the r (horzontal) and the b( vertical) should be
As described above, you should be looking at the g and a channels.
What do you mean by "dont line up properly"? I see no misalignment in terms of offset in horizontal or vertical direction.

What I assume is that the same project is interpreted differently on your hardware than it is on mine.
When I open your reproduction Unity project that you sent via email originally and hit play, the rest of the body has exactly the same lighting appearance after repacking as without repacking. Could you try if this is the case at your clean unmodified reproduction package as well? If it looks different (what I assume), which target platform are you currently on?
User avatar
Harald

Harri
  • Posts: 1490

Anisoft

Your absolutely right about a new project having it look correct. I'm going to have to really look and see why the heck my main project is producing different results.

I am so sorry for wasting so much of your time on this.Thank you so much for your help.

---

Hey by chance... would it be something with the rendering settings color space?

In a empty test project, the results were exactly the same in gamma space, however switching it to linear the newly made normalmap packs were off color.

Gamma Space


Linear Space


In gamma they are all identical (which likely was how it was for in ur testing)

However in linear (the color space I'm working in) the colors of the repack (left image) are wrong

Easy solution would be to just use gamma however Im using Linear for specific post processing / lighting effects. Its not the same unless I use that color space.
Anisoft
  • Posts: 61

Harald

That is interesting, I will have a look at your example project when switching it to Linear color space.

---

I just tested your reproduction package with color space set to Linear. Still I cannot reproduce your problem above of different results after repacking. I get the same lighting behaviour before and after repacking, as with Gamma color space.

Did you perhaps set any of the normal maps not to Texture Type Normal Map? Or did you accidentally set any diffuse texture to not be sRGB?
User avatar
Harald

Harri
  • Posts: 1490

Anisoft

I'm really sorry for having this drag on so long and wasting your time.

linear:



gamma

This is the same exact project I gave you with the only thing changed being the normals being properly flipped, and the hair being resized correctly.

These are the settings applied to the base texture.


Settings for the hair piece



This is tested on a another computer as well. (my dev partners)


Last test with ambient lighting/reflections off;

Top: Gamma
Bottom: Linear


I tried testing this on a standard Sprite-Diffuse shader and found no difference between versions, however testing it with any shader at all. Its the use of those repacked normal maps this issue occurs.

I'm curious if this may have to do with the normal packing format and the colorspace?

I sent an updated demo that is set by default to linear if you have a moment to check it out. Just open the scene and press play.

---

FOUND THE ISSUE

I went into the AtlasUtilities and testing out forcefully setting the output texture to linear if the index was 1 (since I only have two textures the second texture index 1 would be the normal.

And I found that seemed to negate the issue!

Linear
left: repack right: none


Hack that I did
AtlasUtilities.cs
var newTexture = new Texture2D(maxAtlasSize, maxAtlasSize,
(additionalTextureFormats != null && i - 1 < additionalTextureFormats.Length) ?
additionalTextureFormats[i - 1] : textureFormat,
mipmaps, i ==1);


I'm curious how this should translate into a fix? Setting the diffuse to linear isn't a good idea (since that messes up the color)

So maybe we need the ability to specifiy which textures are linear? and also specifically make normals be linear?

What do you suggest? (if I'm even correct in my analysis)
Anisoft
  • Posts: 61

Harald

Thanks for digging in so deep and for your detailled report! Actually I read your email with the reproduction package first and immediately jumped into analysing and fixing the problem, just to come back to see that you did an analysis as well. At least we have come to the same conclusion. :)

I have just released the bugfix, you can download the updated packages here as usual:
Spine Unity Download
Please let me know if this resolved the problem on your sideas well.

The GetRepackedSkin() methods now provide an additional optional parameter:
bool[] additionalTextureIsLinear = null
Usage is as follows:
When additionalTexturePropertyIDsToCopy is non-null,
this array will be used to determine whether linear or sRGB color space is used at the
Texture at the respective property. When additionalTextureIsLinear is null, linear color space
is assumed at every additional Texture element.
When e.g. packing the main texture and normal maps, pass 'new bool[] { true }' at this parameter,
because normal maps use linear color space.
So the default, leaving it at null, shall lead to correct results out of the box.

Thanks for reporting and not giving up on the subject!

For the record:
Issue ticket was: https://github.com/EsotericSoftware/spine-runtimes/issues/1602
User avatar
Harald

Harri
  • Posts: 1490

Anisoft

Thank you for not giving up on the problem! This seems to have worked wonders!

I assume the ability to force stuff to be linear or not is useful beyond just normals for if I have a linear texture I wanted copied over too (so maybe something like a emissive texture?, actually I'm not fully sure what its other uses could be)

Tbh I'm not really sure what the usage of checking stuff not to be srgb in the texture importer is for too, I know I was messing with it for my own normal map generator (and thats how i came across realizing I'd need to make sure my stuff converted to gamma/linear for it to be accurate to my reference)


Thanks again for being so quick to helping me with this issue. It really means a lot!

I'll be writing a system soon that allows for baking out colors from a material into the spine texture (for character creators skin/item colors).

Ill be sure to share when I finish it.


Thanks so much Harald
Anisoft
  • Posts: 61

Harald

Glad to hear that it works as expected now! :)
Anisoft wrote:I assume the ability to force stuff to be linear or not is useful beyond just normals for if I have a linear texture I wanted copied over too (so maybe something like a emissive texture?, actually I'm not fully sure what its other uses could be)
Emissive textures are a pitfall here - they shall be declared as sRGB textures. This thread on the Allegorithmic forum (you know them from Substance Designer and Painter) summarizes it quite well:
https://forum.substance3d.com/index.php?topic=8745.0
Wes McDermott wrote:The simplest way to look at it is that if its color we see, then its to be treated as sRGB and the gamma needs to be removed. This is albedo, diffuse, base color, specular. [..] Yes, emissive should be sRGB. A good way to look at is like this...

If it's a color that you see then it's sRGB. Everything else is linear. Basecolor contains diffuse reflected color and metal reflectance values. Both represent light rays bouncing back and picked up by our eyes. Emissive is the same.
Hm, and Unity seems to expect Metallic textures to have the sRGB property checked (for whatever reason, maybe filtering as mentioned in the forum thread):
https://forum.unity.com/threads/metallic-map-linear-vs-gamma.425872/#post-2752018

One exception to the "if you see color, it's sRGB" rule is if it's .exr or .hdr files or other floating point formats, which will not be gamma-encoded, the content is already stored in linear space. I assume Unity will ignore the sRGB property at these files anyway, so it should not become a problem.
Anisoft wrote:Tbh I'm not really sure what the usage of checking stuff not to be srgb in the texture importer is for too
When the project is set to Linearcolor space, all color and lighting computations will take place in Linear color space. When the sRGB property is enabled at a texture in its import settings, it declares that the 8bit per channel color data was encoded in sRGB color space typically for viewing on a monitor (a.k.a. "my dark values are important") and will be converted to linear space color (a.k.a. "light adds up and multiplies this way") when sampling a texel.

Regarding normalmaps:
Note that when using Photoshop in a default setup, the color chooser's 0-255 values are showing sRGB values. Now a normalmap's red channel is authored as "0 sRGB is left, 128 sRGB is neutral, 255 sRGB is right". Although when storing it as a .png image it is stored in sRGB color space, you want the 8bit integer values to be interpreted directly as 0, 128, 255 and you don't want any gamma correction being removed from these colors. It is true that the colors of the normalmap after being interpreted as linear would look brighter, but we do not want to preserve the "visual look" of the color image, but instead it's integer data values must stay the same. When you enable sRGB in the import settings, you declare that the "visual look" shall stay the same and the data values representing it shall be interpreted accordingly, leading to different data values at the texel than the ones you originally chose in sRGB space.
Ill be sure to share when I finish it.
Great! :)
User avatar
Harald

Harri
  • Posts: 1490

Anisoft

Thanks for the detailed explanation!
It is true that the colors of the normalmap after being interpreted as linear would look brighter, but we do not want to preserve the "visual look" of the color image, but instead it's integer data values must stay the same.
Interesting, that's good to know. I do apply correction for any input textures in my compute shader or the normals are completely wrong.

I found it strange that for the renderTexture, applying a final gamma/linear correction was needed for the visualization of the RT to look like the ref (or it was too bright).

However when exporting to an actual Texture2D I found no visual difference to the output texture whether the final output was corrected or not...

I'm curious if I should be setting my Texture2D there to be linear..


So in all very confusing haha, hopefully I wrap my head around this soon. Ill re-read ur explanation a few times and do more testing and see if I can fully get it (cause as of now its a lot of trial and error)

---

To clarify I apply a RgbGammaToLinear (taken from another post here) to any textures I add to my compute shader so that the normals output a correct looking image.

I don't apply any conversion beyond that to the final output tho due to me saying the output to Texture2D from the RenderTexture appears the same in both either way...

So the RT is super bright, but the Texture2D isn't (if i apply a LinearToGamma, it fixes the RT and appears to have no effect on the final outputted Texture2D from that RT...)

curious as to why that is... (maybe i just think it looks the same... tho it really doesn't 100% look the same)
Anisoft
  • Posts: 61

Harald

Anisoft wrote:Interesting, that's good to know. I do apply correction for any input textures in my compute shader or the normals are completely wrong.
Do they behave completely wrong, or do they just look wrong?
I would test this by replacing the shader code of the sampled texture value with float3(0.5, 0.5, 1.0), as a reference "straight up" normal color (the neutral violet in sRGB).
Anisoft wrote:I found it strange that for the renderTexture, applying a final gamma/linear correction was needed for the visualization of the RT to look like the ref (or it was too bright).
That would make sense to me - if the render texture was linear (or/and a floating point texture) and the visualisation part did not apply any additional LinearToGammaSpace conversion.

Unfortunately, there are quite a lot of possible combinations of how things can be configured and where automatic conversions could take place.
As as summary, I would always test with some hard-coded reference values and see if sampling the texture also returns similar values. I would not rely on the visual appearance.
User avatar
Harald

Harri
  • Posts: 1490

Anisoft

Sorry it took so long to get back, so I found the issue...

It was wrong...because of my color settings.

If you have the game set to gamma or linear (as we discussed) it changes how unity imports stuff. So changing the maps I import for the shader to be linear solved that issue. (Otherwise like you said the import would be different based on the color mode)

I discovered this because my compute shader light profile normal generator was getting diff results in another test project. That test project being gamma instead of linear.


Setting all the imported textures to linear solved both cases.

Just wanted to make sure I followed up. Sorry for the delay, I didn't get around to testing your questions until a few days ago.
Anisoft
  • Posts: 61

Harald

Thanks for letting us know! Glad it all behaved consistently in the end.
User avatar
Harald

Harri
  • Posts: 1490


Return to Unity