We've been working on this issue too...
So far boiled it down to the material is cached at the very beginning and every frame that it reconstructs the mesh, it reverts back to the cached material since it has to keep track of every material per attachment (if I'm using the terminology right).
If you found a good way, do tell.