Generative AI(GAI) and future of asset creation

[Redsky]

There is a massive shift coming to market of asset creation that I'd like to address or at least start outlining here. This is not a blog post, nor an advertisement of the methods described below. The purpose of this thread is to gage the position on the subject as it relates to Unv and strategize about our approach(if any) to the whole thing.

Some applications recent GAI advancements made feasible come to mind:

1. Texture work

   1. upscaling (img2img) legacy textures

   2. text2img for generating textures from prompts

   3. inpainting customizations (cracks, rust, vines, smears etc.) on tile images

   4. creating textures from images, including normal/relief/spect/pbr sets

   5. making textures tileable with inpainting

2. 3d model creation

   1. text-to-3d and img-to-3d for low priority/background prop generation

   2. 3d model generation as a base for finished 3d assets

3. Concept art

   1. concept iteration using img2img over 3d, photos, and drawings

   2. concept initialization using text2img

Definitely not an exhaustive list and one that completely sidelines sound and text generation, but I wanted show how fundamental shift this is and just get some opinions. I'd like to also point out that above are some of the lowest hanging fruits available today. We could actually be using these today!

Here are some issues I foresee coming with using GAI technology:

1. Legality - works created even partially with GAI have unclear copyright status, it's not even clear to me if a CC license can be applied to derivative works. Some platforms like Steam require apps to disclose all GAI content used, a temporary solution in my opinion, giving them an out if copyright law cracks down on generated content.
   I should note here that not all models are created equal and a lot of questions of legality/morality depend on contents of training set for given model.

2. Ethics - other then law, some people, especially in art scene have very heavily condemn the practice

3. Quality and taste - cheap and easy solutions have their price.

   • Easy creation hides the artistic technical debt. Just because you can create something appealing doesn't mean you can make it fit with other things in some consistent way. You may not understand what makes it appealing in the first place. As an analogy, you can allow llm write you some code, but you would have no idea why it works, how you can change it and if perhaps it is broken in some way if you couldn't write it yourself in the first place.

   • AI generation is likely not create value on it's own, unassisted. It's likely to create contextless noise, with no profundity, no 'uh-huh' moment, just ceaseless steam of "Oh! This looks interesting... nope, it's nothing" experiences, which not only undermine the enjoyment of watching a piece of content, but teaches viewer to become disengaged. Which can be very bad for things with actual depth in it.

[Sweet]

I am intrigued by the prospect of generating a set of textures that would fit together in some consistent way. That could give us maps with completely new textures, maybe in the foreseeable future. Do you think that will really happen?

creating textures from images, including normal/relief/spect/pbr sets

As you suggested a while ago, I did some experiments. Either normal or relief/height map is good enough to deduce the other with reasonable quality. PBR sets usually come with a normal map, but not with a height map. I have used the program https://gitlab.com/xonotic/xonotic/-/bl ... eightmap.c with some success. It "integrates" normal maps to yield height maps. In practice this means we only need AI to generate normal maps, and that we can easily import most of what PBR does.

Legality is a really messed up topic. If we do it right, we have to consider every jurisdiction on this planet. When it comes to new things like AI image generation, this is a most difficult task. I guess that, out of neccessity, the stock packages will require a conservative approach.

We could start with a small test set of textures. I could generate a map to view them in the engine. There are not ethical problems in doing experiments, I believe. When do we start?

[Redsky]

In past two days, during our discussions on the subject I've manage to establish roughly what our current position on the matter is.
To illustrate I'll cite chats excerpts of first impressions to the original post.

If it looks good, then I don't really care where it comes from (so long as there are no legal issues)

agree with reaper

[...] basically my own opinion is that it seems OK to me if that is not competing in areas people are competing for creativity.

AI can’t (yet) tell us that we put its textures on backward

Responding to me concluding I'm not getting my priorities straight (due to lack of response).

i do not think so
we are simply lacking people doing actual content

My bad, I always forget how full everyone's plates already are. I should have just asked first impressions in chat to begin with (thanks for reading BTW).

Illwieckz as always keeps focus on art / craft aspect of the project. This a good point: what is really important in all of this?
Whether AI tools limit space for creativity is a difficult question I do not have a ready answer for. As someone who's area this tech influences most I think there definitely is a good way of using it and that there is plenty of work to be shared and/or duplicated to see who can do it better.

Consider also that assets can be recursively used. You can take AI generated textures for a leaf and apply it onto a model of a leaf that will be then used to create a model of branch with leaves that will then become the high poly base for baking a texture for a very low poly single surface very detailed looking in-game model.
Branch represents a perfect example of complex structure that image generator would loose track of... that's my guess at least. I'll return to the subject of how I suspect GAI content can be embedded in later post. The point is on every level there is space for improvement on every level, plenty for skill and craft.
h

Finally GAI being a force multiplier can only partially provide a solution to being shorthanded; as a set of potentially very powerful tools, contrary to what some AI hypemen say, still needs (arguably) skillful, careful operation. Proper usage still requires experience.

We could start with a small test set of textures. I could generate a map to view them in the engine. There are not ethical problems in doing experiments, I believe.

I agree with the sentiment. It's more reasonable to start experimenting with AI generation in isolation to other assets to contain possible damage if later it becomes a problem legally or an issue limiting distribution. It shouldn't IMO, but it's a crazy world we live in.

When do we start?

Two things I wanted to do beforehand (in no particular order): acquire better hardware and stabilize content pipeline I used for producing height/relief maps for atcshd. Best way to do that is by finishing atcshd and pushing out another texture pack.
I think texture pipeline once setup should be as powerful for map makers as AI tools. After all creating model of tiles shouldn't be hard for someone putting together a map. It's literally the same, but at the end you add subdivision surface modifier(optional) and render to texture.

So for month or two I want put it AI on the back burner: I'll slowly add more links to resources and software that may become useful when we decide to engage the subject head on.
Also I think we might want go all out and create a whole map. Maybe something parametric?

---

Example of Low-Rank Adaptation (LoRA) of SDXL tuned for creating sets of PBR textures!

Have a browse through some other models for texturing

Write up on Stable Diffusion WebUI extension for creating depth maps, with possible texture creation application

And here is said img2img depth aproximation (MiDaS)

A user prompt gallery (filtered for textures)

Few tools:
Stable Projectorz (3d generation/texturing program) showcases more streamlined ai assisted design (video)

Krita plugin for inpainting (video)

SaaS Texture Generator

There are plenty of 3d model generators online haven't looked into them yet.

A few example what can be done, not what should be done:
"Codeless development" equivalent for ai asset creation (video)
Using ComfyUI node-based workflow to create "consistency" in "6 easy steps" (video)