Materials

[RonHiler]

There is some significant informatoin coming on this front. Not quite yet, as I have to do a little design work first. And by a little I mean a lot But since I have about 10 hours on a plane to kill, I should get through quite a bit.

This is going to be a major system. And quite a selling point for the engine, I think. I envision big things for this part. Stay tuned, I'll be putting up some serious stuff shortly...

[RonHiler]

I'm itching to talk about this system. I have a lot of notes now, and a pretty good idea of how everything is going to work. But to really effectively talk about it, I need a picture of the GPU pipeline, which I can't do from where I'm at.

One thing I will say. Materials are hard. Gone are the days when you just slapped a 2D texture on a 3D model and you were done. Materials have become a highly complex subject (in fact, from a design perspective, I would argue they are pretty much among the most complex systems, assuming you are using a decent engine), and there simply is no easy way to present them without losing the power you (as the designer) have over the GPU.

So yeah, in order to design materials, you will have to have a certain level of knowledge. My job, as an engine designer, is to simplify all the various systems that make games run so that the game designers can focus on creating games (rather than creating code). But if you take that too far, you lose the flexibility you need to create really good games. So it's a fine balance. And in the case of materials, I'm going to err on the side of flexibility and power over simplification. In my opinion, if you are a material desgner, you should know (or need to learn) at least a bit of HLSL anyway (or GLSL in the case of OpenGL).

That being said, I want to help the designers as much as I can. And that's what this system is all about. I'm going to take a few cues from the DDO planner, actually. Because in some ways, designing a material is not so different from designing a character in DDO. Only instead of feats and spells and equipment, you have constant buffers, samplers, and textures. Fun stuff

[RonHiler]

I wish I could wrap text around this image, but whatever. There's only so much you can ask from your forum software I guess

I probably won't have time to get through everything I have to go over. I have roughly 10 trillion things I have to get done today. But I do want to start, because there is a lot to go over.

So I'm going to post an image of the GPU pipeline as it is implemented for DirectX11. This is an image from one of my reference books (which I'm using without permission, so hopefully they don't yell at me). This is about the clearest and most complete image of the pipeline I've ever seen, they did a really good job with it (I could pick a couple of nits with it, but generally it is great). The other renderers are similar, though simpler (they are in fact exactly the same except you just remove a few parts here and there. For example, the DX10 pipeline is the same minus the Hull shader/Tessellator/Domain Shader group). So I'll talk a little bit about this after the image...



First off, why am I posting this in a discussion about materials?

I've seen materials handled in a variety of ways, which all boil down to connecting nodes together graphically. And I have issues with almost all of them. For example, Maya gives you "Nodes" that you connect together, with each node performing some function on the fragments (it is interesting to note all of Maya's nodes work on the pixel (or fragment) shader, basically that one white box in the diagram, so you can see why I might have issues with it, heh). C4 does a similar thing with connected nodes. Other programs that I've played with do very similar stuff. Presumably, each of the "nodes" ends up creating a subroutine in the shader code, and there is some master program that simply calls those subroutines one by one as determined by your node connections.

I think the reason most programs try to give you nodes that you connect together like this is because materials are created by artists, and they are trying to keep the programming out of the hands of the artists. Which is certainly understandable, artists don't like to write code. However, I think that is a paradigm that is quickly changing. One of my biggest problems with the "node" method is probably pretty clear already. Look at the power available to you in that pipeline! With the nodes system you are limited to basically that one box (the pixel shader). Which is absurd. Of course, you can do lots of things with just the pixel shader, but think how many more you can do using the full pipeline! The problem is any time you try to take a complex system and put your own methodology over the top of it to present to the user, you are going to lose some of the power and flexibility of the underlying system (this is the same reason I tend to stay away from third party libraries and code straight to the metal myself). Nodes are great and you certainly can do a lot with them, but they really limit the power you have over the pipeline.

I guess I probably ought to describe a little bit what you are seeing in that diagram. Ignore the yellow arrows (they are just pointing to bits the text was talking about at that point in the book). Basically, you have two types of stages. The green stages (Input assembler, tessellator, rasterizer, etc) are configurable, but not programmable. They have fixed functionality which you cannot change, though you can configure it (for example, if you were drawing a wireframe rendering of your scene, you would do that by configuring the Rasterizer). The white boxes, on the other hand, are programmable. These are the shaders, and these are what you write HLSL (or GLSL for OpenGL) code for. They also have certain jobs (i.e. the pixel shader must output a fragment color and/or depth value to the Output Merger stage), but in the process of doing those jobs, you can do whatever you like.

The little yellow boxes attached to various points on pipeline stages are your direct inputs to that stage (this is the data they get from the CPU, in addition to the data they get from the prior stage of the pipeline). The shaders all have the same inputs (constant buffers (cb), samplers (s), and buffers (t)). The configurable stages, on the other hand, have different sorts of inputs (e.g the Input Assember has a few vertex buffers, as well as an index buffer, and so forth).

Now, this might seem horrifically complex. And it is, no question about it. But I have a plan

Not long ago, I had the opportunity to work on a piece of machinery called an LCMS (it stands for Liquid Chromatography Mass Spec). This is also a horrifically complicated machine. There is an HPLC on the front of it which has a programmable pump to control flow rate and mixtures of solvents over time. It follows into a UV detector which is also programmable for various wavelengths of detection. It then flows into the actual MS part, which has several components (ion traps, energy colliders, focusing lenses, detectors, and so forth) each stage of which is configurable and/or programmable. But the important thing is that, just like our pipeline, the stages followed one after another with certain expected inputs and certain outputs. And at the end of the pipeline came some data points which were sent to the screen. Really really complicated, right? How the hell do you control something that complex?

What the software manufacturer did (which was not me, BTW, I didn't write this particular bit of software) was to put a diagram of each stage of the machine over the top of the screen. The user then selected a particular stage, and all the options for that stage came up (with their various checkboxes and input fields). If you broke it down into what each stage did, you took a very complicated instrument and made it quite manageable for the end user. And that's just brilliant. Similarly, the DDO character planner does something fairly close. Characters in DDO are orders of magnitude more complex than in your typical MMOG. And we handle that complexity by breaking it down into manageable chunks. And that is very similar to what I intend to do with materials.

I'm going to stop here for a bit. I have more to talk about yet (though I think I've hit the main points). But as I said, I have a few things to take care of. I'll be back...

[RonHiler]

Okay, so where was I?

Let me say next that when you are talking about materials, to be sure, the most important part of the pipeline is the pixel shader part. This is where the majority of the work is going to be done. Hence the reason most material editors focus in on that part of the pipeline. And in fact, that will be one of the drawbacks of a system like the one that I propose. The pipeline has mixed in it not just elements concerned with the material, but also the model and in fact more global elements (such as the depth buffer at the Output Merger stage, which you are likely only to have one of, or perhaps a handful, for your entire game).

Even so, I think it is not wise to totally ignore the other shaders when it comes to materials. Sure, in a lot of cases you are just going to working with the pixel shader. But sometimes you may want to modify the look of the material a bit higher up. There are two possibilities when you are designing materials. You may be creating a material for general use (e.g. wood grain), or you might be creating a material for a specific model (e.g. the main character material). So what I think we will do is give the user the option to input a particular model (and we'll set that into the Input Assember stage at the Vertex and Index buffer inputs). We can provide a couple of generic models (a cube and a sphere), as well as a list of all the models used in the designer's game that have been imported into the engine. That give the user a model to look at while they are designing their material, be it a specific one or a generic one.

When we save the material, we won't save the model data. The connection between a particular model and a particular material will be done at the model end, not at the material end. The material won't care what model it gets fed in the final scene.

Now, for the global aspects, which I think are all sitting at the Output Merger stage (the depth buffer, any pixel buffers), I think we will save those as part of the material. But what we will do is allow the designer to select from a global pool of them. Of course, the designer can set up a new buffer set at any time (they may want one depth buffer for the 3D scene, a second one for the UI, perhaps a third one for a rear-view mirror (if they are writing a racing game or whatever), and so forth.

I haven't really talked much about the UI for this, other than to say we are going to have a pipeline graphic along one side of the screen, which will allow you to select a particular portion of the pipeline for modification. In addition to that, we will have an output panel (which shows the selected model plus the material). There will also be an instruction/help panel which is context sensitive (it will give some basic instruction on whatever part of the pipeline you have selected, perhaps some pros/cons for each setting, and so on). Other than that, the rest of the UI would be dependent on what portion of the pipeline is selected. If it is a shader, there would be a code input panel, texture display panel(s), perhaps buffer displays. If it is a configurable stage, we would show the various options available for that stage (in the form of checkboxes, radiobuttons, and the like) as well as buffer inputs where available.

Now, unanswered questions and complications:

What we have here is a setup for a single pass material. Data goes in at the top, comes out at the bottom and is sent to the screen. However, this is not always what we want. We might divert the data at the Stream Output and return it to a second pipeline for additional processing. Or we may let it get all the way to the Output Merger stage, and then feed those buffers back into a second pipeline (this is how you would do full screen post processing effects, like blurring or underwater effects). And I haven't really figured out how we would handle things like that from a UI perspective.

On top of that, the engine itself might want to override some settings. For instance, in debugging, the user might want to see a wireframe mode of the scene (by hitting some sort of key, like F12 or what have you). In such a case, the engine would have to change all of the material settings at the rasterizer level. We could, I suppose, capture the final output of each material and put it through another pipeline which does very little other than convert the buffers to wireframe. But that's a lot of work for the GPU to change one setting. I'm still thinking about what to do with that.

Also, there may be hidden parts to the pipeline. There is a good chance we are going to use a technique known as deferred rendering, which is a way of handling lighting that overcomes a lot of the problems you see with forward rendering engines (however, it comes with its own set of issues). If we do that, all of the materials will really send their data into a series of what are called g-buffers, and a final processing will be done "behind the scenes" to merge the final image. But that would be transparent to the user, they shouldn't have to deal with that aspect at all.

In some places this is still a pretty rough design. And to be sure, I'm second guessing myself on certain aspects of it. We could end up tossing the entire design and just work with the pixel shader (though I doubt it). So if you guys have any thoughts, do let me know. I know this is very complicated, and for some of you, I may be talking greek. Unless you already have a pretty good foundation for materials, this kind of thing is not easy to understand.