Milestone 2

[RonHiler]

I've posted the second version of the Design Doc, which has little to do with MD Milestone 2 except that is the version we will be using to code M2. Note that you may have to refresh your browser to pull it up (make sure the cover page says version 2 on it). It's gone from 35 pages to 42 pages, which is mostly because I added another couple of milestone details and a progress page, and filled in a few other details.

There are only 8 steps in M2, the overall goal being to get a basic version of the world screen into the game. As far as I can tell, all of M2 is stuff that has been done before, so I anticipate it being pretty quick. I will begin coding M2 sometime this next week (I have to spend a few days working on the DDO character planner).

In the meantime, if anyone wants to contribute, be looking for bugs in the M1 version, and again comments on the current DD are more than welcome.

[RonHiler]

Hey guys, just to let you know, coding for Milestone 2 has begun.

[RonHiler]

I have been working on M2 this week, kind of in tandem with working on the character planner (you may notice I just released 2.75 of the planner, so that's done for a while, at least until we get M2 out the door).

As I said before, M2 only has 8 steps, the primary goal being to get the World screen into shape. It now has the basic shape put together, and I started the second step, which is putting up the framework of the world map.

Now, I could just calculate the points for the quads and draw the thing easily enough, but I wanted to put up the classes and structures needed so that it could be easily expanded when we hit Milestone 7 (which is where we will stretch texture maps across the skeleton and show an acutal map).

So all that lead me to our old version where I ran through the code that I had done for that. Much of it has been pulled lock-stock-and-barrel over to the new version, because I thought the basic premise was correct. I can see several places where it can (and should) be optimized for better speed, but overall it's really in pretty decent shape.

Anyway, the whole thing kind of lead me to think about the memory requirements of that main map, which really is one of the primary hurdles we are going to have to deal with. The calculations are the same as they ever were, and I won't go over them again, but suffice to say, if we want a globe that consists of 64 x 32 quads, and each quad needs a texture of 512x512 (which based on prior experiments is where I think we need to be in order to have adequate resolution) we need about 3 GB of memory (that would hold 2000+ textures in each of 512x512, 256x256, 128x128, and 64x64 resolutions). Clearly that was never going to work (which I knew perfectly well with the old version) so I had begun (and more or less finished) implemention of a tile manager to create textures on the fly. Anyway, all of it was good code, so I'm keeping it, but I'm going to optimize quite a bit of it to make it run faster.

I spent about two hours today working on the design doc talking about this very issue, since it really is, in my opinion, one of the biggest technical problems the game faces. And I felt that was something that should be in the DD sooner rather than later, since what I'm working on now touches on that subject, even if it is just the skeleton of the globe and there won't be any textures on it for a bit.

So, between the fact that we now have a solid plan built into the DD and that I have code already for most of it, I think getting that framework to the screen should be pretty straightforward. I ended up putting most of the code for it in already, just have to make some tweaks to the tile manager and we should be all set. And then of course getting the smaller globe map onto the screen is the same process, only smaller :)

I won't make any definite predictions yet on the release of Mod2, but I do expect it to be relatively soon. Certainly much faster than finishing Mod1 was.

[Strike]

That sounds like something I'd like to read....
But with my current schedule, I'm going to have to carve out the time to do so, and I don't know when that'll happen.

..As much as not having a job sucks, sometimes /having/ a job is worse.

[RonHiler]

Cool. Just don't look for it yet. It will appear in version 3 of the DD, which has not yet been posted.

[RonHiler]

I finished up the last few details of steps 2 and 3 this morning, and did 4 this afternoon. Basically steps 4-7 are all about getting the message box on the world screen up and running, so we're already part way there. Step 8 is just adding a few buttons, none of which are going to do anything yet.

I put in the text box class (step 5) and it works, but it doesn't have access to a scrollbar yet (for when there is more text than will fit in the box). So I'm going to recode it a little to implement that. I'm sure at one point I had a similar class doing that already, so I'm going to look through some older code and see what I have lying around.

So we're halfway through milestone 2 now. Although in reality the hard coding was in steps 2 and 3. The rest ought to be fairly simple.

[RonHiler]

Alright, I'm now done with steps 1 to 5 (out of 8). I'm 95% done with step 6, and 80% done with step 7. The final step will be super short (just putting up buttons, they won't even do anything).

The one thing that may hold us up for a few more days is bugs. As the program has gotten more complex, bugs have started to rear their ugly heads :) So I have two or three to squash, then we should be ready for a milestone release.

[Shane Christopher]

Ron, about the texture sizes not fitting in memory, would it be possible to have the textures saved to disc and load only the textures visible and those next to them into memory? If zoomed out then 256x256 versions of the textures might suffice. Unless you're madly spinning the globe around (at which stage we could have just 128x128 textures for the rest of the map quads preloaded to use) then you should have time to load the textures up into memory and discard the ones that aren't needed anymore.

This would essentially be like google earth but on a simpler scale and direct from your harddrive not over the net.

[RonHiler]

Same idea as what I have, except instead of caching them to the drive they are being created on the fly. The world tile texture manager actually keeps track of what's already been created, so if it already exists, the program won't recreate it. It also keeps track of usage, so when it runs out of room to create a new tile, it destroys the tile which has the longest "not-shown" time.

I THINK creating them on the fly would be quicker than running to a drive cache. I could be wrong. Certainly it could be something to try. Perhaps we leave in my system, but instead of destroying non-used textures, we save them to the drive. Then we just add in an extra step to check the drive before trying to create them. Wouldn't be hard to do.

This is essentially what the manager does now:

1) Does the tile texture exist in memory? If yes, skip to step 5.
2) If no, create the texture from world data.
3) Is there room to save the new texture in memory? If yes, add it to the tile manager.
4) If no, destroy the oldest non-used texture and replace it with the new texture.
5) Display the texture.

We could change it to:

1) Does the tile texture exist in memory? If yes, skip to step 6.
2) If no, does the texture exist on the drive? If yes, load it.
3) If no, create the texture from world data.
4) Is there room to save the new texture in memory? If yes, add it to the tile manager.
5) If no, save the oldest non-used texture (if it does not already exist on the drive) and replace it with the new texture.
6) Display the texture.

That's actually not a bad idea. Presuming loading from the drive is quicker than creating the texture in the first place. If not, then it's a moot point, heh. I'll do some speed testing when we get to that milestone (7, I think) and see how it works out.

We could, in theory, even prebuild the textures at initialization and skip all the creating while in game, saving them to drive. But it's 2.6 gigs worth of data. Might make startup kinda slow.

BTW, I'm finished with M2 except for the bugs. It's looking pretty good, except I'm not happy with the buttons on the world screen. They don't really look very good, but that's what happens when you use programmer graphics, heh.

[Shane Christopher]

I would be very surprised if loading a texture from the drive would take longer then creating it again. Even with a slow 50mb/s drive the image size would be roughly 3/4 of a MB. That would be 0.015s per texture.. theoretically of course ;)

[RonHiler]

A 512x512 texture, assuming 4 bytes per pixel would be 1 Meg, actually :) But your point is well taken. Unless there are seek times that add significantly to that, you're probably going to be right. For sure I will test it out when we get there.

Of course, the smaller ones will be even faster. We're also going with 256x256 and 128x128 depending on the zoom level (or actually, the number of tiles visible at once) and 64x64 for the globe map textures.

So definitely a good idea. Thanks for making me think about it :)

[RonHiler]

Milestone 2 Known Issues
When in the world screen in windowed mode, resizing the screen (either by grabbing the window border and moving it or by maximizing/restoring the window) too many times will cause a program crash. Other screens seem to be unaffected.
When in windowed mode (any screen) minimizing the window will cause a program crash.
When drawing bounded text when the text does not fill the boundaries, placement of the text is incorrect. This is especially noticable in the progress bars.
Text boxes with scroll bars will sometimes not display the last line of text.I may or may not get some or all of these fixed before I release M2, which will be most likely toward the end of this week. Despite the bugs, I'm going to leave the program in windowed mode for M2 rather than switch it to fullscreen mode (starting in M3 you'll be able to select one of these options yourself). Just letting you know these are the bugs I'm working on and they may still be there.

[RonHiler]

Sorry guys, I delayed the release a couple of days. I felt the first two bugs were too important to ignore (since they were crash bugs), so I spent some time with them. The first one in particular was pretty difficult to fix. The good news is I tracked them down and killed them. So we're just about ready. I want to clean up a few things, so figure on seeing the M2 version on Saturday.

[Shane Christopher]

Missing cc3260.dll from the latest release. Gonna go get that and put it into the game folder.

Okay now it's missing STLP45.dll. Getting that..

..led to this:

http://i226.photobucket.com/albums/dd52/Dargish/error.png

Clicking OK exits the program.

It could be something to do with me having a 2900XT. Maybe whatever video card recognition you're using doesn't cope well with newer cards?

[RonHiler]

Hmmm, did M1 run okay for you?

[EDIT]

Okay, when you get a chance, can you do a couple of things for me?

1) Can you post the contents of your log file. It is in your MD folder called "MDestinyLog.rtf"

2) After you do that, download the new version of M2 I just posted. Then rename those two DLL files you downloaded to something else (so they don't get loaded dynamically) and see if the dependencies are gone. I think they should be (I swear, that setting in my compiler drives me nuts, I have no clue about Borland specific dll dependencies until someone else tries to run the program that doesn't have Builder installed. Makes me crazy).

3) If they are gone, run the program. If not, re-enable the DLLs and run the program and see if you still have that VB creation problem. If so, post another log file for me.

These kind of bugs are fun :) Whenever you have the chance, no particular hurry.

[Shane Christopher]

Dependencies are gone but still the same error. Here's the log file:



[0.000350981]*** Logging Started
[0.000403126]
[0.0130567]DirectInput Object Created DirectInputClass::InitDirectInput()
[0.014031]DirectInput Keyboard Device Created DirectInputClass::InitDirectInput()
[0.0141415]Cooperative Level Set DirectInputClass::InitDirectInput()
[0.0144608]Keyboard Data Format Set DirectInputClass::InitDirectInput()
[0.0144948]Keyboard Buffer Created DirectInputClass::InitDirectInput()
[0.0149246]Keyboard Device Acquired DirectInputClass::InitDirectInput()
[0.0212031]Direct3D Object Created Direct3DClass::Enum()
[0.0287456]Combos Enumerated Direct3DClass::EnumDevices()
[0.0304614]Combos Enumerated Direct3DClass::EnumDevices()
[0.0305332]Combos Enumerated Direct3DClass::EnumDevices()Unsupported Device Type Detected (3 - D3DDEVTYPE_SW [This is normal, logged for information purposes] Direct3DClass::EnumDevices()): Not available - D3DERR_NOTAVAILABLE
[0.0305823]Devices Enumerated Direct3DClass::EnumAdapters()
[0.0306141]Direct3D Enumeration Complete Direct3DClass::Enum()
[0.0306497]Retrieved Current Adapter Display Mode Direct3DClass::Enum()
[0.0427197]Direct3D Object Created Direct3DClass::InitD3D()
[0.0495703]Direct3D Device Created Direct3DClass::InitD3D()
[0.0496566]Current Adapter Display Mode Obtained Direct3DClass::InitD3D()
[0.0497337]Cannot Create 2D Vertex Buffer Direct3DClass::InitD3D(): Out of video memory - D3DERR_OUTOFVIDEOMEMORY
[2.85352]Texture created TextureManagerClass::GetTexturePointer: TextRenderTarget
[2.85495]Texture created TextureManagerClass::GetTexturePointer: MovieScreenClass::MovieFrame
[2.94374]Unloaded Graphic File TextureManagerClass::ReleaseTexturePointer: MovieScreenClass::MovieFrame
[2.94528]Unloaded Graphic File TextureManagerClass::ReleaseTexturePointer: TextRenderTarget
[2.94537]
[2.94539]*** Logging Ended

[RonHiler]

Hmmm, that wasn't as helpful as I hoped it would be, basically it gives you the same error as the popup.

Alright, one more thing. Run Dxdiag.exe, and post (or send me) the results. I need to look at your video card stats. I don't really see why the vertex buffer creation should be failing. That is the very first thing D3D allocated in video memory, so you can't possibly be out of video memory. We know the device was successfully created, then the program does this:



D3DDevice->GetDeviceCaps(&D3DCaps);
MaxNumVertices = D3DCaps.MaxPrimitiveCount/4;
//create a 2d and 3d vertex buffer for general use
r = D3DDevice->CreateVertexBuffer(MaxNumVertices*sizeof(GENERIC2D VERTEX),
D3DUSAGE_WRITEONLY | D3DUSAGE_DYNAMIC, D3DFVF_GENERIC2DVERTEX, D3DPOOL_DEFAULT, &TwoDVertexBuffer, NULL);


Do you think your card might be returning something crazy for MaxPrimitiveCount? Maybe I ought to put a cap on that value instead of just letting it go crazy with MaxPrimitiveCount/4

[Shane Christopher]

Well it might be.. I'm not sure why it would cause it to crash though.. I'll run DXDiag now. Actually I know what it might be.. I forgot that I'd used cracked drivers to emulate a FireGL 8600 for 3D work. It speeds up OpenGL precision rendering by about 5 times. I'll revert back to the official drivers and see if that was the problem.

[Shane Christopher]

Well that wasn't the problem. Got the same message when loading it up this time with the latest official drivers.

I've attached my dxdiag file.

[RonHiler]

Nice card!

Alright, I just uploaded a new version (0.080.m2.r3). Give it a shot and see.

There are two possibilities.

One is your MaxPrimitive count is reported something totally insane and the VB creation routine is eating all 512MB of your card memory. I consider this really unlikely (each VB is about 30 bytes, so you'd need to be creating over 178 million VBs to chew up that much memory). I checked the Caps on my card and I'm getting a MaxPrimitiveCount of 5592405, which divided by 4 and multiplied by 30 means I'm chewing up about 40 MB of video memory for each of the buffers (2D and 3D).

80 MB of VB is WAY too much video memory used, heh. I was pretty foolish not to put a cap on those buffers (I had no clue the MaxPrimitive count would be so high) , so now I've capped them at 500 each, which should be plenty. If not we can easily bump it higher. The world and globe maps use 2112 and 544 buffers respectively, but they have thier own static buffers, they don't use the generic dynamic ones you are failing on.

So, on the off chance you really were using up all your video memory on the buffers, that will be fixed now with the 500 cap.

The other possibility is there is your driver doesn't like something about the creation parameters, and is erroneously reporting it as an out of memory error (it really ought to give an "Invalid Call" error, which is D3Ds very unhelpful way of saying "I don't like these parameters you've given me").

We'll see what happens. I added a Display Adapter Cap dump to the log file, so if you do fail again, we should be able to figure out why (I hope!).

[Shane Christopher]

It worked =)

Looks good so far. Might need a tender artists touch someday though ;)

[RonHiler]

Cool! So that must have been it. I still don't believe we were using up 512MB of memory. You must have a hard cap on how many MB can be used for VBs, or possibly how much memory can be allocated at once. Can you look in your log file and tell me what it says for MaxPrimitiveCount? It's about 2/3rds of the way down the Display Device Caps dump.

[Shane Christopher]

Max primitve count: 1431655765

At 30b per primitive that's 42949.67295mb which might well be going over 512mb slightly ;)

[RonHiler]

1.4 billion :eek:
Holy crap! No wonder you were running out of memory, LOL.