Summer's end

The Google Summer of Code is now over, and with that the three student projects we've been hosting this year are considered "finished". That that doesn't mean that they are totally complete yet, something which would be hard to say about Ember anytime, but it means that they are complete enough to be merged into the main trunk. This last week I've been busy integrating them into the trunk: so far I've completed both the improved entity creator and the new support for Mercator terrain mods. Next week I'll work on getting the sound system integrated.
The goal is to have a 0.5.4 release of Ember out as soon as possible, with all SoC projects completely integrated.

I've created two small movies showing off the entity creator and the terrain mods. This all thanks to the excellent glc video capturing tool.
The videos can be seen streaming through Google video or in high def format. The streaming versions are a little too low in resolution to really see what's going on, but they give a good overview of the new additions.

• Entity creator: streaming original
• Terrain modifiers: streaming original

Summer and release

I'm currently away on vacation, touring Sweden and visiting relatives, but just before I left I managed to push version 0.5.3 of Ember out. My initial goal for this release was to make sure that all of the nice media that Jayr has provided should come to better use. My plan was to first work on an updated world, a larger piece of terrain which were more suited for the way Ember handles LOD and which also would use more of the media through the use of new entity types. The idea was also that by working on a new world, authoring it through Ember, I'll be better able to find and fix those places where the authoring tools were lacking.
And this plan worked very well. The first thing I enhanced was the terrain editing tool, which previously only allowed one to edit a single base point at once, but now allows you to edit a wide swath of base points at once. This makes it much more easier to alter large expanses of terrain. I've also added an automated function which generates atlas entity type definitions from existing meshes. A lot of the media that Jayr has provided is lacking corresponding entity types, so I needed an easy way to quickly create entity types, using the new meshes as blueprints.

However, when working on the new terrain I quickly realized that Ember wasn't very efficient in the way it handled memory. Since my development box has quite a lot of memory I hadn't really noticed it, but as I now was trying out different very large terrains it became noticable. I thus had to spend some time going through all of the instances in Ember where it either used memory in a wasteful way, or just plain and simple leaked it. In the end I think I managed to squash almost all of the places though, and as a result Ember now doesn't use nearly as much memory as before. A lot of the memory used however is texture and mesh memory, and I need to take a look at how I can talk to Ogre about better free up textures that aren't currently used. There will probably need to be a texture collecting task running at a certain interval to make sure that unused textures are freed.

The work with the memory cleanup took some time, and I had to revisit my original plan of releasing 0.5.3 together with a new world. I do want to do more regular releases, so I didn't want to wait too long between 0.5.2 and 0.5.3. In addition to that, there were new releases of both Ogre and CEGUI which both weren't compatible with 0.5.2. If we waited too long, we would run the risk of more and more people trying to compile 0.5.2 against these latest releases and failing.
And finally I wanted to do a release because it would provide an opportunity for the GSoC students to sync up with the trunk, preventing their trees from lagging too far behind it.

So in the end I pushed 0.5.3. In the rush to get it out I completely forgot to test it with Lua 5.0. The issue was picked up by Oliver Lehmann when creating ports for FreeBSD. Luckily he was able to update CEGUI to also use Lua 5.1 without breaking any other ports. Lua 5.1 has some nice language constructs which we can't use as long as we also provide support for 5.0, and we'll be looking at whether it might be possible to completely drop support for 5.0. 5.1 has been out for quite some time and I think all modern distros provide support for it, so it shouldn't be any problem.

Looking ahead, the Summer of Code is now entering the final phase, and we'll hopefully be seeing working implementations of the student's projects. It's already looking good, and I'm hoping to be able to merge those projects that are finished into the trunk in time for release 0.5.4, which I aim to have ready no later than three months from now.

Memory usage

My development machine is quite beefy and has 4Gb of memory, so I seldom run the risk of running out of memory. The downside though is that I've been quite lax with regards to memory consumtion, and as a result Ember 0.5.2 contains both some memory leaks as well as uses too much memory in the way it preloads all available resources independent of whether they are used or not. The resource preloading hasn't been a problem previously, but since Jayr has been adding so much more cool media, with normal and specular maps, the memory usage for all media is now quite high.
I've therefore the last week gone through the codebase with memory profiling tools to try to find both unnecessary memory usage as well as memory leaks. The main tool I've used is Valgrind, which is really an excellent tool for all kinds of profiling. KDevelop comes with built in support for the memcheck tool, which finds memory leaks. It will keep tabs on all memory that is allocated and find those places where there's no more references left to unfreed memory. The valgrind massif tool is also very handy. It will track the heap of the application, and with regular interval report how large it is, and from where different sections of the heap has been allocated. The massif tool by default outputs its info in the console, which isn't very easy to navigate. I've however found an excellent tool for displaying the data in a much better graphical view over here.
Using these tools I found an issue with how the terrain splatting materials were generated, which in normal use amounted to a memory leak of ~100Mb. Quite a lot of data. Fixing it was trivial.
I also found some issues with how Ember was shut down, and how the main scene manager failed to be deleted. While this doesn't amount to any in game memory leak (the application is shutting down anyway, so unfreed memory will be freed a second later when the process exits) it did clutter the valgring output, and we should of course always strive to have components that correctly handles the allocated resources.
I'm now trying to find the cause of a problem I have with ModelDefinitions not being correctly deleted. They use the Ogre SharedPtr, which makes it possible for them to keep on living a little longer, even though the ModelDefinitionManager has been deleted, as long as something else is holding on to a reference. I'll have to investigate further.

I've also removed the preloading of all assets. This will make sure that only those assets that are needed are loaded, but the main (and quite big) drawback will be that the game will stutter as the player enters and moves through the world. This is because Ember so far is single threaded, and the main thread will have to wait while new assets are loaded as they are encountered. We're planning to move to a multithreaded resource loading solution, but we haven't started on that blueprint yet.

Much of this work is a result of the work I've done in trying to create a new island world. Since the island world is quite larger than the current world the memory usage issue became much more obvious. The plan is to release 0.5.3 with the new world. However, I'm leaning towards perhaps trying to get a 0.5.3 release out without the new world. One of many reasons is that Ogre 1.4.8 which was released after the Ember 0.5.2 release changes how resources with absolute paths are loaded. This clashes with how we load our terrain config file, and the result is that while Ember 0.5.2 compiles without problem with Ogre 1.4.8 it won't be able to run. We've already had some issues with people encountering this problem, and we'll sure have more and more. In addition, Nvidia releases a new driver version not long ago which contains some bugs which will either hang the application at startup or produce messed up mipmaps when used with Ogre 1.4.8 and lower. As a response Ogre therefore did a 1.4.9 release yesterday which contains fixes for these issues. As more and more people upgrade to the newer nvidia drivers we're sure to see more of this issue. I'm therefore thinking about trying to push a 0.5.3 release out as soon as possible. Before I do that I would however very much like to see all major memory issues resolved.
There's currently some kind of memory leak which leaks ~4kb/s when just running Ember, without even being connected to a server. That's not ok.