Ember 0.5.5 released!

I meant to publish this post almost a month ago, but never got around to it. Better late than never though...
I finally did the release of Ember 0.5.5 some days into December.

Among the usual slew of bug fixes and overall improvements one of the major new features is the compass. I've written about that before on this blog, and I'm really pleased with how it turned out. It's currently very basic, but I have all kinds of plans for it in the future. Hopefully it will right now help people getting oriented in the world.
This video shows the compass in action.

Another big change was the upgrade to Ogre 1.6. While it might not result in any immediate improvements to the client, it has a much improved script parsing engine and resource management. The latter is extremely useful for us since we now finally can do away with the complex concept of having separate "user" and "system" media. This separation was forces because of earlier versions of Ogre's inability to correctly handle duplicate materials. Instead of letting the client handle the clashes Ogre threw an exception and totally corrupted the internal resource state. Not so any more though. The advantages of this will better be shown in the next release.

We've also added a lot of fixes and improvements to the entity creator. It's still not completely functional though since it's missing proper model preview functionality, something which will be fixes with the next release of Eris (which the next release of Ember will depend on).
Alexey and Thomas also worked on improving the OSX support, to a point where Ember now correctly builds and runs on OSX. Unfortunately it turned out to be a little harder to create a contained package with all dependent libraries. I'm confident that it will be sorted though.

A new compass and minimap

The last couple of weeks I've been working on getting a working compass and minimap into the client. Since the world is extremely dynamic and not known to the client beforehand it's a little bit more complex than one would think. In a game where the world is known in beforehand, it's simply just a matter of slapping a prerendered or hand drawn texture of the world onto a square polygon and show that as the map. That's not something that we can do however as the world is totally dynamic and can be changed at any time. Our setup is instead a little bit more complex.
We want to render the world as it is, but we don't want to have to re-render it each frame, since that would bring the frame rate down. In addition, in order to get the water line properly rendered we must render all terrain tiles in the highest LOD-level. That means sending a lot of polygons to the gfx card (this could however be alleviated in the future by using shaders).
The current setup is therefore to place a camera far up in the sky, looking down. The camera uses orthographic projection, which means that objects won't be smaller as they are futher away. This is crucial for the map perspective, since else it will look just like a regular camera looking down.
We then render from this camera into a texture, 512x512px. This provides us with a snap shot of the current world, minus some foliage which we don't render. In most cases this looks pretty good. Finally we then define a smaller area, 128x128px, which is what we show in the compass. By panning this area within the larger texture we can scroll the map, and only have to update the render when the smaller area reaches the border of the larger texture. When that happens we re-center the camera and do a new render.
In this way we get a nifty compass map which doesn't have to be rerendered each frame.

Finally we need to provide a small arrow in the middle to show the position the camera is facing. The problem here is that CEGUI currently doesn't support rotation of windows, so the solution here is to leverage Ogre. We start with an arrow image which we then slap onto a full screen quad. This quad is then rotated to fit the direction of the camera and rendered into a texture, which is then sent to CEGUI, made into a gui element and placed on top of the compass. It all works splendidly, but there are some extra indirections which might bring the performance down a little bit.

All in all the compass is working pretty well now. There are some minor issues with it sometimes not being correctly updated when terrain pages change, and the buttons for zooming in and out could be made better, but it's pretty much ready for release.
Currently it only works as a simple minimap, but the plan is to add functionality for showing direction markers for interesting entities. We could probably reuse some of the functionality currently used for showing the labels.

With the compass now in place the 0.5.5 release is pretty much feature complete. I plan to get a beta build out this weekend.

Quickly creating worlds

I spent some hours just trying the world editing features of Ember out. The main reason was that Jayr had provided a couple of really nifty castle wall sections meshes. With section meshes it's possible to build more complex structures, very much like with LEGO. We currently have three sets of section meshes: a wooden fence, a low stone wall and now a larger castle wall.
I first started by just placing the wall segments next to each other directly on the ground. It looked like this:
The main problem here is that the uneven ground makes the segments not match up. It's not looking very good. And if a player would be walking on top of the wall it would look strange indeed.
One of the Google Summer of Code project was however to add support for terrain mods to both the server and the client. A terrain mod is modification to the terrain which will alter it in some way from it's initial, base point generated shape. So in this case the natural terrain mod to use would be a level mod, which makes sure that a certain area of the terrain is leveled. After applying a level mod the wall looks like this instead. Much better. As can be gleamed from the screenshot we have four different kinds of wall sections: a gate, a simple wall segment, a wall segment with a staircase and a corner segment. By combining these it's pretty easy to quickly create a larger wall.
One thing that I found was that it's quite hard to place the wall segments next to each other so that there's not gap, or no overlapping. We therefore need to add some kind of "snap-to" functionality, as specified in this blueprint.
The scaling of the wall segments seems to be a little off though, as this screenshot shows.
After the wall had been completed I added a little path leading up the castle, a small graveyard enclosed by a low stone wall and an inn and a small cottage near the path. The path required some low level entity editing as we don't have any gui for altering areas yet. But adding and positioning the other entities is really, really easy. The cyphesis server doesn't currently have support for loading back entity data that has been persisted (though it has support for writing data to the database), so unfortunately the world got discarded when I shut down cyphesis. The main point of this little exercise was to see how easy it would be to alter the world. There are two areas that need some more love though:

• Editing terrain mods
• Editing terrain areas
  

There's an even greater collection of screenshots to be found here.

Diverse

The last couple of weeks I've been working on many different parts of Ember without. I've started looking into how to get it compiled on OSX, which seems to require some reading up on the OSX build system first. I've also began work on building the latest Ember release for win32. I was hoping that someone else would have picked up the torch, using the win32-dependency package I provided, but so far none has appeared. I'm thinking that there's bound to be some issues with getting openal integrated, but other than that only Mercator has been updated, so it shouldn't be too hard.

I spent some days delving through the code which handles terrain after I found that the terrain was broken when on worlds with non-square sized. The cause of this was improper conversion between the Ogre coord system and the WF coord system. The problem is that the former is screen centric while the latter is world centric. That means that every time we translate any coord between them we need to convert it. However, if we somewhere misses to do this translation, or we're confusing the two, problems will appear. In this case the code was expecting WF style coords, but got Ogre style ones.
A preventive measure that can be taken to avoid these kind of problems in the future would be to more clearly mark what kind of cood type is expected at any place by requiring that whenever possible 2d or 3d structs are used instead of separate values.
It's much more clear what

getHeight(WFMath::Point<2> position)

expects than

getHeight(float xPosition, float yPosition)

Another area I've done some work in is the entity creator. It's now much more easy to work with thanks to the addition of randomized orientation and default values. The former helps a lot when you quickly want to place a large number of entities in the world, as demonstrated in this video.

This video also shows how easily the terrain can be altered. At the start of the video the world is completely empty, with just a flat terrain. Everything that's altered is altered live on the server, so any other client logged in at the same time would see the exact same changes.

The entity creator is getting more and more useful for each day. It can now be used for quickly creating all kinds of more complex structures, such as the enclosure demonstrated here.

The entity types used here aren't part of stock cyphesis yet though.

I've also recorded a little longer stroll through the world. For some reasons however there's quite a lot of stutter, which I think is caused by the massive IO requirements of the video recording app (GLC). When running through the world without any recoding going on I don't get any stutters. It nonetheless shows of some of the richness in the world. The world as seen here is also not the stock world, instead containing some additions I've done on my local box. I'm not sure how this large amount of entities would work on a slower machine.

Sound integration

The work with integrating the GSoC projects into the Ember trunk have continued. I've now merged the Sound Manager changes that Romulo have provided into the main trunk. In the process I had to do some alterations to the code though. While the code provided by Romulo worked and did indeed play sounds it made some assumptions that made it not so suitable for the very dynamic nature of the entity world. Since we can't know beforehand what kind of entities that we need to show, and must be able to dispose of and create new entities as we move through the world we must make sure to provide good resource handling. The changes I made to the way the SoundService works have much to do with how sound sources and buffers are allocated and handled. I've worked with the assumption that not too many sounds will be playing at any time, and that once a sound has finished playing it should be deallocated.
The original code also worked with the assumption that all sounds allocated from ogg files were "streaming" sounds, while all sounds allocated from wav files were "static" sounds (i.e. where the whole sound could be kept in memory). However, the notion of "streaming" and "static" only relates to how large the sound buffer is, not how it's stored (in ogg or wav). Any sound that is too large should be streamed, which means that it should be read from disk in increments. And any sound that is very small could be kept in memory, making it "static".
As I've reworked the SoundService I've temporarily disabled the support for loading ogg sounds, as the implementation needs to be able to handle data recieved from the Ogre resource system. It should however return as I sort out the code.

With the sound support now in Ember trunk I'll start looking at polishing up the code and updating the media for a 0.5.4 release pretty soon.

Summer's end

The Google Summer of Code is now over, and with that the three student projects we've been hosting this year are considered "finished". That that doesn't mean that they are totally complete yet, something which would be hard to say about Ember anytime, but it means that they are complete enough to be merged into the main trunk. This last week I've been busy integrating them into the trunk: so far I've completed both the improved entity creator and the new support for Mercator terrain mods. Next week I'll work on getting the sound system integrated.
The goal is to have a 0.5.4 release of Ember out as soon as possible, with all SoC projects completely integrated.

I've created two small movies showing off the entity creator and the terrain mods. This all thanks to the excellent glc video capturing tool.
The videos can be seen streaming through Google video or in high def format. The streaming versions are a little too low in resolution to really see what's going on, but they give a good overview of the new additions.

• Entity creator: streaming original
• Terrain modifiers: streaming original

Summer and release

I'm currently away on vacation, touring Sweden and visiting relatives, but just before I left I managed to push version 0.5.3 of Ember out. My initial goal for this release was to make sure that all of the nice media that Jayr has provided should come to better use. My plan was to first work on an updated world, a larger piece of terrain which were more suited for the way Ember handles LOD and which also would use more of the media through the use of new entity types. The idea was also that by working on a new world, authoring it through Ember, I'll be better able to find and fix those places where the authoring tools were lacking.
And this plan worked very well. The first thing I enhanced was the terrain editing tool, which previously only allowed one to edit a single base point at once, but now allows you to edit a wide swath of base points at once. This makes it much more easier to alter large expanses of terrain. I've also added an automated function which generates atlas entity type definitions from existing meshes. A lot of the media that Jayr has provided is lacking corresponding entity types, so I needed an easy way to quickly create entity types, using the new meshes as blueprints.

However, when working on the new terrain I quickly realized that Ember wasn't very efficient in the way it handled memory. Since my development box has quite a lot of memory I hadn't really noticed it, but as I now was trying out different very large terrains it became noticable. I thus had to spend some time going through all of the instances in Ember where it either used memory in a wasteful way, or just plain and simple leaked it. In the end I think I managed to squash almost all of the places though, and as a result Ember now doesn't use nearly as much memory as before. A lot of the memory used however is texture and mesh memory, and I need to take a look at how I can talk to Ogre about better free up textures that aren't currently used. There will probably need to be a texture collecting task running at a certain interval to make sure that unused textures are freed.

The work with the memory cleanup took some time, and I had to revisit my original plan of releasing 0.5.3 together with a new world. I do want to do more regular releases, so I didn't want to wait too long between 0.5.2 and 0.5.3. In addition to that, there were new releases of both Ogre and CEGUI which both weren't compatible with 0.5.2. If we waited too long, we would run the risk of more and more people trying to compile 0.5.2 against these latest releases and failing.
And finally I wanted to do a release because it would provide an opportunity for the GSoC students to sync up with the trunk, preventing their trees from lagging too far behind it.

So in the end I pushed 0.5.3. In the rush to get it out I completely forgot to test it with Lua 5.0. The issue was picked up by Oliver Lehmann when creating ports for FreeBSD. Luckily he was able to update CEGUI to also use Lua 5.1 without breaking any other ports. Lua 5.1 has some nice language constructs which we can't use as long as we also provide support for 5.0, and we'll be looking at whether it might be possible to completely drop support for 5.0. 5.1 has been out for quite some time and I think all modern distros provide support for it, so it shouldn't be any problem.

Looking ahead, the Summer of Code is now entering the final phase, and we'll hopefully be seeing working implementations of the student's projects. It's already looking good, and I'm hoping to be able to merge those projects that are finished into the trunk in time for release 0.5.4, which I aim to have ready no later than three months from now.

Memory usage

My development machine is quite beefy and has 4Gb of memory, so I seldom run the risk of running out of memory. The downside though is that I've been quite lax with regards to memory consumtion, and as a result Ember 0.5.2 contains both some memory leaks as well as uses too much memory in the way it preloads all available resources independent of whether they are used or not. The resource preloading hasn't been a problem previously, but since Jayr has been adding so much more cool media, with normal and specular maps, the memory usage for all media is now quite high.
I've therefore the last week gone through the codebase with memory profiling tools to try to find both unnecessary memory usage as well as memory leaks. The main tool I've used is Valgrind, which is really an excellent tool for all kinds of profiling. KDevelop comes with built in support for the memcheck tool, which finds memory leaks. It will keep tabs on all memory that is allocated and find those places where there's no more references left to unfreed memory. The valgrind massif tool is also very handy. It will track the heap of the application, and with regular interval report how large it is, and from where different sections of the heap has been allocated. The massif tool by default outputs its info in the console, which isn't very easy to navigate. I've however found an excellent tool for displaying the data in a much better graphical view over here.
Using these tools I found an issue with how the terrain splatting materials were generated, which in normal use amounted to a memory leak of ~100Mb. Quite a lot of data. Fixing it was trivial.
I also found some issues with how Ember was shut down, and how the main scene manager failed to be deleted. While this doesn't amount to any in game memory leak (the application is shutting down anyway, so unfreed memory will be freed a second later when the process exits) it did clutter the valgring output, and we should of course always strive to have components that correctly handles the allocated resources.
I'm now trying to find the cause of a problem I have with ModelDefinitions not being correctly deleted. They use the Ogre SharedPtr, which makes it possible for them to keep on living a little longer, even though the ModelDefinitionManager has been deleted, as long as something else is holding on to a reference. I'll have to investigate further.

I've also removed the preloading of all assets. This will make sure that only those assets that are needed are loaded, but the main (and quite big) drawback will be that the game will stutter as the player enters and moves through the world. This is because Ember so far is single threaded, and the main thread will have to wait while new assets are loaded as they are encountered. We're planning to move to a multithreaded resource loading solution, but we haven't started on that blueprint yet.

Much of this work is a result of the work I've done in trying to create a new island world. Since the island world is quite larger than the current world the memory usage issue became much more obvious. The plan is to release 0.5.3 with the new world. However, I'm leaning towards perhaps trying to get a 0.5.3 release out without the new world. One of many reasons is that Ogre 1.4.8 which was released after the Ember 0.5.2 release changes how resources with absolute paths are loaded. This clashes with how we load our terrain config file, and the result is that while Ember 0.5.2 compiles without problem with Ogre 1.4.8 it won't be able to run. We've already had some issues with people encountering this problem, and we'll sure have more and more. In addition, Nvidia releases a new driver version not long ago which contains some bugs which will either hang the application at startup or produce messed up mipmaps when used with Ogre 1.4.8 and lower. As a response Ogre therefore did a 1.4.9 release yesterday which contains fixes for these issues. As more and more people upgrade to the newer nvidia drivers we're sure to see more of this issue. I'm therefore thinking about trying to push a 0.5.3 release out as soon as possible. Before I do that I would however very much like to see all major memory issues resolved.
There's currently some kind of memory leak which leaks ~4kb/s when just running Ember, without even being connected to a server. That's not ok.

Terrain shaping up

The last couple of days I've been toying with a new terrain. I'm thinking of an island with some far away mountains and some harbours. I've put up some preliminary screen shots here. With the new terrain editing features it's really easy to quickly create some nice looking terrain. I should really record some movies of it...
When working on that I found a couple of problems in Ember though. The first was that it used massive amounts of memory. A prime source for this seemed to be the fact that it created a lot of both bitmaps and textures for all of the terrain, which it then never really used, but had lying around anyway. That prompted a refactoring of how the terrain worked which removed most of these instances. While doing that I also realized that we're quite wasteful in how we use full 8 bit bitmaps for all of the layers in the terrain, even if one layer might be the snow, where the 256 steps allowed by 8 bits never are used. To improve this we should combine many layers into one bitmap, where perhaps the grass layer is allocated 64 values, while the snow only is allocated 8 values, and so on. This would only work with shaders though.
After refactoring the terrain texture system I found that it still gobbled up memory, but I haven't run any good benchmark yet. I'm however guessing that it's the Mercator system which is initialized in full when the world is loaded, and then never has it's data removed. Preferrably, we should only load the Mercator segments on demand, and we should only keep those most used around and unload the others (perhaps by keeping a time stamped cache). It should work very much like the Ogre PagingLandscape system works.

I also quite alarmingly found that Ember currently is leaking memory. Just by starting it up and having it running without connecting to any server makes it allocate 2-6Kb/s. Something is seriously wrong here. I need to hunt that down and fix it.

When searching for the cause of these things I did some alterations to the logging system, making it not shut it down so early in the shutdown process. I also cleaned up the headers and moved some classes to separate headers. The logging service is included in pretty much every other file in Ember, so making it slim is good for compilation performance. Unfortunately the changes I did to it will force an almost full recompile after updating...

I also ran into a really strange problem with mouse picking in the terrain when testing out my new island. The picking is simply off. It's however working as it should the Mason world, which makes me think that it's either something to do with the fact the the island world is much larger, or that it's in a non-square form. I'll have to investigate it further.

This week also marks the start of the coding period for the students, which is really exciting. I'm looking forward to see how they'll evolve as they more and more into the WF system.

Shaping worlds

For the last couple of days I've worked on the tools for shaping the world. As my goal for 0.5.3 is to have a better world which shows off all the new media the first step is to alter the world to allow for more expansive forests and far away mountains. There's been a terrain editor in Ember for some time, but it has been very rudimentary and lacked some features which would make it easier to alter the world. One of these features was the ability to alter more than one terrain point at once. I've now added the ability to set a "radius" in the editor. That will alter all terrain points around the one that the user interacts with. This is very helpful since it allows us to quickly raise and lower large swaths of land.

When I started fiddling with that I found a couple of bugs in the terrain code which prevented it from working as smooth as I wanted. As I've squashed those bugs the editing is becoming much more smoother. I also found that the way the terrain is reloaded is very inefficient. Currently we unload the whole terrain page and then load it again. However, all that we change is just the height data, so it would be much more efficient to tell the engine just to recalculate the affected meshes from updated data. I've done some changes to the code which allows for this much more efficient approach, and it's beginning to look good. There's still a problem with the LOD calculation not being correctly updated, but that shouldn't be too hard to fix. All in all this will allow for the ability to quickly create a whole new world from a blank slate of terrain. Very exciting stuff.

Building worlds

The start of the Summer of Code is getting closer and closer and our three students have already started getting involved in the community. They've all got their git repos set up, and Alexey which has planned some downtime during the summer has already started committing code.
There's also two pages on the wiki about two of the projects. All in all it looks very promising.

I sat down and thought about what I wanted to see in the next release. It didn't take long to realize that the major focus must be on making sure that all the new media that Jayr has created must better be shown within the game. Just by looking through the commits one can see that he's put a lot of work into the media repository, and it's a real shame that not much of it is shown in the game. This mainly has to do with the fact that there's either no corresponding entity types on the server, or that they aren't used in the default world.
In addition, with Ember 0.5.2 we now have support for imposters, which means that we can display much more entities on the screen at once. This is mainly of use when rending foliage and forests, of which we would like to have much more. Jayr has created a couple of really nice birch trees which can be seen here:
He's also added some bushes as well as a whole collection of new nice meshes and textures. So the focus for Ember 0.5.3 is to update the world to make sure that all of the new media is shown. All other blueprints will be put on hold to favour the push. The first thing we need to do is to update the default terrain. Ember can handle quite large terrains without problems, but have a harder time with really steep cliffs, due to LOD issues. This is pretty much the opposite the current layout of the world. We therefore need to expand the world, make it larger and with more distant mountains. Making it larger will also allow for more expansive forests. In addition, we have quite a lot of meshes for both a village and a castle, so we need to provide space for that as well, just as we also need to make sure that the village and castle look good and are naturally laid out. There's already some maps of a proposed world, as seen here, but the current world isn't following it, and it presents a major issue in that it's hard to do world boundaries, where the world simply ends, without using water. In the current setup there's a couple of very large mountains to two sides, and water on two other sides. I'm thinking of going with an island completely, just because it's so much easier then to have a natural world boundary. There's nothing that says the world needs to be in "Moraf" either.

The first step towards a new world is creating the terrain, so the last couple of days I've been focusing on getting better terrain editing functionality into Ember, as well as finding and fixing some bugs related to the real time terrain editing. We now have the ability to alter many base points at once, allowing us to much quicker and easier alter large expanses of the terrain. The next couple of days I plan on experimenting with a new world. Once that's done we should start on making sure that all meshes and textures are represented by ingame entity types. There's now a nifty export functionality in the Model Editor which will automatically create and export an Atlas definition from a pre-existing mesh. This has already proven to be very useful in quickly generate atlas types for missing media.

Last week I also moved all Ogre material definitions from the Ember git repo to the media svn repo, where they much better belong. The idea here is that they should be used as reference when working with the media, so that it's possible to use the exact same materials both when authoring media and when showing it in Ember. We're not there yet however as we also need a good mesh viewer application that can work with the way the media repository is set up.

Summer getting close

The Google Summer of Code is just now entering into the phase where student allocations are finalized. We've recieved some really nice applications and will be able to announce the final selection soon. This is the first year that I'm involved in the program, but Kai is a real veteran who's been involved with it since 2005. His help has been really valuable.

We've now used git for a couple of weeks and so far it's been a total success. It's sooooo much nicer to work with than cvs. It's a little bit more complex, but in day to day operation it's just a couple of simple commands you need to know. Also, while superficial at first, the web gui is really nice and useful. The ability to do local commits that aren't published before you decide to is immensely useful and really changes the way you work with code. You now don't have to worry about each and every commit not breaking stuff; it's quite ok to commit non-working code as long as the stuff you push to the common repository is working. If only Kdevelop had some better built in git support it would be even better.

With Ember I'm now trying to get a polished 0.5.2 release out. Both CEGUI and libwfut recently released new versions which fixes a couple of bugs we get in Ember, so the new release will require those versions. Jayr has been adding more and more really high quality media. A lot of new normal and specular mapped textures which helps to make the world much nicer. We now only have to update the server to take advantage of all the new media. The fact that there's no persistance and entity editing thus very much depend on editing the define_world.py script on the server makes that process a bit cumbersome though. I've added some support in Ember for saving entities in a serialized form to disk. There's however no way to load them into the world yet, so that's something that needs to be added.

Sean has also provided more patches, adding support for letting the user save the login information for servers, thus removing the need to having to type both user name and password every time one needs to log in to the world (which is something that you do a lot when you're developing Ember). He's also added functionality for totally disabling the gui (bound to the F10 key). Useful for taking screenshots and so on.

Myself I've been fiddling a bit with the minimap. I started out using CEGUI for all elements of it, but since it doesn't support rotation of element I had to abandon that path. I'm currently working directly with Ogre overlays, which should provide the needed flexibility. I'm not there yet, so it's something that won't appear in 0.5.2.

I'm hoping to get the 0.5.2 release out this weekend. Currently it's mainly issues with misconfigured media holding it back.

Summertime is coding time

Worldforge has been accepted to the Google Summer of Code, which is really excellent in many different ways. Not only will we now get a student working on the code while being paid by Google, we will also get some nice publicity. I've already seen a sharp increase in visits to our sites after Google announces the projects that were approved two days ago. More information about this can be found in the wiki. We've also set up a new forum dedicated to the program.
For prospective students it's worth noting that the project ideas put forth in the wiki page are just suggestions. As such we really encourage the students to either expand on the suggested projects or to submit new projects. The forums are perfectly suited for such discussions.

Other developments lately have been that Sean Ryan has checked in his first code contribution. He's been working on adding logging of in game chat messages to text files in the user's ember home directory, something which might be handy for seeing what's transpired in the world. His code was solid and functional which always is good to see. Getting into the code can be quite daunting since there's so many different sub components used, but he's managed to navigate it all, with a little help.

We've also now completed the conversion from cvs to git. While the cvs repo will remain for some time, all new development now happens through git. Kai has done all the heavy lifting with setting up the server and all. A very nice web interface of it all can be found here.
In order to get the latest development version you should first do

git clone git://git.worldforge.org/ember.git ember

which will checkout the latest code to a new "ember" directory. To keep in sync with the latest changes you would then do

git fetch

More information here.

All over the place

The last couple of weeks I've been all over the code base, working on different things each day it seems. I've upgraded Caelum to a newer version, fresh from svn. It gets better and better. New in this one is nicer sunsets and sunrises, but the same code which make that so nice also produces a little to large halo during the day. But overall it looks nicer and nicer.
I fixed some long standing but rare bugs, such as a crash when you had deleted an entity with the entity editor and wanted to look at another. I also fixed a very spurious problem where sometimes chat messages wouldn't appear in the console. Turned out that the lua garbage collector sometimes destroyed the connection object which binds chat messages to the console output. Garbage collected languages can be very nice, but when there's a problem like this it's really hard to track down due to the nature of the unpredictability of the destruction.

I also spent some days restructuring how media is handled. This is mainly a problem for other people wanting to help with the development. Since it's only been me for quite a while it's been quite easy: my box contains the latest dev media. But as more people get interested we need a better way of distributing the development media. In contrast to the release media this is very much in flux and is often updated. The new solution is to use a rsync service at amber which always will contain the latest media. By using the new make target "devmedia" it's possible for other developers to get the latest media by simply typing

make devmedia

and rsync will take care of it all. Furthermore I've removed the configure variable MEDIA_VERSION in favour of getting that information from ember.conf instead. This makes it much easier to configure and won't require a total recompile every time the media version changes. This is just one move towards a better media system. I've already added some blueprints on how I would see the media system getting even more dynamic.

The icon rendering works nice and all when there's FBO support on your GPU, but when there's not there are a couple of issues. The current way of solving these is to delay the copying of the render information one frame and then blitting it to system memory and then to the icon atlas texture. However, the Ogre::Viewport class has the ability to render into just a smaller section of a larger texture. This would be perfect for the icons, since then we could render directly into the icon atlas. So I implemented that, only to find out that there's either a bug with how it works, or that I've totally misunderstood it. The problem is that every time a full frame render occurs, this seems to mess up the whole icon atlas texture, and renders junk data into all sections that don't have a viewport assigned to them. It seems the problem is that it either gets repainted by mistake, or that it's intended behaviour. I need to research it more closely, but for now that functionality is deactivated in favour of the old way of rendering icons.

It seems that I with regular intervals goes through the code looking for places where I can refactor it to better fit with updated behaviour, or simplify structures and methods that works like a cludge. Just basic cleanup. I spent one day doing that too, adding some nifty methods and classes to the configuration service as well as adding a new time service which can be used to both get local and server time. Should come in handy.

I've also evaluated both git and monotone. They both seems to have pretty much the same functionality, but I just like the feel of git better, so I'm gonna migrate all of Ember into that. We'll still keep the old cvs as read only, but all new development will happen in git.

Currently I'm working on adding an overhead map. It's good progress, but there are some different approaches to it. I can use CEGUI and do pretty much all of it with CEGUI elements. Or I can use Ogre overlays. Or use the Ogre compositor system. There's pros and cons with all of these, so currently I'm just getting a feel for it. There should be a working version in some days though.

Oh, and today is the day that Google announces the projects that it has accepted for the Summer of Code. This year we applied, so hopefully we've gotten a slot. But you never know.

Reeds

With the new terrain foliage system it's now very easy to add new layers of foliate by just altering the .terrain xml file. Yesterday I set out to try this by adding some reeds to the world. We've had textures for reeds for a while, done by zzorn, but since there hasn't been a good system for adding them to the world they've been unused so far. Adding them now was however trivial. We want the reeds to grow by the water, and only there. They are also very much like grass, so they should use the same rendering technique as the normal grass. Adding them to the world was thus as simple as editing the moraf.terrain file so that the sand layer definitions looks like this:

<layer shadername="sand" diffusetexture="3d_objects/environment/ground/textures/sand/sand_AD.png" normalmaptexture="3d_objects/environment/ground/textures/sand/sand_AN.png" tilesize="8">
<foliage planttype="grass" populationtechnique="cluster" rendertechnique="grass">
<param key="minClusterRadius">2</param>
<param key="maxClusterRadius">10</param>
<param key="clusterDistance">25</param>
<param key="density">2</param>
<param key="falloff">0.6</param>
<param key="material">/global/plants/grass/ducktail/single</param>
<param key="minHeight">2.0</param>
<param key="maxHeight">2.5</param>
<param key="minWidth">1.0</param>
<param key="maxWidth">1.5</param>
<param key="swayLength">0.25</param>
<param key="renderTech">quad</param>
</foliage>
</layer>

This tells the terrain system to add foliage to the sand layer, and the sand layer only. It also tells it to use the "cluster" population technique for placement, and the "grass" technique for rendering, using the "ducktail" material. The other parameters should be pretty self-explanatory. The end result can be seen here:


Not too bad. The main point here though is how easy it is to alter and add new foliage layers. Worth noting also is that we used the built in cluster populating technique, but this would be a prime candidate for using Mercator Areas. By using areas, the world designers would have much greater control of where to put the reeds, and it could also be possible for players to affect them in the world, for example by using a scythe to cut them down.

Yesterday I also committed some performance improvements to the foliage engine, mainly by using batch stores for the sub areas, thus avoiding having to iterate through all plants for each geometry page.
I stumbled upon a bug in the entity editor which caused a crash when deleting entities in the world. It was however an easy fix (don't hold on to deleted entities...) which I will commit later today.

New skies opening up

I've spent the last couple of days updating the version of the sky system Caelum that Ember uses. We've been using an almost two year old version, and there's been some nice recent improvements to it. It turned out to be fairly easy to do, some classes and interfaces had changed, but all in all it was a straight forward update process. The end result is a better looking world, if I might say so myself. The sky model has been refined, and support for clouds and sky transitions is more smooth and robust. One interesting addition is that it now uses an astronomically correct earth model, where both earth position and absolute day must be specified. This opens up some nice possibilities for virtual worlds, since it then is possible to specify where on the earth sphere your world should be placed, and get a different lighting model. A world placed at the north pole would have a very different day-night cycle than one placed at the equator. Alternatively, the server shouldn't assume that we're on an earth sphere and could instead just specify direct values for night and day cycle, as well as the sun's trajectory over the sky. That would require us to create our own solar system model classes to be used by caelum, but it's certainly doable. But there's a certain simplicity to just letting the server specify the longitude and latitude that's quite alluring.
Another issue is how to present the world during night. In the current caelum implementation it just gets darker, until you can't pretty much see anything. This isn't very nice for players, and would probably force them to always manually set the time to noon to be able to see something. Instead we want to provide a world that's as clear during the night as it is during the day. WoW does this splendidly, and we need to adapt something similiar. But until then we'll use the current hack, where the time of day on the client always is adjusted so it's sometime around noon, even when it's in the middle of the night on the server.

Another thing that's I've committed is some updates to the PagedGeometry system where it now uses classes specifically tailored for Ember, which renders near objects as entities while switching to imposter rendering for far away ones. Previously we had to use the DummyPage class and use the Entity::renderDistance functionality for hiding far away entities, but now that's all smoothly taken care of by classes which work directly with the PagedGeometry engine. There are still some issues with both performance and updates that needs to be taken care of, but it works much better now.

On the Worldforge front, there's two new exciting developements which has unfolded the last couple of weeks. One is that we now have a new Ember developer, Sean Ryan, who's just starting out getting a feel of the Ember code base. He's already taken upon himself to implement event logging as an initial task which is an excellent way to start.
The other development is that we're applying for the Google Summer of Code this year. We've already set up a wiki page for this purpose and will gladly accept any applicants or project ideas. If you think this is something for you, or if you know someone that might be suited, please get in touch with us, preferrably by using the mailing lists.

The grass is greener

Work on implementing a better foliage system has continued, and I'm getting more and more pleased with the results. Here are some recent screen shots.
As with all other functionality in Ember I've strived to make it as general and modifiable as possible. It should be trivial to both change the behaviour and to extend it. The way it's implemented is through the use of the terrain layers as a base. This is how the xml definition for the grass terrain layer looks like (I've removed some texture definitions):

<?xml version='1.0' encoding='UTF-8'?>
<terrain>
 <layers>
  <layer shadername="grass">
   <foliage planttype="grass" populationtechnique="cluster" rendertechnique="grass">
    <param key="minClusterRadius">2</param>
    <param key="maxClusterRadius">10</param>
    <param key="clusterDistance">25</param>
    <param key="density">2</param>
    <param key="falloff">0.3</param>
    <param key="material">/global/plants/grass/bittergrass/single</param>
   </foliage>
   <foliage planttype="fern" populationtechnique="cluster" rendertechnique="shrubbery">
    <param key="mesh">3d_objects/plants/shrubs/models/fern/fern.mesh</param>
    <param key="minScale">0.7</param>
    <param key="maxScale">2.0</param>
    
    <param key="minClusterRadius">1</param>
    <param key="maxClusterRadius">3</param>
    <param key="clusterDistance">30</param>
    <param key="density">2</param>
    <param key="falloff">0.7</param>
   </foliage>
   <foliage planttype="camelia" populationtechnique="cluster" rendertechnique="shrubbery">
    <param key="mesh">3d_objects/plants/flowers/models/camellia.mesh</param>
    <param key="minScale">2</param>
    <param key="maxScale">5</param>
    
    <param key="minClusterRadius">2</param>
    <param key="maxClusterRadius">5</param>
    <param key="clusterDistance">40</param>
    <param key="density">1</param>
    <param key="falloff">0.7</param>
   </foliage>
   <foliage planttype="leaves" populationtechnique="cluster" rendertechnique="shrubbery">
    <param key="mesh">3d_objects/plants/shrubs/models/leaf/leaves.mesh</param>
    <param key="minScale">0.5</param>
    <param key="maxScale">1.2</param>
    
    <param key="minClusterRadius">2</param>
    <param key="maxClusterRadius">5</param>
    <param key="clusterDistance">35</param>
    <param key="density">2</param>
    <param key="falloff">0.7</param>
   </foliage>
  </layer>
 </layers>
</terrain>

Every foliage type has both a population technique and a render technique. The first one determines where to put the vegetation and the latter determines how it should be rendered. Both of these share the same set of arbitrary parameters. We currently only support the cluster population technique. This creates a series of clusters of varying sizes randomly dispersed over the map. In most cases this looks quite nice, since plants tends to grow in clusters in the nature. All of the plants positions are calculated when the terrain is loaded, and not changed thereafter. However, every time a plant is placed we check all the layers that are above the current layer. If the combined value of all the layers above are above a threshold value we don't show the plant. This has the effect that roads, fields and other areas don't get vegetation, unless there's foliage defined for that particular layer. And since we've already determined the positions for the plants we can update foliage swaths as areas are added without having the positions of existing plants getting jumbled each time.

Regarding the render techniques we currently support two different: grass and shrubbery. The former autogenerates grass meshes, and animates it whereas the latter uses preexisting meshes. So far the latter haven't got support for animations, but that's something that we'll surely add.

The population techniques are implemented in Ember so far, but the idea is to put them in Mercator as they stabilize. That way it should be possible to specify foliage on the client, very much like we now specify Mercator shaders.

All in all it's looking mighty promising, even though as can be seen in the screen shots we need to fix up the colours a little.

Impostering

I've now finally implemented the Paged Geometry engine into Ember. This is an component which allows for much more nicer looking foliage through a variety of different techniques. While we've had grass through the use of batched geometry for a while, the implementation wasn't really optimal and it didn't contain the more advanced features needed. The main thing lacking has always been a proper implementation of imposters, which are 2d billboards used in the distance. Instead of burdening the GPU with full geometry for far away trees, which in the end will end up just taking up a few pixels on screen, we render the tree once to a 2d sprite and reuse that. The end result is an enormous speed up for forests scenes, where there can be thousands of trees on the screen at any one time.
All of this is provided by the Paged Geometry engine, which is why I'm very excited to finally have it in place. So far I've partially converted the grass system to use the new engine. It's currently lacking some of the filtering features, which makes the grass appear on paths and other areas where there should be no grass. It's also just randomly dispersed, which makes it look a little bit unnatural. It would preferably be dispersed in a more cluster like fashion, to better simulate the way grass grows in the real world. All of this is currently being addressed however.
The trees now use billboards when viewed from afar. I've extended the Model format to take an additional "rendering scheme" subsection. This allows one to specify different rendering schemes for different models. Thus we can tell Ember to use the rendering scheme "forest" for our trees, which will set up everything and make them switch to imposters at far away distances, while keeping the regular entity rendering at close distance. The Paged Geometry engine allows us to use batched geometry up close, which means that the different entities are "baked" into one large mesh. However, due to the dynamic nature of Ember and the Worldforge world that's not feasible. Thus we need to render the entities normally up close. The current solution is to do that using the regular rendering system and then set max render distance to the same distance as that when imposters take over. But since the imposters work in groups the distance where a certain entity begins to be rendered by imposters isn't exact. This means there are instances where this leads to either both the imposter and the entity being rendered at the same time (not a big problem) or that none gets rendered (a bigger problem). I therefore need to extend the Page Geometry engine with some Ember specific classes which will allow it to smoothly transition between normal entity rendering and imposter rendering.
I've already made some refactoring changes to the Page Geometry, which I intend to try to push back into the main development branch.
I've put up some more screen shots of the imposters and grass over at amber.

On a different note, there have recently been some issue with the Worldforge infrastructure. I've been trying to contact Balinor about the media svn for some time, but have never gotten any response. Last week however the svn server completely broke down: when trying to update I get a message about an internal error. A quick google shows that it stems from the server running out of disc space. This is bad, since not only can no one commit to the repository, no one can even update their local copy against it. I therefore sent some emails to the mailing lists, only to find that they also have been broken. Apparently there was a purge of all the old inactive lists last week, something sorely needed. Instead of having 15+ different lists, we now only have 4, which hopefully will mean that messages won't be lost. Unfortunately this changes seems to have broken the lists, since none of my mails got to the lists. I've been trying to contact the people responsible this weekend to no avail, but hopefully it will be fixed soon.

Grass

The last weeks have been really productive for Ember. I've closed most of the bugs reported in the bug tracker and also managed to add some new features. One that I've been needing for some time was a teleport function for admin mode. Previously when you were logged in as admin you had to move just as a regular person, which could take some time when you wanted to go to a different area of the map. But now there's a nifty "teleport here" option available when you click on the world, which will instantly transport you to the clicked location. It was very easy to implement, no more than 30 minutes work.
Another thing that has been bugging me for a while was that when you entered and exited building your orientation would be messed up. I initially assumed that there was something wrong with how the server sent updates to the client, but after some thorough testing I found that it was solely a client issue. The problem was that when the user controls the avatar and walks through the world, Ember ignores orientation and position updates sent for the avatar. This is to avoid the issue on lagging connections where the avatar continuously will snap back to a previous position, as sent from the server. When the avatar stops moving it will be adjusted to the data sent from the server however.
The problem here was then when the avatar moves into a house and changes it's container, it needs to be have it's orientation updated relative to the orientation of the new container. And this new orientation is correctly sent from the server, however not as I incorrectly assumed in the onLocationChanged event, but in the subsequent onMoved event. But if the user is moving the avatar Ember ignores the onMoved event and the old orientation will be used instead, resulting in the avatar looking in the completely wrong direction. The solution was to change the functionality so that the onMoved event would be honoured by Ember for the avatar even when the user was moving it, if it occurred right after a onLocationChanged event. This fixed the problem, something which I think has been in Ember for some years now.

A new feature I've been working on this week is support for a better foliage system. I've so far used a home grown system which uses static geometry, but it has some problems and I didn't want to put much effort into it since there are other third party components for Ogre which does it all much better. Especially the Paged Geometry plugin shows real promise. So the last couple of days I've worked on integrating that into Ember, throwing out my own solution. It's a very good library, excellent in both design and documentation. So far I've had no problems getting it integrated, but I think I need to do some alterations to it in order to support the demands of Ember, especially the dynamic nature of the terrain. So far I've focused on getting the grass to work, but the idea is to also use if for the trees and other objects that are both numerous and fairly static. This will hopefully bring the ability to display really vast forests.

Catching up

I spent some time getting the Ember win32 binary done. Previously I've done it in MSVC and had to employ all manners of hacks to the code in order to get it to compile, but this time I wanted to do it with GCC (no need for code alteration) and in MSYS (no need to create separate projects for all components since the standard autotools setup will work).
Though I managed to get a couple of components built in MSYS I in the end had to settle for a hodge podge of msys, Code Blocks and precompiled libraries. Turns out it's not too easy to build stuff the *NIX way on Windows, shockingly enough!
In the end I got a working Ember binary which is now released. The good news is that now that I've got a build environment set up on Windows the next release should be much easier to squeeze out.

Now that I've got that done I've moved on the working on actual Ember features again. There's been some really great activity on the Launchpad with nice bug reports being filed. Before I start on some serious new features in Ember I'm thinking that I should at least get the new bugs fixed. Most of them are quite easy to handle, but there's a couple of problems with the way the icons are handled. One is that when switching between full screen and the desktop on Windows the icons will get messed up. This is quite straight forward, since the way the icons are done is that they are rendered and then blitted to a larger GPU texture. That way we can keep the number of render state changes down and make it a little quicker. The downside is that it's all happening on the GPU, and when switching back to the desktop the GPU releases all of its resources, and expects them to be restored by the app when the user switches back. There's however no such functionality in Ember for repopulating the texture, thus it ends up with uninitialized data. This is not that big a deal since there are a couple of ways to solve this: either by always keeping an in memory copy of the texture, or by repopulating it when requested.

The other problem is that our blit function won't work on GPU's that don't support Frame Buffer Objects. This is a little bit harder to tackle. The main problem is that rendering happends asynchronous. The app sends render ops to the GPU which queues them and then process them in order. So normally the GPU is a couple of frames behind the main app. However, this means that once we make our call from Ember to render the icon view, we might have to wait a couple of frames until it actually gets rendered. Now, right after we've rendered our icon we then want to blit it to our main icon texture. On cards which supports FBO the blit operation is just another render op, so it will be put on the render queue and happen after the rendering has taken place.
But on cards that don't support FBOs Ogre has to fake this by copying the data itself. And this is where there's a problem, since if we try to copy the data right after we've issued the render call the GPU will most probably not have had time to perform the render, and we'll end up with garbage data (which is exactly what happens).
So the solution is to implement a mechanism where we render the icon, wait a couple of frames and then blit it, thus making sure that the icon actually is rendered. I'm working on such a solution currently and it's looking promising. I've had to alter the structure of the icon rendering framework a little, but thankfully it's quite modular so it's no large alteration.

One thing that I really need to take care of it updating the example world of Mason. Mainly to make it use all of Jayr's new media. He's been adding a lot of really high quality media, which unfortunately isn't shown since the default Mason world haven't been updated with new entities. Both me and Al are quite busy adding new features to the code, so neither have had time to do this, and I was hoping that someone else in the project would be willing to go over it.
Another thing that needs altering is the layout of the world. Ember really likes large vistas with none too sharp height differences, since the LOD will then work best. Unfortunately the current world is pretty much the exact opposite: a small world with steep mountains. If we could move the mountain away quite a distance and make the elevation not as steep I think the world would both look and perform much better.

Windows, yeah that's fun

After spending the holidays away from almost all things Ember I'm back again. I did manage to do a little Ember work since it turned out that the .package file I uploaded to Sourceforge had become corrupt: that's what you get for claiming it from the upload pool before it has finished uploading. Wasn't harder than reuploading though.
There's been some really promising work done over Christmas however mainly by Alex Torkhov who have contributed greatly to the project through mainly the Launchpad. Our hope was that the Launchpad would make the project more transparent and provide a more technical forum for bug reports and suggestions, and it makes me happy to see that it's being used in that way.
Also, there's been some nice additions to the wiki by Alabandit, mainly concerning an introductory guide to Ember, something I've never had time to do.
All in all some promising new additions.

When I did the 0.5.1 release I only had time to do a precompiled version for Linux. I was hoping that someone could be able to help me with creating a precompiled win32 binary, but in the end I have to roll up my sleeves and do it myself. It's not a pretty job. But I've made a promise that I won't do any further Ember development until there's a binary version available for those using Windows.
Previously I've used MSVC for compilation, but it means I constantly have to hack the wf libs Ember just to make it compile. So instead I'm working on getting a fully working build environment set up in Windows through MSYS. It's getting there, but it slow as hell. Just running configure can take up to 10 minutes. I don't know what's making it so slow, and I have no interest in trying to hunt it down, as long as it works. But then there's all kinds of wonderful stuff, such as Windows case-insensitive naming system, as well as the inability to separate data from file system location ("This file is locked by another process." Yes, but would it be too hard to tell me which process is locking it?). And so on. It's just a tedious work.

And I have so many ideas for what I want to do next in Ember, but all of that will have to wait until I've conquered Windows.