Ember development blog

Properly shaded foliage

2011-06-08T23:12:00.000+02:00

Ember has a system for generating grass and other kind of foliage. It looks pretty good in most cases, but the shading of the grass has always been off. This is even more apparent when the time of day changes, as the grass doesn't change colour when the world does. In addition, the grass didn't receive any shadows, making it look strange when running in a higher graphics settings.

All this has changed now however, as I've altered the grass to be both properly lit and to receive shadows correctly. The effect can be seen below.

Ember 0.6.1 out of the door

2011-05-24T21:07:00.003+02:00

I finally managed to get Ember 0.6.1 out of the door.
It seems to be common procedure for me to never be able to get the releases out in accordance to my plans, but this time it took even longer than would seem "normal".
The two main reasons for this were Windows and 32-bit architecture, both legacy systems (hurr hurr hurr).
I've always striven to make Ember cross platform. This can be seen in the components used and the way its internals are laid out. The main problem is that I'm not too fond of touching Windows unless I absolutely have to, and therefore the Windows releases have lagged behind the Linux ones. Fortunately for us a contributor in the Worldforge community, James Lovejoy, has been working on streamlining the process of building Ember for Windows, and with his help we were set on releasing a working Windows binary along with the 0.6.1 release.
It turned out however that some of the low level networking stuff we were doing in the Eris library didn't play to well with Windows. Ok, cue a rewrite in Eris by Alistair, some testing of that, and we got it fixed after a while.
The other issue was more subtle. I've added some mechanisms for making sure that Ember is running on a fixed fps rate (60 by default). This relies on getting and comparing millisecond timestamps from the OS. Problem here was that I was using the C++ "long" type, which on 64 bit machines (such as the one I'm developing on) is 64 bits, but on 32 bit machines is 32 bit. And 32 bits are too few, since the timestamps will then wrap around after something like a month's time. Not good. And this was of course something I didn't discover until I basically had already made the release and was testing our 32-bit precompiled binary out (we're now using AppImage since Autopackage is defunct).

Just some snags on the way, but enough to delay the release a little bit too long.

Currently we've just started up the coding part of Google Summer of Code. This year I'll be mentoring two students: Martin and Peter. They're already at work improving Ember, with some exciting weeks ahead of us all.

Oh, and here's a small movie showing how the Model Editor can be used.

Worldforge in Google Summer of Code 2011

2011-03-20T11:10:00.005+01:00

Worldforge has been accepted into Google Summer of Code 2011!

We've had a remarkable experience with the program from before and are very excited about this opportunity. If you're a student which are interested in working on a cool Worldforge project during the summer, don't hesitate to visit our GSoC page. The sooner you start getting involved in the project the better, so don't be shy about making yourself heard on the mailing lists or irc.

Simplify tolua++ with autotools

2010-06-19T14:44:00.006+02:00

In Ember we use Lua for our scripting needs. The bindings to the C++ parts of the client are provided by the tolua++ library. Tolua++ works by generating C++ source code from .pkg files, which are simplified .h files. This works out extremely well; the developer only needs to copy the .h file to a similarly named .pkg file and remove those methods and fields that shouldn't be exported. The command line tool "tolua++" is then run to produce the C++ source.

Previously in Ember we handled all of this by keeping the generated C++ source checked into the source, and requiring that each developer had to run the tolua++ command to regenerate these sources each time a lua binding was added or changed. This however had a couple of downsides, chief amongst them that the code generated differs a bit depending on the version of tolua++ used. It also resulted in some often very large Git commits of generated code, which tended to pollute the Git history.

A much better solution would be to instead automate the generation of the binding source, and bake it into the normal build system. Fortunately the Autotools already provides all the facilities for making this happen.

An example of how this is handled in Ember though a Makefile.am can be found here:


SUFFIXES: .cxx .pkg .lo .la .cpp .o .obj

.pkg.cxx:
 cd $(srcdir) && TOLUAXX=${TOLUAXX} $(abs_top_srcdir)/scripts/update_lua_bindings.sh `basename $@ .cxx` `basename $@ .cxx`.pkg $(abs_builddir)/`basename $@` $<

INCLUDES = -I$(top_srcdir)/src -I$(srcdir) -I$(top_builddir)/src -DPREFIX=\"@prefix@\"

noinst_LIBRARIES = liblua_EmberServices.a
liblua_EmberServices_a_SOURCES = EmberServices.cxx

CLEANFILES = EmberServices.cxx
TOLUA_PKGS = ConfigService.pkg EmberServices.pkg IInputAdapter.pkg Input.pkg InputService.pkg LoggingService.pkg MetaserverService.pkg ScriptingService.pkg ServerService.pkg
EXTRA_DIST = $(TOLUA_PKGS)
EmberServices.cxx: $(TOLUA_PKGS)

The files references in TOLUA_PKG are the .pkg files which define the bindings. These will be fed through the update_lua_bindings.sh script to generate the file EmberServices.cxx, which is then compiled and added to the liblua_EmberService.a archive. Note that we need to add EmberServices.css to the CLEANFILES variable to make sure that it's deleted when we're cleaning up. Since it's generated through the tolua++ tool Automake can't keep track of it itself.

The update_lua_bindings.sh script looks like this:


 #! /bin/sh
 #tolua++ will for some reason translate "const std::string" into "const,std::string", so we need to remove these incorrect commas from the final code
 #some versions will also for some unexplainable reason not correctly remove the tolua specific directive tolua_outside, so we need to clean that out also
 #We'll also replace the inclusion of "tolua++.h" with our own version which has better support for building on win32.
 echo "Updating lua bindings."

 #If the TOLUAXX environment variable isn't set default to using the command "tolua++".
 if [ x${TOLUAXX} = x ]; then
 TOLUAXX=tolua++
 fi
 ${TOLUAXX} -n $1 $2 > $3
 grep -q '** tolua internal error' $3 && cat $3 && exit 1
 sed -i -e 's/const,/const /g' -e 's/tolua_outside//g' -e 's/tolua++\.h/components\/lua\/tolua++\.h/' $3

This script basically runs the tolua++ command, as defined in the TOLUAXX environment variable (with "tolua++" as fallback) and then applies some replacement to fix some issues we've been having with the generated code.

The TOLUAXX environment variable is set at configuration time. The default is "tolua++", but we'll provide the option to use an alternative command. Our acinclude.m4 file has this snippet (which is called from configure.ac):


AC_DEFUN([AM_CHECK_TOLUAXX],
[
 AC_ARG_WITH(tolua++,AS_HELP_STRING([--with-tolua++=CMD],[Tolua++ command (default=tolua++)]),
    toluaxx_command="$withval", toluaxx_command="tolua++")

 AC_CHECK_TOOL(TOLUAXX, $toluaxx_command)

 if test "x$TOLUAXX" = "x"; then
  AC_MSG_ERROR([Could not find a working tolua++ command (tried '$toluaxx_command'). Use the --with-tolua++ switch to set the proper command to use.])
 fi
])

This setup will allow the script bindings to be generated at compile time, but only the first time compilation occurs, or if any of the .pkg files have changed. More examples of how this is used can be found in the Ember sources.

Using automake to generate ChangeLog from git

2010-06-15T14:05:00.003+02:00

When deploying a GPL application it's required that you provide a change log containing information on all the changes to the code base. This file is normally named ChangeLog.
In the olden days, before today's fancy revision control systems, people often had to edit the ChangeLog files themselves for each and every change to the code. This is what we did in the Worldforge project when we used CVS for our source code needs.

However, for the modern developer used to distributed revision control systems such as Git this seems archaic. Why provide a separate list of all changes when the Git repository already contains a complete log? Instead of keeping the ChangeLog file updated with each commit, wouldn't it make more sense to generate it from the repository history when a new release is made?

When a new release needs to be created in a project using the autotools the "make dist" make target is invoked (or in reality the "make distcheck" target). This will package the source and produce tar archive of it all. The Makefile target "dist-hook" is provided to allow for us to hook into this process. So what we want to do is to provide some shell scripting which generates a ChangeLog file from the git history. It will look something like this:

cd ${top_srcdir} && git log --stat --name-only --date=short --abbrev-commit > ${distdir}/ChangeLog

This will generate a log of all the changes, in a condensed format not unlike the ones suggested by GNU.

We could be done here, but there are some things that could need improvement. For one thing, this will only work when the "make dist" target is invoked in the original git directory. Some distributions might want to take the dist release, add some distro specific patches, and then make a new dist from that. It would therefore be better to add some logic which can recognize whether there already exists a generated ChangeLog, and if so won't try to generate a new one from the git log. This will need some more logic so we'll be splitting this functionality out to a separate script and call that from the dist-hook target. We'll also add a ChangeLog with exactly one line:

This file is autogenerated from the Git history when a the "dist" make target is invoked. If you find this file in an official release something has gone wrong and you should contact [maintainer_email]. It needs to be exactly one line long in order for the ChangeLog generating script to work.

Our script will check the length of the ChangeLog file. If it's exactly one line, we know we should generate it from the git log, else we know it's already been generated and we shouldn't do anything. Our script (generate-ChangeLog.sh) will look something like this:

#! /bin/sh
top_srcdir=$1
distdir=$2
if [ `cat ${distdir}/ChangeLog | wc -l` = "0" ]; then
chmod u+w ${distdir}/ChangeLog && cd ${top_srcdir} && git log --stat --name-only --date=short --abbrev-commit > ${distdir}/ChangeLog
fi

The dist-hook will now look like this:

dist-hook:
sh $(top_srcdir)/generate-ChangeLog.sh $(top_srcdir) $(distdir)

Now we're pretty set. With Worldforge there's however still an issue with the older CVS ChangeLog. When the source was migrated from CVS to Git the log messages from CVS were directly imported into Git. These however were in a quite verbose format with the date and authors included. Since this data also is available as meta data in the Git repo, the result will be that these log entries will have a lot of redundant data, making the ChangeLog both very large and confusing. To prevent this, we'll store a copy of the old ChangeLog with the CVS entries with the source (as ChangeLog-CVS) and combine this with the Git log history. After looking at the Git history we can see that at the commit with id "f12012e7616c191a8926432faf866c8e43854062" marks where the transition from CVS to Git happened. We'll also replace the ChangeLog-CVS with a notice about it's previous use (as it's not needed anymore, but must be present in the dist) Our script will then look like this:

#! /bin/sh
top_srcdir=$1
distdir=$2
if [ `cat ${distdir}/ChangeLog | wc -l` = "0" ]; then
chmod u+w ${distdir}/ChangeLog && cd ${top_srcdir} && git log f12012e7616c191a8926432faf866c8e43854062..HEAD --stat --name-only --date=short --abbrev-commit > ${distdir}/ChangeLog && cat ${top_srcdir}/ChangeLog-CVS >> ${distdir}/ChangeLog
chmod u+w ${distdir}/ChangeLog-CVS && echo "This file was needed for generating the proper ChangeLog as an aggregate of the code held in git and older code in CVS. It's now empty, but needs to be included in the source distribution to not upset automake." > ${distdir}/ChangeLog-CVS

fi

And now we're done. We have an automated system which will generate the ChangeLog from a combination of both the old CVS provided ChangeLog and the Git log history. It will also make sure that if the "make dist" target is run again on an already generated dist the ChangeLog will be left as it is. Just remember to also include all relevant files in the EXTRA_DIST target of the Makefile.am.

Up to date versions of how this is used in Ember can be found in the Ember repository.

Freedom of Speech

2010-05-20T11:00:00.000+02:00

Today I'll be posting about something other than FOSS game development, something far more important (yes, such a thing exists!). Today's topic is Freedom of Speech and how we never must take that for granted.

I live in Sweden where the last couple of years there has been a couple of incidents where islamic fascists have tried through violence and threats of violence to impose their world view on other people. Central to that world view is an active aggression against the principle of Freedom of Speech which underpins our whole democratic and parliamentary system. The most prominent examples are the threats and active murder attempts on the Danish cartoonists who drew pictures of Muhammed, or the threats made against South Park. In Sweden the artist Lars Vilks has been both physically assaulted and subjected to arson as a result of his drawing of Muhammed in 2007.

It's easy to take the Freedom of Speech for granted and to think that those people that want to bring it down only operate in chaotic countries such as Pakistan or Somalia. That is unfortunately not the case, as shown in this video of Lars Vilks presenting a lecture on the topic of "Art and Freedom of Speech". Vilks suffered some bruises and a broken pair of glasses. The day after someone tried to kill him by setting his home on fire.

Do not be tempted to draw the conclusion from this video that Sweden, or any other European country is under any threat of "being overrun by Islam" or similar. That is absolutely not the case; it is in fact the opposite. When faced with a stable society and a high standard of living most people tend to distance themselves from religion and totalitarian ideologies, even if it might take a generation or two.
But it does show that Freedom of Speech is something we never can take for granted, and not something we can barter with. It is the firmament on which our whole society is built on, and we cannot let fascist or totalitarian ideologies or groups make us infringe on it one iota. As long as people feel threatened by violence for expressing their opinions there is no real Freedom of Speech. For every Lars Vilks who has the guts to criticize Islam and as a consequence gets assaulted and beaten, there are thousands of other people who hold their tongue. Some of them in Western countries such as Sweden, but most of them in countries where Freedom of Speech already is curtailed, such as Iran or Pakistan.

Therefore I'm today celebrating the annual Draw Muhammed Day by presenting my own drawing of Muhammed, the founder of Islam. By providing my own drawing of Muhammed I'm showing my support and solidarity with the artists currently under death threats. Note that the drawing is pretty tame (and badly drawn); I know this blog is syndicated so I'm putting a little restraint on myself. To more properly show my complete and total outrage of having islamic fascists trying to impose their warped totalitarian world view on me it would be more proper to show Muhammed being raped by a pack of dogs. But I'll wait with that picture for another day (not that the Internet is lacking in such depictions).

Muhammed, shown here during one of his schizophrenic episodes trying to come up with another verse for the Qur'an.

Ember 0.5.8 released

2010-05-14T11:26:00.002+02:00

I've just released Ember 0.5.8. The official release announcement can be found on the main Worldforge site.

The main new feature is the multi threading framework which I've written about before on this blog. It's a pretty Big Thing, as it we're so utterly dependent on having dynamic data.
Another thing which went into this release, but which won't be apparent for any user, is a large restructuring of the media repository. It's mainly Jayr's work, but I've also helped out. Basically it amounts to defining common naming and structuring standards for the media repository so that it becomes easier to both contribute and use the assets therein. I've updated some of the media guidelines to reflect this.
And for the first time in a couple of years we now provide win32 binaries. Getting Ember to build on win32 (through msys) was a real chore, but the plan is to keep the code base compatible enough so that each subsequent release of Ember will have win32 binaries.

There's been some other developments in the FOSS game world too.
Ryzom recently released all of their source code under the GPL and all of their media under the CC-BY-SA license. This is a huge boost for FOSS gaming, as there's now a boatload of new media assets available. A quick comparison of the Ryzom media contra the Worldforge media:
* Ryzom has a much more assets.
* Ryzom textures are smaller (between 256 and 512 pixels whereas most of Worldforge's are 1024 - 2048 pixels).
* Ryzom textures are only available as png whereas Worldforge provides source assets for all textures (usually .psd).
* Ryzom only provides diffuse textures whereas Worldforge usually provides specular and normal/heightmap versions too.
* Ryzom assets are made in a particular art style, whereas Worldforge has opted for a natural style. This is however just for the Mason world. I would love to see a world using Ember and Cyphesis with the Ryzom assets.

Some have noted that the source meshes in Ryzom only are available in .max format. That's no big deal though as they can be converted to .blend (a process which is always underway). As long as the source is available it's always possible to convert it; it's harder when the only available asset is an exported format. (Worldforge provides .blend files for most of the meshes, though we have some .max versions too.)

Another big thing is the Indie Humble Bundle, which is a collection of six excellent indie games, available for a user set price. It's truly an excellent deal, especially since part of the money paid is donated to charities. And not only that, since they so far received more than 1 million USD they've now opened up the source code for four of the games as FOSS. It's really a remarkable thing.

New animals from Jayr

2010-04-01T10:14:00.002+02:00

This is a short post about some of the new media recently added by our extremely talented artist Jayr to the Worldforge media repository.
Jayr keeps his own blog (unfortunately missing rss) over at Hourglass3d where he posted about some of the new animals he added to Worldforge.
Head over there and take a look.

GSoC student applications are now open

2010-03-30T11:24:00.003+02:00

Google has now opened the student applications for Google Summer of Code 2010.
This year, in order to help you figure out if you have the skills needed to work on Ember, and in order to help us to filter out non-serious applications, we ask students who are applying for ember to find and fix two errors in this special code drop. Also see the Ember prerequisites on this.
We encourage students to apply as soon as possible, so that there's time for us to give feedback and work out the details. The sooner the better.
If you have any questions please use the general mailing list.

Worldforge in Google Summer of Code 2010

2010-03-22T15:02:00.002+01:00

Worldforge has for the third year in a row been accepted into the Google Summer of Code 2010 program. This means that students now have an opportunity where they can spend the summer writing code for free and open virtual worlds rather than flipping burgers and mowing lawns.
Student's applications will open on March 29, but if you are interested in applying the sooner you contact us the better.
We prefer that students use the general mailing list for discussing project ideas, or send mail to the mentors directly.
More information about the program, and a list of project suggestions can be found at the Worldforge GSoC page. Note that the suggestions are just that; suggestions. Any interested student is encouraged to also build on the suggestions, or propose their own project.

New world

2010-03-10T12:30:00.002+01:00

For quite some time we've had our default world Mason. While Worldforge is a project aimed to provide basic tech for any kind of virtual world, we still need to have a showcase world in which we can try out different things. The default world has had some issues however.

It's not using much of the new media provided by Jayr, and the media used is often outdated.
It's confusing for newcomers. There's not clear direction of what new players should do in the world.
The steep mountains surrounding the area are hard to manage in a client, as the steep cliffs makes level-of-detail optimized rendering hard.
It's a quite small world.

All of this is changing as I've been working on a new default world. There are a couple of improvements to both the server and client which allows for this.

Persistence on the server. As cyphesis is a work in progress, it's possible that bugs are uncovered when developing the world which will cause a crash. Having the whole world disappear when the server crashes is quite unproductive. With constant database persistence there's no risk of that.
The ability to take a snapshot of the world on the server. This snapshot is saved as an xml file, which can then be added to a versioning system. This way it's possible to try things out, and revert to older versions if needed.
Freezing the world on the server. When designing the world it's important that it doesn't change. That means that trees shouldn't grow, animals shouldn't move, living things shouldn't starve. Through a new feature which allows the server to be frozen any world author can first design the world, and once that's done unfreeze it.
Advanced entity editing on the client. It's important that it's easy to edit all aspects on the fly on the client. The advanced entity widgets available in Ember allows for this. Aspects such as terrain areas or terrain modifications are easily editable through a point and click interface.
Easy entity creation. Just as important as editing entities is making sure it's simple and quick to create new ones. The Entity Recipe system in Ember allows for entities to be created with point and click, with a powerful scripting environment in the backend.

The new world is therefore created in a series of steps. The first step involves working with a frozen world, where all entities are placed, and all terrain is sculptured. Once the world is complete it needs its npcs and creatures to be given behaviour and goals. This should be done using the existing python scripting in cyphesis. Once that's done the world should be available as both an xml file defining all entities and an accompanying python script file which defined behaviour.

When creating the new world there are a couple of goals I've had in mind.

Use as much of the new media as possible, while clearing out any out of date media (i.e. media which simply doesn't look so good any more).
Provide a large enough world that it will take some time to traverse it.
Make the terrain friendly to any clients. That means avoiding too steep cliffs as that brings havok to any LOD mechanism.
Provide a clear path for any new player. There should never be any confusion of what the player is supposed to do once he or she has entered into the world.
Provide new game play. Apart from our current pig herding and fishing game play we have many ideas of additional tasks. These tasks also require a larger terrain, and new media.

So far things have gone very smoothly. Working with a much larger world is also good for Ember since it really pushes the engine in terms of memory usage and performance. Our current world is simply too small to really take advantage of the many dynamic paging features in Ember.

Below are a couple of screenshots of the work in progress. Some of them show the new starting area, which is on a peninsula with a clear path to the main area. Together with a couple of helpful signs and a greeting npc it should be much more clearer for any newcomer where they should go.

Adding proper multi threading

2009-12-22T11:04:00.002+01:00

The last couple of weeks I've been working on adding proper multi threading support in Ember. This builds on the work Manuel did during the 2009 Google Summer of Code.
Basically there are two goals with adding better multi threading:
1) Make Ember seamless, so there aren't any stuttering when large pieces of data is processed in the main thread.
2) Leverage the way modern CPUs today come with multiple cores, making the client faster overall.

The first goal is the most pressing. Since Ember is totally controlled from the server, and nothing about the world is known beforehand, situations often occur where as the user moves through the world new data is presented which requires extensive processing. The best example of this is the terrain and the way new terrain features are streamed to the user in real time.

When researching how best to implement multi threading two approaches presented themselves. Either I could devise a system where each subsystem had a dedicated thread, and then make sure that these subsystems communicated with each other in a safe way. Or I could try to identify smaller pieces of functionality which could be run in parallel and define tasks for these.
The first approach would solve the main issue, i.e. making sure that the main rendering thread works uninterrupted, but wouldn't help with the second goal. There's only so many subsystems to work with, and in some years time it won't be uncommon with 16+ cores. Instead I opted for the second approach, which is to try to identify "tasks" which then can be designed to be executed in separate threads. This approach puts more demand on the ability to identify tasks, and to design them so that the data they operate on is safely separated from other threads. But the benefit is that it's now possible to better utilize multiple cores.

For this purpose I've implemented a simple task execution framework in Ember. The first target of improved multi threading support using this framework has been the terrain system. Unfortunately our terrain library, Mercator, is not designed at all to allow multiple threads to access it, so currently I still have to make sure that only one thread at a time is interacting with Mercator. But this interaction now always happen in a thread separate from the rendering thread. As the terrain changes settle I'll be moving on to other parts of Ember which also will benefit from multi threading.

Ember 0.5.7 released!

2009-10-21T20:44:00.002+02:00

I've just pushed the 0.5.7 release of Ember. More information here.

This release includes a lot of new nifty authoring features, such as snap-to movement. We've also put some effort into getting it to run correctly (with all nice shader effects) on ATI cards. And there are lots of speedups, especially in the terrain handling code.

The multi-threading work done during this year's Google Summer of Code isn't included in this release because of a couple of regressions; it will hopefully be merged in soon though.

Easier authoring

2009-10-03T18:48:00.003+02:00

The latest addition to Ember is a new authoring mode in which all entities in the world are easily selectable through graphical representation. While most entities in the world normally are represented by a graphical mesh, not all are, and not all meshes are easily selectable. This is most obvious when trying to alter terrain areas and mods, since these in most cases don't have any graphical representation at all. Previously one then had to either know the id of the entity and access the editor for it through the console, or use the entity browser to find it.
With the new visualizations this becomes much more natural, since it's now only a matter of selecting the entity in the game.

Another new addition is a feature for easier selecting entities obscured by other entities. The need for this arose from changes on the server for how the ocean is handled. Previously it were solely a client effect, putting a transparent plane in the world to represent the ocean. Now it's however a entity just like any other. This however presents a problem when picking things that are in the ocean, since the normal operation then would be for Ember to always report that the ocean was picked. The solution for this is a new feature in Ember which allows multiple "mouse pick results" to be presented. If any of the entities picked is transparent (such as the water) Ember will also look for any entity beyond the transparent entity. Through a simple paging mechanism in the mouse picker widget the user can then flip through the picked entities and select the one she wants to interact with.
I think it works pretty well, but I'm a bit biased so I would love to get some feedback from other users about this feature.

Major refactoring

2009-08-05T20:13:00.002+02:00

I've just pushed a major refactoring I've been working on the last couple of weeks to the master repo. In the end it turned out really well.
The main part of the refactoring was to totally revamp the whole entity hierarchy in Ember, and how the entities are tied to graphical Ogre representations (meshes, lights, particle systems etc.). As with most refactorization, I had made some assumptions about how things worked a couple of years back which didn't hold anymore, and the code was beginning to creak from workarounds and special cases.
One of the assumptions made was that all entities in the entity hierarchy tree would be represented as an Ogre scene node. Another was that all graphical entities would be shown using the Model class. As one of our GSoC students, Amey, began to implement his task and needed some client extension for this it became apparent that these assumptions do not always hold. It's not always certain that an entity will be arranged in the Ogre scene graph through the use of a scene node. The most common instance is when something is attached to a skeleton (like a person wielding something). In addition, there are many kinds of entities which need graphical representation, but not through the use of a Model. In Amey's case it was the ocean.
Furthermore, when I first made the entity class system I used inheritance, so that each different entity would be of either just a normal EmberEntity class, or a PhysicalEmberEntity class, depending on whether it had a graphical representation in the world. However, as I work more and more with software I've come to realize some of the issues with depending too much on inheritance. Instead I now more often find myself component based design. In such a system the behaviour of instances are mainly determined by the interactions between the components that make up them. This of course leads to more complex systems, where it's a bit harder for a newcomer to get the full picture of how any given part should work. But it also leads to much more powerful systems, without many of the incorrect inheritances one tend to end up with when over using inheritance.

All this lead to me totally redesigning the entity class system. There's now only one entity class (well, except for the world entity, but that is a truly unique instance). The way something is shown is now handled completely separate in a graphical component, owned by the entity. And the way it's arranged in the Ogre scene graph is also handled by another distinct component. The end result is that there's now a much more robust system in place which more correctly maps how things really are arranged. This means that we now have a much more cleaner design, which is also extensible enough to handle more advanced cases than the previous system could. I've already pushed code which will connect an ocean entity to a water representation, something which in the old system would have required all kinds of hacks.

ATI graphics fixes

2009-06-12T11:48:00.002+02:00

The last month saw both the release of Ember 0.5.6 and the start of the Google Summer of Code.
The Ember release included the really nice addition of real time shadows, thanks to the work done by Alexey. It also included some really nifty area editing tools for world authors, as well as the usual slew of polish and bug fixes.
After the release was done Alexey and me did some additional work on the shadow system, to fix some of the bugs and artifacts that existed in the release. Amongs them was some issues with shadow texture fidelity at certain distances and a bug where lights that were created in the world never disappeared. Another issue we found was that lights which were attached to entities which a character was wielding did't get updated as the character moved. The prime example here was the torch, which in turn had a "fire" child entity.
All of these issues has now however been fixed in git master.
I've also added a new nifty snap-to-move feature. It's activated when you move an entity in the world, and allows you to easily make it snap to another entity. The snapping is done by matching the bounding boxes of the entities. This is really useful when building fences or walls, since each wall section then can snap into the other.

However, the bulk of my time the last couple of weeks has been spent on graphic glitches, mainly those found on ATI cards. We've known for a little while that there are a couple of artifacts on ATI cards, but since none of the main developers use ATI cards (we use Nvidia, since at least historically their Linux support has been orders of magnitudes better than ATI) we haven't really known how bad these artifacts are.
After upgrading to Mandriva 2009.1 I began to run into X server segfaults when running OpenGL applications. I therefore had to abandon my Nvidia card until the issue has been fixed, and it just so happened that I was able to borrow an ATI cards from work.
Turns out there were all kinds of issues with ATI cards, amongs them the issue with random segfaults within the OpenGL driver. Just this issue had me spend more than a week trying to track down. Intially it wasn't clear that the crashes were random, since they seemed to occur near certain materials. However, after disabling more and more features it became apparent that they were indeed completely random. Thanks to cdleonard, developer of the Caelum environment system, we concluded that the issue was that using hardware generated mipmapping on ATI cards, as supplied by the SGIS_generate_mipmap extension, resulted in random memory corruption withing the (closed source) GL stack. Yep, pretty much the worst possible kind of bug to track down. Once we did track it down it down and disabled the crashes went away.
There are however still some artifacts on ATI cards. The paged geometry has some issues, since it in places mixes vertex shaders with fixed function fragment processing. And the high detail glsl shaders doesn't produce any output, so when running Ember in high detail level all meshes are hidden when viewed up close. But all of these issues are easily fixable and it shouldn't take long until we can do a new release, which will work much better on ATI cards.

The Summer of Code is also progressing nicely. Manuel reports on the progress over at the wiki. We're now mainly in the discovery stage, where we're identifying where in the code base it would be suitable to add threading support. Nonetheless Manuel has already submitted a series of patches for various code cleanup issues which I've merged into Ember trunk. Git is truly a wonderful tool!

Realtime correct shadows

2009-04-25T00:03:00.003+02:00

Finally, Ember has complete real time shadows. It took some time, and it wasn't until Alexey some months ago dove into it that we finally got a working implementation. But not only did he add shadows to the world, he also added a complete graphics level framework, where Ember will itself detect the kind of graphics card the user has and tune the graphics accordingly. This means that if you have low end card you'll still be able to run the client at playable frame rates, while if you have a newer card you'll also get real time shadows and other nice effects. One of the reasons we couldn't do this before was that it wasn't until Ogre 1.6 that we got easy access to the techniques needed to do shadows on the larger scale that our world presents (it's a bit harder to do outdoor shadows than shadows in an enclosed space).
Alexey did most of the work, and I got in at the end and fixed some issues with how the shadow system differentiated between semi-transparent and opaque materials. It turned out that Ogre doesn't have functionality where you can specify a template shadow material per technique (for example one shadow material for the "High" technique and another one for the "Medium" technique) and then have Ogre automatically supply all of the material settings from the regular material (alpha settings, textures to use, etc.). It can do that if you specify a scene manager general shadow material, but if you specify a shadow material per technique it expects you to provide all of the information needed in that material. Given the amount of materials we use that's not a viable solution; it's much more easier if we used a material template which would for each shadow pass get the correct data set. Thanks to the modular design of Ogre this wasn't hard to do, as can be seen in this commit. The result is both working shadows and a material setup that's both simple and flexible.

I've also put some work into improving the way the minimap for the compass is rendered. Previously it used the same lightning as the regular world, which mean that it was completely dark during the night. That's been fixed now, so it's always rendering it with a nice midday light. I also made it use a simpler material. This fixes some stuttering which presented itself when you moved rapidly through the world, as well as fixed the issue with ugly shadows appearing on the minimap.

A demonstration of the correct shadows, the runtime graphical settings changes and the new minimap lightning is shown here.

And a higher quality version is to be found here.

Using terrain mods to level ground for buildings

2009-03-21T19:32:00.003+01:00

One of my pet peeves for quite a while is that most of the buildings in the game stand on uneven ground and have the terrain poking through their floor. This is an effect of the way we use Mercator to create a dynamic terrain, so that it's never guaranteed to be any level area in it.
And that's where the terrain mods come into play. By using terrain mods we can alter the terrain at specified areas. There are a couple of different types of "mods" we can apply, and for this problem the "level mod" seems most suited. It will make sure that the terrain specified is altered so that's it's leveled to a specified height.
There are two ways of specifying this height. Either we use a fixed height, telling the system for example that the area should be leveled to a height of 20 meters. Or we use a offset height, so that the height specified is relative to the position of the entity to which the terrain mod is attached. For this case the latter is more suitable, since we want the terrain to be leveled to the same height as the entity.
The clip below shows how this can be done in real time on the server. We first create a house, which when positioned on uneven ground will have the terrain poking through the floor. By adding a level terrain mod, with an offset positioning of 0 meters, the terrain will be altered so that it's level under the house. As we move the house, the terrain mod is moved as well.

Higher quality version here.

The level mod has one disadvantage however in that it will both raise and lower the terrain to the specified height. In this case, we would want it to lower the terrain, but not always raise it. It would be nice then to have floor functionality, so that terrain that is higher is lowered, but lower terrain is unaltered. The current terrain mods available don't allow for this yet however. If we would implement it it would also need a change in the terrain mod interface in the Mercator library, since the TerrainMods currently doesn't know about the existing terrain.

Switching topics, I'm glad to announce that for the second year Worldforge was accepted to the Google Summer of Code. The last year we had three students who all performed splendidly, and the whole program was a great experience. I'm really looking forward to this year. We're now in the phase where we're accepting students' applications. If you're a student which would rather spend your summer getting paid for writing FOSS code for virtual worlds than turning hamburgers or digging ditches, please take a look.

Completed terrain area editing

2009-03-09T19:07:00.003+01:00

Initially I planned to have gotten the terrain area editing completed some weeks ago. However, when I started looking into the existing entity editor code I found that it was lacking in some areas, mainly the structure which made it cumbersome to add new adapters and to have it be easily extendable and handle suggestions of both attribute names and attributes types. So I spent a couple of days refactoring the entity editor to turn it into a better shape. It paid off in the end, since it's now much easier to extend it. And the lua code, which was quite haphazardly written, is now much better.
The main visible improvement is that we now have the ability to suggest an attribute name, not just a type. This will help with editing, since any author then won't need to keep track of the names of the important attributes.
The new terrain area adapter allows you to add new areas to any entity, and to edit any existing area. You can add or delete new points, and you can alter the layer which is to be used.
I've put together a small movie of this in action. In it I show how you easily can create a new entity (in this case a barrel), add an area to it, alter and edit the area, and then remove the entity from the world.

Higher res version over here (be sure to try the HQ version).

The nice thing about the area editing is that it's implemented as a general feature, so now that we have support for editing terrain areas, it should be easy to also add support for editing terrain mods, which in much of their functionality are similiar to terrain areas (the former alters the heightmap while the latter alters the terrain layers).

Graphical area editing

2009-02-16T20:07:00.002+01:00

I've committed a new feature to Ember, which is graphical area editing in real time. I've always felt that it was something that was missing and was crucial for easy world editing. A lot of what makes up the features in the terrain is based on areas. It's a really nifty feature which allows us to add all kinds of different details to the world. There are also some ideas of expanding them to be used for server side foliage definitions, as well as integrating the area code with the terrain modifier code.
However, editing areas have always been cumbersome, even with the improved entity editor in Ember. Basically the areas are made up of polygons, which are defined using a list of points. Editing an area thus involves manually altering the individual points, which is really time consuming and hard to do.
The new area editing feature presents a nice graphical way of altering the areas through simple dragging and dropping of points. The current implementation only allows for alterations of existing areas, and you cannot add or remove points. That's all to come however, but the current behavior is already a vast improvement over the previous functionality, as can be seen in this movie.

High quality version available on YouTube.

When creating the code for the placement and movement of the nodes I could reuse a great deal of existing placement code in Ember, much thanks to the way it previously was written to be generic. It's always nice to see some of the earlier hard work paying of in these ways.

Unified terrain mods

2009-01-12T21:19:00.003+01:00

Since this summer's Google Summer of Code we've had support in both Ember and Cyphesis for terrain modifiers. The server side implementation was however incomplete and didn't support all of the shape types available. Furthermore, Sear didn't have any support for terrain mods.
When working on providing a more unified functionality for both the clients and server it quickly became apparent that we needed to push some of the functionality in Ember down to lower level libraries in order to avoid code duplication.
The first step was to extend WFMath to have atlas parsing and generating methods for the more complex shapes. Previously only primitives such as Position and Vector had such methods. One of the advantages of this is that all clients and servers won't have to bother themself with the format of the atlas data, since that's taken care of by WFMath.
Another issue was that the actual code for connecting terrain modifications to the entities to which they belonged was duplicated in both Ember and Sear. The main issue was that it would normally go into the Eris library, but Eris didn't have any dependency on Mercator. That was easily fixed by adding such a dependency, after which the bulk of the duplicated code could be moved into Eris.
The end result of all this is that we now finally have correct and unified support in both Cyphesis, Ember and Sear for all available terrain mods and the shapes used by them. One of the most useful of the mods are the level mod, which makes sure that an area of the terrain is levelled. This is extremely useful when placing houses and other structures, because you want to make sure that the house sits on level ground. If not, you run the risk of having ground poking through the floor, or the foundation of the house seemingly resting on air.
In the following movie you will see a house to which a rectangular polygon shaped level mod is attached. The level mod doesn't have any specific height set and will therefore use the height of the entity itself.

Higher quality can be found here.

Improved shadows

2009-01-11T23:02:00.003+01:00

Alexey has spent some time lately improving both the shader and shadow support in Ember. The shaders for both the terrain and the regular meshes have been reimplemented in GLSL and have had a nice slew of improvements added to them. One of the most exciting feature is proper support for real time shadows.

One of the issues we've been having is that regular texture shadows tends to result in very low res and jagged shadows when used with a large scene such as our outdoor scenes. This is because the texture that is used can only be so large, and the outdoor scenes makes it hard for the shadow system to setup the most optimized camera setup for the shadows. However, thanks to the new support in Ogre 1.6 for PSSM shadows, which basically is a technique where multiple shadows maps are used for variable distances, we now have the option of having shadows that looks good both up front and in the distance.
Alexey has already done a great job in adding support for this, which can be seen in these videos showing off the shadows. Note that they are still a work in progress, so some things such as taking texture alpha into account or shadowing of normal meshes are not yet implemented.

Some more videos can be found over at YouTube.

There are still some work to be done, but the shadows are already looking mighty nice. Of course, lower end machines won't be able to take advantage of this, and one of the things that we need to work on is better support for fall back materials.

Ember 0.5.5 released!

2008-12-09T16:46:00.003+01:00

I meant to publish this post almost a month ago, but never got around to it. Better late than never though...
I finally did the release of Ember 0.5.5 some days into December.

Among the usual slew of bug fixes and overall improvements one of the major new features is the compass. I've written about that before on this blog, and I'm really pleased with how it turned out. It's currently very basic, but I have all kinds of plans for it in the future. Hopefully it will right now help people getting oriented in the world.
This video shows the compass in action.

Another big change was the upgrade to Ogre 1.6. While it might not result in any immediate improvements to the client, it has a much improved script parsing engine and resource management. The latter is extremely useful for us since we now finally can do away with the complex concept of having separate "user" and "system" media. This separation was forces because of earlier versions of Ogre's inability to correctly handle duplicate materials. Instead of letting the client handle the clashes Ogre threw an exception and totally corrupted the internal resource state. Not so any more though. The advantages of this will better be shown in the next release.

We've also added a lot of fixes and improvements to the entity creator. It's still not completely functional though since it's missing proper model preview functionality, something which will be fixes with the next release of Eris (which the next release of Ember will depend on).
Alexey and Thomas also worked on improving the OSX support, to a point where Ember now correctly builds and runs on OSX. Unfortunately it turned out to be a little harder to create a contained package with all dependent libraries. I'm confident that it will be sorted though.

A new compass and minimap

2008-12-01T14:22:00.003+01:00

The last couple of weeks I've been working on getting a working compass and minimap into the client. Since the world is extremely dynamic and not known to the client beforehand it's a little bit more complex than one would think. In a game where the world is known in beforehand, it's simply just a matter of slapping a prerendered or hand drawn texture of the world onto a square polygon and show that as the map. That's not something that we can do however as the world is totally dynamic and can be changed at any time. Our setup is instead a little bit more complex.
We want to render the world as it is, but we don't want to have to re-render it each frame, since that would bring the frame rate down. In addition, in order to get the water line properly rendered we must render all terrain tiles in the highest LOD-level. That means sending a lot of polygons to the gfx card (this could however be alleviated in the future by using shaders).
The current setup is therefore to place a camera far up in the sky, looking down. The camera uses orthographic projection, which means that objects won't be smaller as they are futher away. This is crucial for the map perspective, since else it will look just like a regular camera looking down.
We then render from this camera into a texture, 512x512px. This provides us with a snap shot of the current world, minus some foliage which we don't render. In most cases this looks pretty good. Finally we then define a smaller area, 128x128px, which is what we show in the compass. By panning this area within the larger texture we can scroll the map, and only have to update the render when the smaller area reaches the border of the larger texture. When that happens we re-center the camera and do a new render.
In this way we get a nifty compass map which doesn't have to be rerendered each frame.

Finally we need to provide a small arrow in the middle to show the position the camera is facing. The problem here is that CEGUI currently doesn't support rotation of windows, so the solution here is to leverage Ogre. We start with an arrow image which we then slap onto a full screen quad. This quad is then rotated to fit the direction of the camera and rendered into a texture, which is then sent to CEGUI, made into a gui element and placed on top of the compass. It all works splendidly, but there are some extra indirections which might bring the performance down a little bit.

All in all the compass is working pretty well now. There are some minor issues with it sometimes not being correctly updated when terrain pages change, and the buttons for zooming in and out could be made better, but it's pretty much ready for release.
Currently it only works as a simple minimap, but the plan is to add functionality for showing direction markers for interesting entities. We could probably reuse some of the functionality currently used for showing the labels.

With the compass now in place the 0.5.5 release is pretty much feature complete. I plan to get a beta build out this weekend.

Quickly creating worlds

2008-11-02T17:24:00.002+01:00

I spent some hours just trying the world editing features of Ember out. The main reason was that Jayr had provided a couple of really nifty castle wall sections meshes. With section meshes it's possible to build more complex structures, very much like with LEGO. We currently have three sets of section meshes: a wooden fence, a low stone wall and now a larger castle wall.
I first started by just placing the wall segments next to each other directly on the ground. It looked like this:
The main problem here is that the uneven ground makes the segments not match up. It's not looking very good. And if a player would be walking on top of the wall it would look strange indeed.
One of the Google Summer of Code project was however to add support for terrain mods to both the server and the client. A terrain mod is modification to the terrain which will alter it in some way from it's initial, base point generated shape. So in this case the natural terrain mod to use would be a level mod, which makes sure that a certain area of the terrain is leveled. After applying a level mod the wall looks like this instead. Much better. As can be gleamed from the screenshot we have four different kinds of wall sections: a gate, a simple wall segment, a wall segment with a staircase and a corner segment. By combining these it's pretty easy to quickly create a larger wall.
One thing that I found was that it's quite hard to place the wall segments next to each other so that there's not gap, or no overlapping. We therefore need to add some kind of "snap-to" functionality, as specified in this blueprint.
The scaling of the wall segments seems to be a little off though, as this screenshot shows.
After the wall had been completed I added a little path leading up the castle, a small graveyard enclosed by a low stone wall and an inn and a small cottage near the path. The path required some low level entity editing as we don't have any gui for altering areas yet. But adding and positioning the other entities is really, really easy. The cyphesis server doesn't currently have support for loading back entity data that has been persisted (though it has support for writing data to the database), so unfortunately the world got discarded when I shut down cyphesis. The main point of this little exercise was to see how easy it would be to alter the world. There are two areas that need some more love though:

There's an even greater collection of screenshots to be found here.

Diverse

2008-09-29T17:00:00.002+02:00

The last couple of weeks I've been working on many different parts of Ember without. I've started looking into how to get it compiled on OSX, which seems to require some reading up on the OSX build system first. I've also began work on building the latest Ember release for win32. I was hoping that someone else would have picked up the torch, using the win32-dependency package I provided, but so far none has appeared. I'm thinking that there's bound to be some issues with getting openal integrated, but other than that only Mercator has been updated, so it shouldn't be too hard.

I spent some days delving through the code which handles terrain after I found that the terrain was broken when on worlds with non-square sized. The cause of this was improper conversion between the Ogre coord system and the WF coord system. The problem is that the former is screen centric while the latter is world centric. That means that every time we translate any coord between them we need to convert it. However, if we somewhere misses to do this translation, or we're confusing the two, problems will appear. In this case the code was expecting WF style coords, but got Ogre style ones.
A preventive measure that can be taken to avoid these kind of problems in the future would be to more clearly mark what kind of cood type is expected at any place by requiring that whenever possible 2d or 3d structs are used instead of separate values.
It's much more clear what

getHeight(WFMath::Point<2> position)

expects than

getHeight(float xPosition, float yPosition)

Another area I've done some work in is the entity creator. It's now much more easy to work with thanks to the addition of randomized orientation and default values. The former helps a lot when you quickly want to place a large number of entities in the world, as demonstrated in this video.

This video also shows how easily the terrain can be altered. At the start of the video the world is completely empty, with just a flat terrain. Everything that's altered is altered live on the server, so any other client logged in at the same time would see the exact same changes.

The entity creator is getting more and more useful for each day. It can now be used for quickly creating all kinds of more complex structures, such as the enclosure demonstrated here.

The entity types used here aren't part of stock cyphesis yet though.

I've also recorded a little longer stroll through the world. For some reasons however there's quite a lot of stutter, which I think is caused by the massive IO requirements of the video recording app (GLC). When running through the world without any recoding going on I don't get any stutters. It nonetheless shows of some of the richness in the world. The world as seen here is also not the stock world, instead containing some additions I've done on my local box. I'm not sure how this large amount of entities would work on a slower machine.

Sound integration

2008-09-02T10:31:00.001+02:00

The work with integrating the GSoC projects into the Ember trunk have continued. I've now merged the Sound Manager changes that Romulo have provided into the main trunk. In the process I had to do some alterations to the code though. While the code provided by Romulo worked and did indeed play sounds it made some assumptions that made it not so suitable for the very dynamic nature of the entity world. Since we can't know beforehand what kind of entities that we need to show, and must be able to dispose of and create new entities as we move through the world we must make sure to provide good resource handling. The changes I made to the way the SoundService works have much to do with how sound sources and buffers are allocated and handled. I've worked with the assumption that not too many sounds will be playing at any time, and that once a sound has finished playing it should be deallocated.
The original code also worked with the assumption that all sounds allocated from ogg files were "streaming" sounds, while all sounds allocated from wav files were "static" sounds (i.e. where the whole sound could be kept in memory). However, the notion of "streaming" and "static" only relates to how large the sound buffer is, not how it's stored (in ogg or wav). Any sound that is too large should be streamed, which means that it should be read from disk in increments. And any sound that is very small could be kept in memory, making it "static".
As I've reworked the SoundService I've temporarily disabled the support for loading ogg sounds, as the implementation needs to be able to handle data recieved from the Ogre resource system. It should however return as I sort out the code.

With the sound support now in Ember trunk I'll start looking at polishing up the code and updating the media for a 0.5.4 release pretty soon.

Summer's end

2008-08-24T20:40:00.002+02:00

The Google Summer of Code is now over, and with that the three student projects we've been hosting this year are considered "finished". That that doesn't mean that they are totally complete yet, something which would be hard to say about Ember anytime, but it means that they are complete enough to be merged into the main trunk. This last week I've been busy integrating them into the trunk: so far I've completed both the improved entity creator and the new support for Mercator terrain mods. Next week I'll work on getting the sound system integrated.
The goal is to have a 0.5.4 release of Ember out as soon as possible, with all SoC projects completely integrated.

I've created two small movies showing off the entity creator and the terrain mods. This all thanks to the excellent glc video capturing tool.
The videos can be seen streaming through Google video or in high def format. The streaming versions are a little too low in resolution to really see what's going on, but they give a good overview of the new additions.

Entity creator: streaming original
Terrain modifiers: streaming original

Summer and release

2008-07-17T15:53:00.002+02:00

I'm currently away on vacation, touring Sweden and visiting relatives, but just before I left I managed to push version 0.5.3 of Ember out. My initial goal for this release was to make sure that all of the nice media that Jayr has provided should come to better use. My plan was to first work on an updated world, a larger piece of terrain which were more suited for the way Ember handles LOD and which also would use more of the media through the use of new entity types. The idea was also that by working on a new world, authoring it through Ember, I'll be better able to find and fix those places where the authoring tools were lacking.
And this plan worked very well. The first thing I enhanced was the terrain editing tool, which previously only allowed one to edit a single base point at once, but now allows you to edit a wide swath of base points at once. This makes it much more easier to alter large expanses of terrain. I've also added an automated function which generates atlas entity type definitions from existing meshes. A lot of the media that Jayr has provided is lacking corresponding entity types, so I needed an easy way to quickly create entity types, using the new meshes as blueprints.

However, when working on the new terrain I quickly realized that Ember wasn't very efficient in the way it handled memory. Since my development box has quite a lot of memory I hadn't really noticed it, but as I now was trying out different very large terrains it became noticable. I thus had to spend some time going through all of the instances in Ember where it either used memory in a wasteful way, or just plain and simple leaked it. In the end I think I managed to squash almost all of the places though, and as a result Ember now doesn't use nearly as much memory as before. A lot of the memory used however is texture and mesh memory, and I need to take a look at how I can talk to Ogre about better free up textures that aren't currently used. There will probably need to be a texture collecting task running at a certain interval to make sure that unused textures are freed.

The work with the memory cleanup took some time, and I had to revisit my original plan of releasing 0.5.3 together with a new world. I do want to do more regular releases, so I didn't want to wait too long between 0.5.2 and 0.5.3. In addition to that, there were new releases of both Ogre and CEGUI which both weren't compatible with 0.5.2. If we waited too long, we would run the risk of more and more people trying to compile 0.5.2 against these latest releases and failing.
And finally I wanted to do a release because it would provide an opportunity for the GSoC students to sync up with the trunk, preventing their trees from lagging too far behind it.

So in the end I pushed 0.5.3. In the rush to get it out I completely forgot to test it with Lua 5.0. The issue was picked up by Oliver Lehmann when creating ports for FreeBSD. Luckily he was able to update CEGUI to also use Lua 5.1 without breaking any other ports. Lua 5.1 has some nice language constructs which we can't use as long as we also provide support for 5.0, and we'll be looking at whether it might be possible to completely drop support for 5.0. 5.1 has been out for quite some time and I think all modern distros provide support for it, so it shouldn't be any problem.

Looking ahead, the Summer of Code is now entering the final phase, and we'll hopefully be seeing working implementations of the student's projects. It's already looking good, and I'm hoping to be able to merge those projects that are finished into the trunk in time for release 0.5.4, which I aim to have ready no later than three months from now.

Memory usage

2008-06-17T13:37:00.002+02:00

My development machine is quite beefy and has 4Gb of memory, so I seldom run the risk of running out of memory. The downside though is that I've been quite lax with regards to memory consumtion, and as a result Ember 0.5.2 contains both some memory leaks as well as uses too much memory in the way it preloads all available resources independent of whether they are used or not. The resource preloading hasn't been a problem previously, but since Jayr has been adding so much more cool media, with normal and specular maps, the memory usage for all media is now quite high.
I've therefore the last week gone through the codebase with memory profiling tools to try to find both unnecessary memory usage as well as memory leaks. The main tool I've used is Valgrind, which is really an excellent tool for all kinds of profiling. KDevelop comes with built in support for the memcheck tool, which finds memory leaks. It will keep tabs on all memory that is allocated and find those places where there's no more references left to unfreed memory. The valgrind massif tool is also very handy. It will track the heap of the application, and with regular interval report how large it is, and from where different sections of the heap has been allocated. The massif tool by default outputs its info in the console, which isn't very easy to navigate. I've however found an excellent tool for displaying the data in a much better graphical view over here.
Using these tools I found an issue with how the terrain splatting materials were generated, which in normal use amounted to a memory leak of ~100Mb. Quite a lot of data. Fixing it was trivial.
I also found some issues with how Ember was shut down, and how the main scene manager failed to be deleted. While this doesn't amount to any in game memory leak (the application is shutting down anyway, so unfreed memory will be freed a second later when the process exits) it did clutter the valgring output, and we should of course always strive to have components that correctly handles the allocated resources.
I'm now trying to find the cause of a problem I have with ModelDefinitions not being correctly deleted. They use the Ogre SharedPtr, which makes it possible for them to keep on living a little longer, even though the ModelDefinitionManager has been deleted, as long as something else is holding on to a reference. I'll have to investigate further.

I've also removed the preloading of all assets. This will make sure that only those assets that are needed are loaded, but the main (and quite big) drawback will be that the game will stutter as the player enters and moves through the world. This is because Ember so far is single threaded, and the main thread will have to wait while new assets are loaded as they are encountered. We're planning to move to a multithreaded resource loading solution, but we haven't started on that blueprint yet.

Much of this work is a result of the work I've done in trying to create a new island world. Since the island world is quite larger than the current world the memory usage issue became much more obvious. The plan is to release 0.5.3 with the new world. However, I'm leaning towards perhaps trying to get a 0.5.3 release out without the new world. One of many reasons is that Ogre 1.4.8 which was released after the Ember 0.5.2 release changes how resources with absolute paths are loaded. This clashes with how we load our terrain config file, and the result is that while Ember 0.5.2 compiles without problem with Ogre 1.4.8 it won't be able to run. We've already had some issues with people encountering this problem, and we'll sure have more and more. In addition, Nvidia releases a new driver version not long ago which contains some bugs which will either hang the application at startup or produce messed up mipmaps when used with Ogre 1.4.8 and lower. As a response Ogre therefore did a 1.4.9 release yesterday which contains fixes for these issues. As more and more people upgrade to the newer nvidia drivers we're sure to see more of this issue. I'm therefore thinking about trying to push a 0.5.3 release out as soon as possible. Before I do that I would however very much like to see all major memory issues resolved.
There's currently some kind of memory leak which leaks ~4kb/s when just running Ember, without even being connected to a server. That's not ok.

Terrain shaping up

2008-05-26T15:59:00.002+02:00

The last couple of days I've been toying with a new terrain. I'm thinking of an island with some far away mountains and some harbours. I've put up some preliminary screen shots here. With the new terrain editing features it's really easy to quickly create some nice looking terrain. I should really record some movies of it...
When working on that I found a couple of problems in Ember though. The first was that it used massive amounts of memory. A prime source for this seemed to be the fact that it created a lot of both bitmaps and textures for all of the terrain, which it then never really used, but had lying around anyway. That prompted a refactoring of how the terrain worked which removed most of these instances. While doing that I also realized that we're quite wasteful in how we use full 8 bit bitmaps for all of the layers in the terrain, even if one layer might be the snow, where the 256 steps allowed by 8 bits never are used. To improve this we should combine many layers into one bitmap, where perhaps the grass layer is allocated 64 values, while the snow only is allocated 8 values, and so on. This would only work with shaders though.
After refactoring the terrain texture system I found that it still gobbled up memory, but I haven't run any good benchmark yet. I'm however guessing that it's the Mercator system which is initialized in full when the world is loaded, and then never has it's data removed. Preferrably, we should only load the Mercator segments on demand, and we should only keep those most used around and unload the others (perhaps by keeping a time stamped cache). It should work very much like the Ogre PagingLandscape system works.

I also quite alarmingly found that Ember currently is leaking memory. Just by starting it up and having it running without connecting to any server makes it allocate 2-6Kb/s. Something is seriously wrong here. I need to hunt that down and fix it.

When searching for the cause of these things I did some alterations to the logging system, making it not shut it down so early in the shutdown process. I also cleaned up the headers and moved some classes to separate headers. The logging service is included in pretty much every other file in Ember, so making it slim is good for compilation performance. Unfortunately the changes I did to it will force an almost full recompile after updating...

I also ran into a really strange problem with mouse picking in the terrain when testing out my new island. The picking is simply off. It's however working as it should the Mason world, which makes me think that it's either something to do with the fact the the island world is much larger, or that it's in a non-square form. I'll have to investigate it further.

This week also marks the start of the coding period for the students, which is really exciting. I'm looking forward to see how they'll evolve as they more and more into the WF system.

Shaping worlds

2008-05-22T10:07:00.002+02:00

For the last couple of days I've worked on the tools for shaping the world. As my goal for 0.5.3 is to have a better world which shows off all the new media the first step is to alter the world to allow for more expansive forests and far away mountains. There's been a terrain editor in Ember for some time, but it has been very rudimentary and lacked some features which would make it easier to alter the world. One of these features was the ability to alter more than one terrain point at once. I've now added the ability to set a "radius" in the editor. That will alter all terrain points around the one that the user interacts with. This is very helpful since it allows us to quickly raise and lower large swaths of land.

When I started fiddling with that I found a couple of bugs in the terrain code which prevented it from working as smooth as I wanted. As I've squashed those bugs the editing is becoming much more smoother. I also found that the way the terrain is reloaded is very inefficient. Currently we unload the whole terrain page and then load it again. However, all that we change is just the height data, so it would be much more efficient to tell the engine just to recalculate the affected meshes from updated data. I've done some changes to the code which allows for this much more efficient approach, and it's beginning to look good. There's still a problem with the LOD calculation not being correctly updated, but that shouldn't be too hard to fix. All in all this will allow for the ability to quickly create a whole new world from a blank slate of terrain. Very exciting stuff.

Building worlds

2008-05-19T09:22:00.002+02:00

The start of the Summer of Code is getting closer and closer and our three students have already started getting involved in the community. They've all got their git repos set up, and Alexey which has planned some downtime during the summer has already started committing code.
There's also two pages on the wiki about two of the projects. All in all it looks very promising.

I sat down and thought about what I wanted to see in the next release. It didn't take long to realize that the major focus must be on making sure that all the new media that Jayr has created must better be shown within the game. Just by looking through the commits one can see that he's put a lot of work into the media repository, and it's a real shame that not much of it is shown in the game. This mainly has to do with the fact that there's either no corresponding entity types on the server, or that they aren't used in the default world.
In addition, with Ember 0.5.2 we now have support for imposters, which means that we can display much more entities on the screen at once. This is mainly of use when rending foliage and forests, of which we would like to have much more. Jayr has created a couple of really nice birch trees which can be seen here:
He's also added some bushes as well as a whole collection of new nice meshes and textures. So the focus for Ember 0.5.3 is to update the world to make sure that all of the new media is shown. All other blueprints will be put on hold to favour the push. The first thing we need to do is to update the default terrain. Ember can handle quite large terrains without problems, but have a harder time with really steep cliffs, due to LOD issues. This is pretty much the opposite the current layout of the world. We therefore need to expand the world, make it larger and with more distant mountains. Making it larger will also allow for more expansive forests. In addition, we have quite a lot of meshes for both a village and a castle, so we need to provide space for that as well, just as we also need to make sure that the village and castle look good and are naturally laid out. There's already some maps of a proposed world, as seen here, but the current world isn't following it, and it presents a major issue in that it's hard to do world boundaries, where the world simply ends, without using water. In the current setup there's a couple of very large mountains to two sides, and water on two other sides. I'm thinking of going with an island completely, just because it's so much easier then to have a natural world boundary. There's nothing that says the world needs to be in "Moraf" either.

The first step towards a new world is creating the terrain, so the last couple of days I've been focusing on getting better terrain editing functionality into Ember, as well as finding and fixing some bugs related to the real time terrain editing. We now have the ability to alter many base points at once, allowing us to much quicker and easier alter large expanses of the terrain. The next couple of days I plan on experimenting with a new world. Once that's done we should start on making sure that all meshes and textures are represented by ingame entity types. There's now a nifty export functionality in the Model Editor which will automatically create and export an Atlas definition from a pre-existing mesh. This has already proven to be very useful in quickly generate atlas types for missing media.

Last week I also moved all Ogre material definitions from the Ember git repo to the media svn repo, where they much better belong. The idea here is that they should be used as reference when working with the media, so that it's possible to use the exact same materials both when authoring media and when showing it in Ember. We're not there yet however as we also need a good mesh viewer application that can work with the way the media repository is set up.

Summer getting close

2008-04-17T22:40:00.005+02:00

The Google Summer of Code is just now entering into the phase where student allocations are finalized. We've recieved some really nice applications and will be able to announce the final selection soon. This is the first year that I'm involved in the program, but Kai is a real veteran who's been involved with it since 2005. His help has been really valuable.

We've now used git for a couple of weeks and so far it's been a total success. It's sooooo much nicer to work with than cvs. It's a little bit more complex, but in day to day operation it's just a couple of simple commands you need to know. Also, while superficial at first, the web gui is really nice and useful. The ability to do local commits that aren't published before you decide to is immensely useful and really changes the way you work with code. You now don't have to worry about each and every commit not breaking stuff; it's quite ok to commit non-working code as long as the stuff you push to the common repository is working. If only Kdevelop had some better built in git support it would be even better.

With Ember I'm now trying to get a polished 0.5.2 release out. Both CEGUI and libwfut recently released new versions which fixes a couple of bugs we get in Ember, so the new release will require those versions. Jayr has been adding more and more really high quality media. A lot of new normal and specular mapped textures which helps to make the world much nicer. We now only have to update the server to take advantage of all the new media. The fact that there's no persistance and entity editing thus very much depend on editing the define_world.py script on the server makes that process a bit cumbersome though. I've added some support in Ember for saving entities in a serialized form to disk. There's however no way to load them into the world yet, so that's something that needs to be added.

Sean has also provided more patches, adding support for letting the user save the login information for servers, thus removing the need to having to type both user name and password every time one needs to log in to the world (which is something that you do a lot when you're developing Ember). He's also added functionality for totally disabling the gui (bound to the F10 key). Useful for taking screenshots and so on.

Myself I've been fiddling a bit with the minimap. I started out using CEGUI for all elements of it, but since it doesn't support rotation of element I had to abandon that path. I'm currently working directly with Ogre overlays, which should provide the needed flexibility. I'm not there yet, so it's something that won't appear in 0.5.2.

I'm hoping to get the 0.5.2 release out this weekend. Currently it's mainly issues with misconfigured media holding it back.

Summertime is coding time

2008-03-20T00:57:00.003+01:00

Worldforge has been accepted to the Google Summer of Code, which is really excellent in many different ways. Not only will we now get a student working on the code while being paid by Google, we will also get some nice publicity. I've already seen a sharp increase in visits to our sites after Google announces the projects that were approved two days ago. More information about this can be found in the wiki. We've also set up a new forum dedicated to the program.
For prospective students it's worth noting that the project ideas put forth in the wiki page are just suggestions. As such we really encourage the students to either expand on the suggested projects or to submit new projects. The forums are perfectly suited for such discussions.

Other developments lately have been that Sean Ryan has checked in his first code contribution. He's been working on adding logging of in game chat messages to text files in the user's ember home directory, something which might be handy for seeing what's transpired in the world. His code was solid and functional which always is good to see. Getting into the code can be quite daunting since there's so many different sub components used, but he's managed to navigate it all, with a little help.

We've also now completed the conversion from cvs to git. While the cvs repo will remain for some time, all new development now happens through git. Kai has done all the heavy lifting with setting up the server and all. A very nice web interface of it all can be found here.
In order to get the latest development version you should first do

git clone git://git.worldforge.org/ember.git ember

which will checkout the latest code to a new "ember" directory. To keep in sync with the latest changes you would then do

git fetch

More information here.

All over the place

2008-03-17T00:08:00.002+01:00

The last couple of weeks I've been all over the code base, working on different things each day it seems. I've upgraded Caelum to a newer version, fresh from svn. It gets better and better. New in this one is nicer sunsets and sunrises, but the same code which make that so nice also produces a little to large halo during the day. But overall it looks nicer and nicer.
I fixed some long standing but rare bugs, such as a crash when you had deleted an entity with the entity editor and wanted to look at another. I also fixed a very spurious problem where sometimes chat messages wouldn't appear in the console. Turned out that the lua garbage collector sometimes destroyed the connection object which binds chat messages to the console output. Garbage collected languages can be very nice, but when there's a problem like this it's really hard to track down due to the nature of the unpredictability of the destruction.

I also spent some days restructuring how media is handled. This is mainly a problem for other people wanting to help with the development. Since it's only been me for quite a while it's been quite easy: my box contains the latest dev media. But as more people get interested we need a better way of distributing the development media. In contrast to the release media this is very much in flux and is often updated. The new solution is to use a rsync service at amber which always will contain the latest media. By using the new make target "devmedia" it's possible for other developers to get the latest media by simply typing

make devmedia

and rsync will take care of it all. Furthermore I've removed the configure variable MEDIA_VERSION in favour of getting that information from ember.conf instead. This makes it much easier to configure and won't require a total recompile every time the media version changes. This is just one move towards a better media system. I've already added some blueprints on how I would see the media system getting even more dynamic.

The icon rendering works nice and all when there's FBO support on your GPU, but when there's not there are a couple of issues. The current way of solving these is to delay the copying of the render information one frame and then blitting it to system memory and then to the icon atlas texture. However, the Ogre::Viewport class has the ability to render into just a smaller section of a larger texture. This would be perfect for the icons, since then we could render directly into the icon atlas. So I implemented that, only to find out that there's either a bug with how it works, or that I've totally misunderstood it. The problem is that every time a full frame render occurs, this seems to mess up the whole icon atlas texture, and renders junk data into all sections that don't have a viewport assigned to them. It seems the problem is that it either gets repainted by mistake, or that it's intended behaviour. I need to research it more closely, but for now that functionality is deactivated in favour of the old way of rendering icons.

It seems that I with regular intervals goes through the code looking for places where I can refactor it to better fit with updated behaviour, or simplify structures and methods that works like a cludge. Just basic cleanup. I spent one day doing that too, adding some nifty methods and classes to the configuration service as well as adding a new time service which can be used to both get local and server time. Should come in handy.

I've also evaluated both git and monotone. They both seems to have pretty much the same functionality, but I just like the feel of git better, so I'm gonna migrate all of Ember into that. We'll still keep the old cvs as read only, but all new development will happen in git.

Currently I'm working on adding an overhead map. It's good progress, but there are some different approaches to it. I can use CEGUI and do pretty much all of it with CEGUI elements. Or I can use Ogre overlays. Or use the Ogre compositor system. There's pros and cons with all of these, so currently I'm just getting a feel for it. There should be a working version in some days though.

Oh, and today is the day that Google announces the projects that it has accepted for the Summer of Code. This year we applied, so hopefully we've gotten a slot. But you never know.

Reeds

2008-03-05T11:12:00.003+01:00

With the new terrain foliage system it's now very easy to add new layers of foliate by just altering the .terrain xml file. Yesterday I set out to try this by adding some reeds to the world. We've had textures for reeds for a while, done by zzorn, but since there hasn't been a good system for adding them to the world they've been unused so far. Adding them now was however trivial. We want the reeds to grow by the water, and only there. They are also very much like grass, so they should use the same rendering technique as the normal grass. Adding them to the world was thus as simple as editing the moraf.terrain file so that the sand layer definitions looks like this:

<layer shadername="sand" diffusetexture="3d_objects/environment/ground/textures/sand/sand_AD.png" normalmaptexture="3d_objects/environment/ground/textures/sand/sand_AN.png" tilesize="8">
<foliage planttype="grass" populationtechnique="cluster" rendertechnique="grass">
<param key="minClusterRadius">2</param>
<param key="maxClusterRadius">10</param>
<param key="clusterDistance">25</param>
<param key="density">2</param>
<param key="falloff">0.6</param>
<param key="material">/global/plants/grass/ducktail/single</param>
<param key="minHeight">2.0</param>
<param key="maxHeight">2.5</param>
<param key="minWidth">1.0</param>
<param key="maxWidth">1.5</param>
<param key="swayLength">0.25</param>
<param key="renderTech">quad</param>
</foliage>
</layer>

This tells the terrain system to add foliage to the sand layer, and the sand layer only. It also tells it to use the "cluster" population technique for placement, and the "grass" technique for rendering, using the "ducktail" material. The other parameters should be pretty self-explanatory. The end result can be seen here:

Not too bad. The main point here though is how easy it is to alter and add new foliage layers. Worth noting also is that we used the built in cluster populating technique, but this would be a prime candidate for using Mercator Areas. By using areas, the world designers would have much greater control of where to put the reeds, and it could also be possible for players to affect them in the world, for example by using a scythe to cut them down.

Yesterday I also committed some performance improvements to the foliage engine, mainly by using batch stores for the sub areas, thus avoiding having to iterate through all plants for each geometry page.
I stumbled upon a bug in the entity editor which caused a crash when deleting entities in the world. It was however an easy fix (don't hold on to deleted entities...) which I will commit later today.

New skies opening up

2008-03-04T15:07:00.003+01:00

I've spent the last couple of days updating the version of the sky system Caelum that Ember uses. We've been using an almost two year old version, and there's been some nice recent improvements to it. It turned out to be fairly easy to do, some classes and interfaces had changed, but all in all it was a straight forward update process. The end result is a better looking world, if I might say so myself. The sky model has been refined, and support for clouds and sky transitions is more smooth and robust. One interesting addition is that it now uses an astronomically correct earth model, where both earth position and absolute day must be specified. This opens up some nice possibilities for virtual worlds, since it then is possible to specify where on the earth sphere your world should be placed, and get a different lighting model. A world placed at the north pole would have a very different day-night cycle than one placed at the equator. Alternatively, the server shouldn't assume that we're on an earth sphere and could instead just specify direct values for night and day cycle, as well as the sun's trajectory over the sky. That would require us to create our own solar system model classes to be used by caelum, but it's certainly doable. But there's a certain simplicity to just letting the server specify the longitude and latitude that's quite alluring.
Another issue is how to present the world during night. In the current caelum implementation it just gets darker, until you can't pretty much see anything. This isn't very nice for players, and would probably force them to always manually set the time to noon to be able to see something. Instead we want to provide a world that's as clear during the night as it is during the day. WoW does this splendidly, and we need to adapt something similiar. But until then we'll use the current hack, where the time of day on the client always is adjusted so it's sometime around noon, even when it's in the middle of the night on the server.

Another thing that's I've committed is some updates to the PagedGeometry system where it now uses classes specifically tailored for Ember, which renders near objects as entities while switching to imposter rendering for far away ones. Previously we had to use the DummyPage class and use the Entity::renderDistance functionality for hiding far away entities, but now that's all smoothly taken care of by classes which work directly with the PagedGeometry engine. There are still some issues with both performance and updates that needs to be taken care of, but it works much better now.

On the Worldforge front, there's two new exciting developements which has unfolded the last couple of weeks. One is that we now have a new Ember developer, Sean Ryan, who's just starting out getting a feel of the Ember code base. He's already taken upon himself to implement event logging as an initial task which is an excellent way to start.
The other development is that we're applying for the Google Summer of Code this year. We've already set up a wiki page for this purpose and will gladly accept any applicants or project ideas. If you think this is something for you, or if you know someone that might be suited, please get in touch with us, preferrably by using the mailing lists.

The grass is greener

2008-02-21T21:49:00.003+01:00

Work on implementing a better foliage system has continued, and I'm getting more and more pleased with the results. Here are some recent screen shots.
As with all other functionality in Ember I've strived to make it as general and modifiable as possible. It should be trivial to both change the behaviour and to extend it. The way it's implemented is through the use of the terrain layers as a base. This is how the xml definition for the grass terrain layer looks like (I've removed some texture definitions):


<?xml version='1.0' encoding='UTF-8'?>
<terrain>
 <layers>
  <layer shadername="grass">
   <foliage planttype="grass" populationtechnique="cluster" rendertechnique="grass">
    <param key="minClusterRadius">2</param>
    <param key="maxClusterRadius">10</param>
    <param key="clusterDistance">25</param>
    <param key="density">2</param>
    <param key="falloff">0.3</param>
    <param key="material">/global/plants/grass/bittergrass/single</param>
   </foliage>
   <foliage planttype="fern" populationtechnique="cluster" rendertechnique="shrubbery">
    <param key="mesh">3d_objects/plants/shrubs/models/fern/fern.mesh</param>
    <param key="minScale">0.7</param>
    <param key="maxScale">2.0</param>
    
    <param key="minClusterRadius">1</param>
    <param key="maxClusterRadius">3</param>
    <param key="clusterDistance">30</param>
    <param key="density">2</param>
    <param key="falloff">0.7</param>
   </foliage>
   <foliage planttype="camelia" populationtechnique="cluster" rendertechnique="shrubbery">
    <param key="mesh">3d_objects/plants/flowers/models/camellia.mesh</param>
    <param key="minScale">2</param>
    <param key="maxScale">5</param>
    
    <param key="minClusterRadius">2</param>
    <param key="maxClusterRadius">5</param>
    <param key="clusterDistance">40</param>
    <param key="density">1</param>
    <param key="falloff">0.7</param>
   </foliage>
   <foliage planttype="leaves" populationtechnique="cluster" rendertechnique="shrubbery">
    <param key="mesh">3d_objects/plants/shrubs/models/leaf/leaves.mesh</param>
    <param key="minScale">0.5</param>
    <param key="maxScale">1.2</param>
    
    <param key="minClusterRadius">2</param>
    <param key="maxClusterRadius">5</param>
    <param key="clusterDistance">35</param>
    <param key="density">2</param>
    <param key="falloff">0.7</param>
   </foliage>
  </layer>
 </layers>
</terrain>

Every foliage type has both a population technique and a render technique. The first one determines where to put the vegetation and the latter determines how it should be rendered. Both of these share the same set of arbitrary parameters. We currently only support the cluster population technique. This creates a series of clusters of varying sizes randomly dispersed over the map. In most cases this looks quite nice, since plants tends to grow in clusters in the nature. All of the plants positions are calculated when the terrain is loaded, and not changed thereafter. However, every time a plant is placed we check all the layers that are above the current layer. If the combined value of all the layers above are above a threshold value we don't show the plant. This has the effect that roads, fields and other areas don't get vegetation, unless there's foliage defined for that particular layer. And since we've already determined the positions for the plants we can update foliage swaths as areas are added without having the positions of existing plants getting jumbled each time.

Regarding the render techniques we currently support two different: grass and shrubbery. The former autogenerates grass meshes, and animates it whereas the latter uses preexisting meshes. So far the latter haven't got support for animations, but that's something that we'll surely add.

The population techniques are implemented in Ember so far, but the idea is to put them in Mercator as they stabilize. That way it should be possible to specify foliage on the client, very much like we now specify Mercator shaders.

All in all it's looking mighty promising, even though as can be seen in the screen shots we need to fix up the colours a little.

Impostering

2008-02-04T10:48:00.000+01:00

I've now finally implemented the Paged Geometry engine into Ember. This is an component which allows for much more nicer looking foliage through a variety of different techniques. While we've had grass through the use of batched geometry for a while, the implementation wasn't really optimal and it didn't contain the more advanced features needed. The main thing lacking has always been a proper implementation of imposters, which are 2d billboards used in the distance. Instead of burdening the GPU with full geometry for far away trees, which in the end will end up just taking up a few pixels on screen, we render the tree once to a 2d sprite and reuse that. The end result is an enormous speed up for forests scenes, where there can be thousands of trees on the screen at any one time.
All of this is provided by the Paged Geometry engine, which is why I'm very excited to finally have it in place. So far I've partially converted the grass system to use the new engine. It's currently lacking some of the filtering features, which makes the grass appear on paths and other areas where there should be no grass. It's also just randomly dispersed, which makes it look a little bit unnatural. It would preferably be dispersed in a more cluster like fashion, to better simulate the way grass grows in the real world. All of this is currently being addressed however.
The trees now use billboards when viewed from afar. I've extended the Model format to take an additional "rendering scheme" subsection. This allows one to specify different rendering schemes for different models. Thus we can tell Ember to use the rendering scheme "forest" for our trees, which will set up everything and make them switch to imposters at far away distances, while keeping the regular entity rendering at close distance. The Paged Geometry engine allows us to use batched geometry up close, which means that the different entities are "baked" into one large mesh. However, due to the dynamic nature of Ember and the Worldforge world that's not feasible. Thus we need to render the entities normally up close. The current solution is to do that using the regular rendering system and then set max render distance to the same distance as that when imposters take over. But since the imposters work in groups the distance where a certain entity begins to be rendered by imposters isn't exact. This means there are instances where this leads to either both the imposter and the entity being rendered at the same time (not a big problem) or that none gets rendered (a bigger problem). I therefore need to extend the Page Geometry engine with some Ember specific classes which will allow it to smoothly transition between normal entity rendering and imposter rendering.
I've already made some refactoring changes to the Page Geometry, which I intend to try to push back into the main development branch.
I've put up some more screen shots of the imposters and grass over at amber.

On a different note, there have recently been some issue with the Worldforge infrastructure. I've been trying to contact Balinor about the media svn for some time, but have never gotten any response. Last week however the svn server completely broke down: when trying to update I get a message about an internal error. A quick google shows that it stems from the server running out of disc space. This is bad, since not only can no one commit to the repository, no one can even update their local copy against it. I therefore sent some emails to the mailing lists, only to find that they also have been broken. Apparently there was a purge of all the old inactive lists last week, something sorely needed. Instead of having 15+ different lists, we now only have 4, which hopefully will mean that messages won't be lost. Unfortunately this changes seems to have broken the lists, since none of my mails got to the lists. I've been trying to contact the people responsible this weekend to no avail, but hopefully it will be fixed soon.

Grass

2008-01-29T11:28:00.000+01:00

The last weeks have been really productive for Ember. I've closed most of the bugs reported in the bug tracker and also managed to add some new features. One that I've been needing for some time was a teleport function for admin mode. Previously when you were logged in as admin you had to move just as a regular person, which could take some time when you wanted to go to a different area of the map. But now there's a nifty "teleport here" option available when you click on the world, which will instantly transport you to the clicked location. It was very easy to implement, no more than 30 minutes work.
Another thing that has been bugging me for a while was that when you entered and exited building your orientation would be messed up. I initially assumed that there was something wrong with how the server sent updates to the client, but after some thorough testing I found that it was solely a client issue. The problem was that when the user controls the avatar and walks through the world, Ember ignores orientation and position updates sent for the avatar. This is to avoid the issue on lagging connections where the avatar continuously will snap back to a previous position, as sent from the server. When the avatar stops moving it will be adjusted to the data sent from the server however.
The problem here was then when the avatar moves into a house and changes it's container, it needs to be have it's orientation updated relative to the orientation of the new container. And this new orientation is correctly sent from the server, however not as I incorrectly assumed in the onLocationChanged event, but in the subsequent onMoved event. But if the user is moving the avatar Ember ignores the onMoved event and the old orientation will be used instead, resulting in the avatar looking in the completely wrong direction. The solution was to change the functionality so that the onMoved event would be honoured by Ember for the avatar even when the user was moving it, if it occurred right after a onLocationChanged event. This fixed the problem, something which I think has been in Ember for some years now.

A new feature I've been working on this week is support for a better foliage system. I've so far used a home grown system which uses static geometry, but it has some problems and I didn't want to put much effort into it since there are other third party components for Ogre which does it all much better. Especially the Paged Geometry plugin shows real promise. So the last couple of days I've worked on integrating that into Ember, throwing out my own solution. It's a very good library, excellent in both design and documentation. So far I've had no problems getting it integrated, but I think I need to do some alterations to it in order to support the demands of Ember, especially the dynamic nature of the terrain. So far I've focused on getting the grass to work, but the idea is to also use if for the trees and other objects that are both numerous and fairly static. This will hopefully bring the ability to display really vast forests.

Catching up

2008-01-19T18:44:00.000+01:00

I spent some time getting the Ember win32 binary done. Previously I've done it in MSVC and had to employ all manners of hacks to the code in order to get it to compile, but this time I wanted to do it with GCC (no need for code alteration) and in MSYS (no need to create separate projects for all components since the standard autotools setup will work).
Though I managed to get a couple of components built in MSYS I in the end had to settle for a hodge podge of msys, Code Blocks and precompiled libraries. Turns out it's not too easy to build stuff the *NIX way on Windows, shockingly enough!
In the end I got a working Ember binary which is now released. The good news is that now that I've got a build environment set up on Windows the next release should be much easier to squeeze out.

Now that I've got that done I've moved on the working on actual Ember features again. There's been some really great activity on the Launchpad with nice bug reports being filed. Before I start on some serious new features in Ember I'm thinking that I should at least get the new bugs fixed. Most of them are quite easy to handle, but there's a couple of problems with the way the icons are handled. One is that when switching between full screen and the desktop on Windows the icons will get messed up. This is quite straight forward, since the way the icons are done is that they are rendered and then blitted to a larger GPU texture. That way we can keep the number of render state changes down and make it a little quicker. The downside is that it's all happening on the GPU, and when switching back to the desktop the GPU releases all of its resources, and expects them to be restored by the app when the user switches back. There's however no such functionality in Ember for repopulating the texture, thus it ends up with uninitialized data. This is not that big a deal since there are a couple of ways to solve this: either by always keeping an in memory copy of the texture, or by repopulating it when requested.

The other problem is that our blit function won't work on GPU's that don't support Frame Buffer Objects. This is a little bit harder to tackle. The main problem is that rendering happends asynchronous. The app sends render ops to the GPU which queues them and then process them in order. So normally the GPU is a couple of frames behind the main app. However, this means that once we make our call from Ember to render the icon view, we might have to wait a couple of frames until it actually gets rendered. Now, right after we've rendered our icon we then want to blit it to our main icon texture. On cards which supports FBO the blit operation is just another render op, so it will be put on the render queue and happen after the rendering has taken place.
But on cards that don't support FBOs Ogre has to fake this by copying the data itself. And this is where there's a problem, since if we try to copy the data right after we've issued the render call the GPU will most probably not have had time to perform the render, and we'll end up with garbage data (which is exactly what happens).
So the solution is to implement a mechanism where we render the icon, wait a couple of frames and then blit it, thus making sure that the icon actually is rendered. I'm working on such a solution currently and it's looking promising. I've had to alter the structure of the icon rendering framework a little, but thankfully it's quite modular so it's no large alteration.

One thing that I really need to take care of it updating the example world of Mason. Mainly to make it use all of Jayr's new media. He's been adding a lot of really high quality media, which unfortunately isn't shown since the default Mason world haven't been updated with new entities. Both me and Al are quite busy adding new features to the code, so neither have had time to do this, and I was hoping that someone else in the project would be willing to go over it.
Another thing that needs altering is the layout of the world. Ember really likes large vistas with none too sharp height differences, since the LOD will then work best. Unfortunately the current world is pretty much the exact opposite: a small world with steep mountains. If we could move the mountain away quite a distance and make the elevation not as steep I think the world would both look and perform much better.

Windows, yeah that's fun

2008-01-03T23:15:00.000+01:00

After spending the holidays away from almost all things Ember I'm back again. I did manage to do a little Ember work since it turned out that the .package file I uploaded to Sourceforge had become corrupt: that's what you get for claiming it from the upload pool before it has finished uploading. Wasn't harder than reuploading though.
There's been some really promising work done over Christmas however mainly by Alex Torkhov who have contributed greatly to the project through mainly the Launchpad. Our hope was that the Launchpad would make the project more transparent and provide a more technical forum for bug reports and suggestions, and it makes me happy to see that it's being used in that way.
Also, there's been some nice additions to the wiki by Alabandit, mainly concerning an introductory guide to Ember, something I've never had time to do.
All in all some promising new additions.

When I did the 0.5.1 release I only had time to do a precompiled version for Linux. I was hoping that someone could be able to help me with creating a precompiled win32 binary, but in the end I have to roll up my sleeves and do it myself. It's not a pretty job. But I've made a promise that I won't do any further Ember development until there's a binary version available for those using Windows.
Previously I've used MSVC for compilation, but it means I constantly have to hack the wf libs Ember just to make it compile. So instead I'm working on getting a fully working build environment set up in Windows through MSYS. It's getting there, but it slow as hell. Just running configure can take up to 10 minutes. I don't know what's making it so slow, and I have no interest in trying to hunt it down, as long as it works. But then there's all kinds of wonderful stuff, such as Windows case-insensitive naming system, as well as the inability to separate data from file system location ("This file is locked by another process." Yes, but would it be too hard to tell me which process is locking it?). And so on. It's just a tedious work.

And I have so many ideas for what I want to do next in Ember, but all of that will have to wait until I've conquered Windows.

Christmas with the Ember

2007-12-21T13:56:00.000+01:00

The last couple of weeks have been really hectic, as it's always before the end of the year. With not much free time work on Ember has suffered, but I've now finally gotten a stable release of 0.5.1, just in time for the holidays. This release adds a lot of new features, chief amongst them a much nicer interface with inventory and status widgets. As always there's a lot of bug fixes and improvements in almost all areas of Ember. Here's as small list of some of the improvements:

totally refactored ingame chat system with better performance and functionality
support for both Lua 5.0 and 5.1
a new icon system
a new inventory system with drag and drop equipment
status widgets for both the avatar and current targets
update of all materials, with support for normal and shader maps for most entities
refactored terrain system with support for both shader and fixed pipeline functionality

After the 0.5.0 release earlier this year I wanted to do a 0.5.1 release pretty close to it, but for various reasons it didn't happen, and the release comes almost a full haft year after the previous one. The main reason is that I want to release really stable code without any regressions and strange graphic glitches. The problem with this is however that it's really hard to test out all of the various graphical cards when I'm the only one that works on the codebase, and I only really have one development box with a Nvidia card. Previously this has led me to disable the shader terrain rendering, since I found out at release time that it didn't work with ATI cards.
This can't work in the long run, and I'm looking at trying to better embrace the "release early, release often" motto, even if it means that some releases will bring regressions with them. Since I'm the only developer I'm really dependent on good bug reporting from end users. Lately I've been seeing some increased activity on the Launchpad bug tracker, which is really great, since it's just the kind of input I need from the users.
So I will try to get a release of 0.5.2 out not too soon, even if that release might break something. I'm also looking at perhaps using the Google BreakPad for even better crash reporting.

The new release of Ember also uses a lot of Jayr's new materials with support for normal and specular maps for most surfaces. This looks really good and hold great promises for the future as I move to include a better foliage rendering mechanism into Ember (after all, the mason world is an outdoors world, so natural foliage rendering seems almost like a requirement). It also brings the used media up to 90~ Mb.

With future versions of Ember we need to refactor the media system so that world developers can specify their own media repositories in a better way. Currently Ember gets all its media from one place and places it statically in ~/.ember/ember-media-0.5.1. However, we would like to let each world provide its own media if so desired, so that when you connect to one server you might download (through wfut) media specific to just that server.

For this release I haven't had time to set up a Windows build environment and provide a working windows binary. If anyone else feel better suited for this task I would greatly appreciate it, else I will do this myself after the holidays.

In the shadows

2007-11-25T11:16:00.000+01:00

The new terrain rendering system is working really nice and I've now fully integrated a shader based terrain material. The idea is to optimize the material by trying to reduce the number of passes needed. By combining the different coverage layers into one single texture we can keep the number of passes down. On my card with 16 texture units I can squeeze in 12 splatting layers in one pass.

Now that I've got that part done I'm working on getting some more advanced features such as normal mapped terrain working. There are a couple of performance issues to take into account though. There's no reason to render the terrain with normal mapping on when it's in the distance. I'm thinking of adding some features which will allow the engine to automatically create pre-baked textures for the distance, where all of the layers already are put together without the need for splatting.

On the media front, Jayr continues to bring out really nice media. He's reworked the two room house, the gallows, the jetty and the stalls. Most of these are already brought into Ember, with full support for normal and specular mapping.

Shadows

2007-11-07T10:20:00.000+01:00

I've finally started redesigning the terrain system, with some very promising result. The first issue to tackle was how the shadows on the terrain were computed. Previously I left all of that to the scene manager, which used a simple vertex lit approach which didn't look good with lower LOD (and also taxed the GFX card as the shadows had to be recalculated each frame).
I now have a simple shadow generation framework in place. The idea is to provide a couple of pluggable shadow providers for different shadow techniques. Currently I only provide a very simple technique where the shadows are calculated directly from the surface normals (just as vertex lightning). It already looks much better than before though, although I intend to provide some more advanced techniques later on.
Here are some example screenshots:
I've also started on a more thorough refactoring of how the whole terrain system works. The goal is to more easily allow for more advanced features, such as normal mapped terrain, as well as being able to alter the terrain materials in a simple gui ingame. As with all refactoring the code is currently a mess, but yesterday at least I got it to compile. The blueprint for this can be found here and I've so far followed it pretty much to the letter. The material compiler works, though it's very inefficient in the way it creates new materials.

Jayr have been churning out even more great looking media with a lot of new normal maps. It's a real joy to update the material definitions and see how the world comes to life when normal mapped materials are used. It's really such a big change in how everything gets depth and shadows.

Paths

2007-10-19T18:50:00.000+02:00

Jayr have been adding more and more excellent media as well as overhauled the media repository layout. He's been moving source files into an "art" directory, so that the "3d_objects" and "3d_skeletons" directories only contains exported formats. This is great since it means that client developers that only want to get access to the used media, without being burdened with having to download a lot of high res source media only need to check out the two latter directories.

With the new media in Ember it becomes more and more apparent that I really need to spruce up the terrain rendering code. As can be seen in this screenshot the terrain really looks awful in the distance when the LOD kicks in.
This is because it's using vertex shading, where the lightning is set by interpolating the normals of the surrounding vertices. As the number of vertices change with the LOD the lightning becomes messed up. All from the beginning the idea has been that this is a purely temporal solution and it should be replaced with a more robust lightning solution. I've just never had time to implement a better model and thus it's been this way for some time. But with jayr's great looking media it becomes more and more jarring.
I've thus started looking into how to better this. There are a couple of different techniques that can be used, with shaders, normal maps and all. Currently I'm experimenting with just creating a one channel shadow texture from the terrain normals I can get from Mercator (compared to the direction of the sun). This won't take into account occlusion though since no ray checking is done, but hopefully it would look good enough (it would actually look pretty much like normal vertex lightning). We want to update the shadow dynamically as the sun moves over the sky, so we can't use too computing intensive techniques.
With some upscaling and perhaps some gaussian blur I think the result will be good enough.

Another pet peeve of mine that's been bugging me for some time was a bug with the foliage code where it didn't take into account "negative" ares, such as the path and the sty, where there should be no foliage. The result was that foliage was added everywhere, even in places where there should be no grass growing. I've been trying to fix it for a while but couldn't find what was wrong. I finally tracked down the problem to a en error in what lookup container for terrain surfaces was used. So now we finally have clear paths! This together with the work I'm doing on nicer terrain shadows should allow for some much nicer visuals in the near future.

Normalize!

2007-10-17T16:48:00.001+02:00

The last couple of weeks Jayr have been providing really great new textures. The biggest improvement is that he's created both diffuse, normal and specular versions. This together with a some nice shaders allows us to get some really nice visuals in Ember. There's already other media in Ember that uses normal mapping, such as the trees, but we now have a much wider variety of textures, as well as a better naming system. This together with improvements to Ogre's material framework, chief amongst them inheritance, allows us to much easier implement and handle the normal mapping.
Here are some example screenshots:

These show the new fern model (the colour is a bit off against the ground, that will be adjusted) as well as the new normal mapped texture for the tinderbox.

Getting the fern out in the world highlights the problem with the ugly lightning on the terrain. The problem is that I'm currently using vertex lightning. This has some disadvantages in that it won't work with the level of detail system, since far away terrain have fewer vertices, thus screwing up the lightning. There also seems to be a problem with the normals, such as terrain that shouldn't be in shadow is in shadow and so on. All and all it's not a good way of doing terrain lighting. The reason however that I haven't switched to a better model (such as baked texture lightmaps) is that I first need to refactor the terrain generation framework. The current framework works, but it's hard to do things like add different materials depending on whether you have shader support or not. We need a better framework. I've created a blueprint for this. Once this is done we can start working on some much better terrain. I'm pondering whether to actually put this on top of my things to do, since getting a nicer terrain will make the visuals tremendously better (since the major part of the screen is covered in terrain).

You will also see the new status widget in the upper right. He's actually also provided normal maps, though I'm not sure how to use them yet. :)

Missing files

2007-09-26T14:00:00.000+02:00

I had some people try to build and run the Ember source by themselves. It turned out that apart from some missing media files (such as the whole gui texture file) some of the xml files for CEGUI wasn't conforming to the schema specification. The reason I haven't noticed this before is that CEGUI can use many different backends for parsing xml. It can use either expat, libxml, xerces or tinymxl. Very nice component design there and all. But I had always compiled it with tinyxml, which as the name implies is very tiny and don't have any validation support. All incorrect tags are just read as they are. However, if CEGUI was compiled with Xerces instead, which is quite large and has schema validation, the loading would fail since Xerces would throw an exception when encountering invalid xml. And this is precisely what happened when other people tried building Ember, since they had CEGUI compiled with Xerces support. I've now fixed the issue and committed. I have to do a 0.5.1rc2 release soon though, since the modeldef format has changed, disallowing the use of the 0.5.0 media with cvs ember.

On the icon front I've gotten even further with the drag-n-drop functionality. There are some small usability bugs still left, but overall I think it's quite usable. I've been starting to think of how to improve the entity context menu. Currently it's just a regular menu that pops up, but I'm thinking of having a radial menu with icons instead.

Dolls

2007-09-21T14:29:00.000+02:00

I've further expanded on the icon code, and in the process added more and better lua connections. I'm getting better and better at writing lua code. As with most new languages, it's not hard to learn, but it takes some while to start using the advantages of the language. Usually you tend to start out writing code in the same way that you would write C++ code, and then gradually replacing that with code written in the "Lua way".

Yesterday I added a first prototype for a user doll, where it should be possible to see what items currently are equipped. By dragging and dropping items on certain slots it should be possible to change equipment. It worked out fine. The icon system contains two elements: slots and icons. Both are handled by the EntityIconManager. The script asks the manager for some slots, puts the slots above the image of the avatar and attached some scripts to them. Then later on when an icon is dragged from the inventory and dropped on the slot a signal is emitted, which triggers a script. This script calls the necessary server side methods for wielding and changing the outfit, and voila! the item has been equipped. It's very elegant so far and should be easily expandable.

When dropping items, instead of pressing the "drop" button, the icon should be dragged and dropped on the world, at what point it should appear in the world as a 3d element. Similarly, when trading the user should only need to drag and drop items from the inventory into a "trade" box.

The EntityIconManager takes care of all bookkeeping, so it's really easy for the scripts to just request an icon or an icon slot. This should all make the player experience more smooth.

Here's a couple of screenshots demonstrating how to wield an axe.

Further icons

2007-09-05T17:00:00.000+02:00

I've continued on the icon framework and started redesigning the inventory widget. I now have a working solution where the icon management framework provides automatically rendered icons to the widget. A work in progress screen shot can be seen to the left. This shows the icons in the inventory widget to the right, as well as the asset manager to the left, showing the icon texture.
Icons for entities will probably be used all over the interface. Apart from showing basic inventory I can see them being used in equipment dolls, quick bars, trading dialogues and so on.
Therefore I need to design a more robust generic system for dragging and dropping entity icons. Thankfully CEGUI already has excellent support for draggable windows. The icons in the inventory widget can already be dragged. What's left to do however is to allow for them to also be dropped. Preferably in a way that can easily be defined or extended in a lua script. The idea is to allow for entity icons to be dragged and dropped in a generic fashion, so that you should be able to drag and drop an icon from your quickbar to a trade dialogue in the very same way that you would be able to drag and drop it from your inventory.

In addition, I quickly realized that some entities are rendered nicely by default. For example, the coins are by default rendered lying down. Since they don't have any depth, this will result in an empty icon. We therefore need some way of defining how certain models should be rendered when shown as icons. This was easily fixed by extending the .modeldef format with a new "" tag. This tag allows you to define arbitrary "views" for the models. When rendering an icon Ember looks for a view by the name of "icon", and if found sets the camera distance and orientation from the values defined. I'll have to add support for setting this to the model editor though, since it's a bit hard to guess the correct orientation values by hand.

The inventory also needs some better looking graphics. Well, the whole of the gui needs better looking graphics. I've created a blueprint for that here.

Icons galore!

2007-09-03T19:17:00.000+02:00

After adding the status widgets and updating the entity labels the only thing left that made Ember look ugly in the default mode was the inventory widget in the lower right. It's always been meant to be a placeholder widget only showing the most basic info (a list of entities with name and type). The grand idea was to instead use small icons for the different entities, much like each and every other game in existance. I could easily have used the EntityCEGUITexture class to render individual entities into separate textures, but this being Ember I opted to go for a more robust solution. Icons is something that we're gonna need here and there in the client, not only in the inventory. I therefore though it would be prudent to provide a more generic icon manager class which uses an icon cache, handles rendering and updates of icons and does house keeping of all icons. I've added a blueprint for it to the launchpad. So that's what I've been doing the last week. I've gotten so far as to have the basic classes done for letting other components check out icons for both entities and Eris base types. They are automatically rendered in a small size (64x64) and copied to a single larger texture. This will increase performance compared to using multiple small textures.
While I've worked on this feature I've also added some much needed asset browsing features to Ember. I found that while debugging I had a great need for a widget which listed all textures used in the system and allowed me to see real time changes to them. This is something that I've felt the need for before, but never had a pressing enough issue. The first iteration allows for texture browsing only, but the idea is to build on it to allow for browsing of all different resources used, such as materials, scripts, meshes, models and so on.
All in all, some nice features for a much better looking UI are on their way.

Chatting

2007-08-21T11:17:00.000+02:00

I've now finished the restructuring of how labels and chat bubbles are handled. The new system is much more robust and efficient. Instead of creating the chat widgets from scratch every time a npc says something I now keep a dynamically growing pool of already allocated widgets from which I check out and return widgets once they're not shown anymore. I created some utility classes for general widget pool behaviour which should hopefully be reusable for other widgets in the future ("WidgetPool.h").
Labels are now shown for all entities that inherit from the "character" entity type. That includes pretty much all npc and animals in the game world. For now it's a hard coded check, but it could either be added as an Action to the modelmapping definitions, or as a new rule format (perhaps in lua?). If an entity is controlled by an outside client (i.e. a human) the name is pre- and suffixed with "!" signs. This will be very useful as it will now be easier to see which entities that are controlled by other users.
I also use a new, bolder font for the text. And I added some gradients. The latter might look a little garish, but it helps alot with showing the text on light backgrounds (white text on white background is no good...).
I also solved a strange issue where the text previously always lagged one frame behind the camera. So when the camera moved the text "swayed" a little. It looked quite strange. However, since I now listen for the actual Ogre render call before I position the windows, that issue has disappeared.

The guys over at fedora-games have been interested in packaging WF tech for a while. They already have builds of both Cyphesis and Sear, but alas no build of Ember. There were two reasons for this:

Ember used WFUT which in turn used Java, which was kinda cumbersome
Ember required Lua 5.0 and didn't work with Lua 5.1 which is what's shipped with Fedora

However, since some releases we're using libwfut and have no dependency on java anymore. That leaves the Lua issue. Yesterday I took a look at how hard it would be to allow for both Lua 5.0 and 5.1 to work with Ember. Turns out it wasn't hard at all: some #ifdefs later I had a working app which used Lua 5.1. This does however require that CEGUI is built with Lua 5.1, but there are patches available for that (Fedora itself builds CEGUI with 5.1).

All in all this might make me do a 0.5.1 release pretty soon, so that the Fedora guys can get cracking on getting Ember to be included in their package.

Labelling

2007-08-17T12:39:00.000+02:00

Yesterday I spent some time revamping the way labels are shown on entities. Currently they aren't shown at all, unless the entity speaks. Then a little "talk bubble" widget is shown. If the entity has some suggested responses, they will appear as clickable buttons. But after a while the widget will fade and disappear. This makes for very bad usability, since it's not possible to easily see the names of any near npc or other players without clicking on them. And it's of course not the way that all other virtual worlds does it.
The plans is therefore to provide a better mechanism for displaying labels on certain entities. I've added a blueprint for this to the Launchpad. The first step is to refactor the quite inefficient way of showing the labels that are currently in use. Instead of creating new CEGUI widgets every time a new label should be shown, and then deleting them afterwards we'll keep a pool of reusable widgets. And instead of doing a lookup of the active widgets every frame we'll use listeners that we attach to the entities and the models to actively listen for events that tell us directly whether to show or hide the labels. Currently it's all still in C++, but the plan is to somehow separate the actual widget creation and rendering into Lua scripts, while keeping the heavy book keeping code to C++. This should also let us mark those avatars that are currently possessed by a human players. There's already an attribute ("external") which marks these, though it's not used. By providing a marking it should be much easier to interact in the world. Currently you often start out in a world with 20+ inactive avatars, with no way to discern which ones are actual human beings (granted, one of the design goals of the AI is that the npc's should behave like humans, but we're not there yet).
So far I've gotten to the stage were a new labelling backend system is fully functional and in use. The next step would be to extend the Model format with information on what entity types to show labels for. Or perhaps it would be better to put that information in the ModelMapping format as an Action? Hmm...

Heads up

2007-08-15T08:41:00.000+02:00

After returning from my vacation I yesterday added some status widgets to Ember. That's one of those things that's been bugging me for a while. It's not hard to do; I just never had time. Before I started I looked through most other major mmorpgs that people are playing. People generally like when they can reuse their existing knowledge between different applications. The way the interface works in almost all mmorpgs on the market today is very much standardized: you have your status widget, your map, your inventory and your message window.
So I looked through WoW, LOTR and Vanguard and used that as a base for how the default status widget looks in Ember. I reused some simple graphical elements that jayr had created earlier and think it looks ok. The left element always shows the status of the avatar, while the right element shows the status of the entity that the avatar uses something on.
I'm using the Launchpad more and more. I've put up bugs and blueprints there. My hope with the blueprints it that it will make the development process more transparent and in the end allow for other developers to chip in. Ember is still a one man project and I could sure need some help with it.

Release

2007-07-25T00:14:00.000+02:00

The last couple of days I've been trying to get a release of Ember out. It seems I've finally succeeded. First I had to tackle my box crashing all the time. Turns out that it wasn't the SATA card after all (but not until I already had ordered a new card from the local web retailer). Apparently it was the VMWare network kernel module which didn't want to play nice with my Mandriva kernel, resulting in random kernel panics. After applying a "vmware any-to-any patch" it seems to work a little better, even though I've had the box lock up after applying it.
Moving on, of course a lot of different problems would rear their ugly head once I started testing on Windows. Some of them had to do with lousy memory handling on my part, but some seemed to originate from the Ogre GPU code. I've disabled specular normal mapping with Pixel Shader 1 for now, since it seems there's some problems in how Ogre handled my scripts for that.

After all this however I finally managed to get a working version of Ember out. Version 0.5.0. The main news is that it's using Ogre 1.4.3 and CEGUI 0.5.0. The latter means that the gui now has a lot more capabilities than before. There's also a lot of changes going into this release, such as a new entity editing framework. And a lot of internal restructuring to make for a better foundation.

I'm going on vacation now for two weeks or more, so this is pretty much the worst time to do a release. But I'm so sick of having to wait for it; I just want to get it out there. All bugs that are sure to appear as people start using it (and I know people are, even though only a very few turn up at IRC or the forums) will be handled once I get back.

Oh, and another thing. We've started a trial of Launchpad, which is the Ubuntu project management tool. I've put up some initial blueprints and bug reports there. I think the tool has a lot of potential, but it remains to be seen if we're really gonna use it. The entry for Ember can be found here. All bug reports regarding Ember should be posted there from now on.

Pleasantness

2007-07-18T10:30:00.000+02:00

As always I end up with all kinds of problems getting a working version of Ember out. I use VMWare Server to host an Ubuntu image on which I build the linux autopackage version. The last week however I've been experienced random kernel panic locks up of my main box. I initially thought it was caused by a faulty SATA chip, but now it seems it's caused by the VMWare network module. Oh well...

All is not bleak however. I've been working with the extremely talented Jayr Kalugin with gettings some more bling into Ember. We've focused on creating some nicer trees, and using more advanced materials with normal maps. I think the initial results are very promising, even thought there's still a lot of work to be done.

Keeping up with the make-it-look-nicer theme I also went over the foliage code. Ever since I changed how the terrain was handled some time ago it's been broken, and I've been reluctant to fixing it since it's basically one big client side hack: it doesn't take into account new areas such as the path to the village, resulting in grass where there should be no grass, and it's not supported on the server. However, it's one of the things that people notice the most, and which I get a lot of questions about. So, in order to present some nice screen shots of the upcoming release I fixed the broken implementation. It looks quite good, but is very resource hungry.

Balance

2007-07-11T17:18:00.000+02:00

After releasing the beta 1 of 0.5.0 I've gotten some nice feedback from both Al and Simon, mainly related to gui and user interaction issues. As a developer it's too easy to become accustomed to all the quirks and inconsistencies in the interface, to the point that you don't notice them any more. Therefore it's great to get some fresh outside perspective on the player experience. I've fixed most of the issues in cvs and should publish a beta 2 soon. I still haven't looked at getting it to compile on Windows though. Hopefully roger will be able to provide me with a working compilation solution.

One thing that's still bothers me is that the world doesn't look as good as it should be able to. There's still a lot of rough edges and bad lightning. The main reason for this is that I've focused on providing other functionality, such as editing and underlying frameworks for scripting, model definitions etc.. There's tons of things that could be done to improve the look, chief amongst them is to make better use of shaders. Currently there's pretty much no shaders used at all. In previous versions I've used shaders for the terrain, but since I got some strange artifacts on ATI cards I disabled that. Using shaders I would get a much better looking terrain, as well as a nice speed boost (since I don't have to do the multipass texturing I do now).
Another thing which would improve the look a lot would be some normal maps on the characters. The current vertex lightning makes them look very blocky and jagged. With normal mapping and per pixel lightning it would be possible to give them a much smoother look.
As I've gotten the entity editor in a working state I'm gonna take a closer look at some of these issues. It's a shame not to utilize the many, many nice features of Ogre.

Navigating

2007-07-03T19:39:00.000+02:00

I've added two improvements to Ember that I think a lot of people will appreciate. The first is finally putting in checks so that the camera never dips below the terrain. In the end it wasn't that hard to do, I just had to make sure it wouldn't bring everything to a crawl due to massive ray checks. It's still not perfect; when the camera is close to the ground it will still show some of the underworld as it pokes through parts of the terrain. I'll see if I can enhance it to use a more smoother algorithm for positioning the camera a little bit above the terrain.
The other thing I added was terrain decals for showing where the avatar was heading when using the "move to" functionality. It looks really nice and adds some visual hints to the user. Some screenshots of it in action.
Right now it's used for the movement marking only, but it could be used for all kinds of nice in game markings.

Release

2007-07-01T10:21:00.000+02:00

I finally got a beta1 of Ember 0.5.0 out. An announcement can be found here. No Windows version this time since I haven't even began looking at the windows build. And also because I got some offers on helping out with the Windows build.
It's always hard to do a release, mainly because I have to halt adding new features for a while and focus on fixing bugs and handle package peculiarities. Basically all development is put on hold until the release is out. Now, this has gotten better as I now have more helper functions in place, but it's still a break from the normal progress.
This last week I've focused on finding and fixing all of the lingering memory corruption issues. I've managed to hunt most of them down I think.
I also reworked how binreloc was integrated. This only applies to Linux builds, and basically means that Ember can either be built as a normal UNIX app with hard coded paths, or as a "relocatable" app where all paths are relative. The main issue is that the "ember" script was meant to be run if you wanted the relative paths functionality, while the "ember.bin" executable was meant to be run if you wanted the standard behavior (for example if you compile from source yourself). This is of course unnecessarily confusing, so it's now changed so that the "ember" command will always work.

Buggers

2007-06-24T08:43:00.000+02:00

I've been spending the last couple of days finding and fixing smaller bugs in the Ember codebase. The client itself is pretty solid, I very rarely get crashes, and when I do I can most often fix the underlying problem right away. But then there's the issue of shutdown... It's so much easier to not clean up and instead just exit the application, because it doesn't really matter if there's a segfault when shutting down; the memory is about to be reclaimed anyway, right? Well, there's the problem with sockets and other data structures that aren't that nicely shut down when the app crashes. And there's the issue with shutting down partially and then restarting, like disconnecting from a server and the reconnecting. And so on...
As I wrote in this entry I started Ember as a project to teach me C++, so initially I didn't care much about proper resource handling, because there were so many other things that I had to wrap my head around. But not so any more. Which means that I have to fix all of those lingering problems with resources not being released as they should. The most common culprit is Ogre, CEGUI and Lua. The interdependencies of these libraries forces me to be very careful about how I both acquire resources and release them.

I'm thinking it's about time 0.5.0 was released. The thing is that I always dread doing releases because it means I have to put development on hold for usually more than a week while I try to package Ember for both Linux and Windows. The latter one is always the most time consuming, since I do all development on Linux and then port to Windows at release time. So far I've used gcc and the automake toolchain for Linux and Visual Studio for Windows. This means however that I always have to update the Visual Studio project every time I do a release, and more worrying; that I have to work around all the missing features in msvc (compared to gcc). I'm thinking that it might perhaps be better to set up a mingw system on Windows so I can use gcc for both platforms. If I can get the autotools to work on Windows that would be even more excellent.

Move dammit!

2007-06-19T12:00:00.000+02:00

Yesterday I implemented the ability to double click on the terrain and have the avatar go there. It's something that users have been requesting for some time. I've put it on the back burner because I'm myself so used to moving by using mouselook and wasd, and I was hoping that it would be one of those simple things that aspiring developers could tackle to better getting to know the code. All of the functionality already existed in both Eris and Cyphesis, I just hadn't done the connecting. So in the end it took no more then 20 minutes to implement it. Double clicking in the world now makes the avatar move.
The mouse picking system is very flexible and extensible. This flexibility is what allows us to react differently when the user clicks on an entity to when clicking and dragging a world base point in terrain editing mode. It's also what makes an addition such as this so easy. Ember development sometimes feels frustrating because so much work has been put into making base architecture and functionality, without any obvious "external" improvements. But it feels good that at least some of that work pays off. :)

Collisions

2007-06-18T19:57:00.000+02:00

Ember uses Opcode (coupled to Ogre through OgreOpcode) for mouse picking. Opcode is a generic collision detection library and could probably be useful for many different things, but currently it's only used for mouse picking. The version used is however many moons too old and also a WIP prerelease. I've been itching for some time to upgrade to a more modern and robust version. One of the things that I've missed is the ability to do raychecks without backface culling. Not being able to do that results in not being able to pick the market stalls from behind, since the fabric is a single face where the material has backface culling turned off.
This weekend I finally took care of this nagging issue by upgrading to the latest version of OgreOpcode. It took 2h to upgrade the code, and then 4h of hunting down why no symbols could be found after the update. It turned out that it's not a good idea to use "," as delimiter in Makefile.am files (instead of the preferred " " version). Ah, the joys of stupidity.
Opcode is however not actively developed, and uses some dodgy "optimization" techniques, resulting in the code being nigh unreadable. It's working for me right now though, but if we in the future get cramped by it it might be wise to pursue other options such as Bullet.

Delisting?

2007-06-16T09:25:00.000+02:00

By chance I went over to the wikipedia entry on Worldforge to see what it said. Not so good things it turned out. Apart from devoting a full third of the entry to "Problems", it said nothing of the different clients, the feature set of servers, the media repository and so on. Instead there's outdated and totally incorrect information (branching out into other parts of entertainment?).
What's worse however is that there's a notice now that there's not enough "notability" and that the article will be taken down unless it's filled with more information. Now, I've previously haven't wanted to touch the Worldforge entry on Wikipedia because I understand their position on preventing bias and avoiding original research. As one of the developers of Worldforge I'm quite biased. And therefore I've felt that it would be better if some of the regular Wikipedia editors gathered the information needed and updated the entry.
But it obviously doesn't work that way. And faced with the prospect of disappearing from Wikipedia I had to update the entry with some correct information. I added an software info box and added a screenshot of Ember, but I'm not sure how much more I should add.
I don't want to astroturf, and I fully understand that Wikipedia wants to keep the database clean. If Worldforge isn't notable, well then it shouldn't be there, just as so much other information that's added to Wikipedia every second. But I with my full heart don't believe that, and I think it's more of a case of a neglected entry. There's so much to write about Worldforge, about the clients, the tools, the servers, the AI, the Atlas protocol, the Mercator terrain generation etc.

Adapting

2007-06-13T11:21:00.000+02:00

Currently I've been working on the Entity Editor widget. The idea is to allow for world builders to edit every aspect of entities in the world.
The properties of an entity are very loosely defined. Apart from some standard attributes, such as position, id, name, orientation etc. WF puts no specific constraint on the attributes. So, for example, it's possible to create an attribute with the name "age" which is a map of two strings. Such an attribute would of course make no sense. So I'm trying to present some more user friendly representations to the user of the most common attributes.
An example can be see here, editing a settler. With the exception of "bbox" all attributes are shown using the low level representation widgets. So for example, the "guise" attribute is a map, which contains two other maps, named "material" and "mesh". The "material" map in turn contains three string values, identified by the names "dress", "head" and "pants".
The bbox on the other hand is presented with a "Size" widget. In essence, the "bbox" attribute is just a list of six float values, but using the regular List widget to present that would just be cumbersome. In addition, there's no point in removing or adding another value to the list, something that the List widget allows. Therefore, a Size widget is used instead.
So far I've created specific widgets for Size, Orientation, Position and 2DPosition. The latter one is used in areas where the area is defined by a list of 2d points.
By adding more specific widgets I'm hoping to make it much more simpler for world builders to update the world.

Expanding areas

2007-06-12T10:23:00.000+02:00

Well, turns out there's still some work to be done in the area department. Mercator has some missing methods in the area handling code. For one thing, there's a Mercator::Area::setLayer method signature, but no implementation. I don't know however if it makes much sense for an area to change it's layer once it's been created; perhaps that's why it's missing?
In addition, when an area is added to the terrain, through Mercator::Terrain::addArea, there's no corresponding removeArea method. Writing such a method is however not completely straight forward since when an area is added to the terrain, it's in turn added to only those Segments that it intersects. One would however think that when removing an Area all one has to do is to iterate over all Segments and remove it when found.
The third problem is that since an area only is added to those segments that it intersects, problems arises when an area is either moved to another segment, or changed so that it now intersects another segment. The effect is that no shading will occur on the new segment. The solution for this would be to add a new method Mercator::Terrain::refreshArea(Mercator::Area* area) which will iterate through all segments, and remove the area if it no longer intersects the segment, and add it where it does.

Also, I had to restart cyphesis since all the NPCs had died. They start out at full health, but since they never eat, they eventually starve to death. When restarting I took the opportunity to update to the latest cvs version. I also added some boulders to the landscape. We have some very nice boulder meshes that Jay did, along with a nice texture, but for some reason it's not used. A normal forest landscape is filled with all kind of boulders. The result can be seen here. Note that the shading of the boulders for some reason is a little bit strange; I'll have to investigate why that is.

The areas of my expertise

2007-06-11T09:49:00.000+02:00

I went away over the weekend so I didn't have much time for Ember, but I did manage to fix another one of those things that have been bugging me for a while.
While Ember has supported areas for a while, it hasn't allowed for dynamic updating, either of changes to the area itself, or the position of the parent entity. So yesterday I set out to try to rectify this. Turned out that it was really simple to make that happen. Some new events on the TerrainArea class, some listening glue and some TerrainGenerator logic later it all worked. It's now possible to edit an area on the fly and see the updates directly. The area will also be moved when the entity is moved.

In addition to that, I finally found and fixed a bug in the EntityEditor which made refreshing take ~5 seconds. That's quite the leap from the 0.02 seconds it used to take, so I was a bit perplexed by what might cause it. Turned out that I had accidentally created a int-wrapping loop, which looped from 1 to 2^32. Oh well...

In other news, Ogre 1.4.2 was released today. No major changes as far as I can see, but this version will contain a fixed OGRE.pc file, which will make Ember's configure.ac work as it should.

Back from vacation

2007-06-08T10:22:00.000+02:00

I haven't posted to this blog in a long time, mainly because I didn't have anything to say and I didn't think that anyone would be interested in the things I didn't say either.
However, I'll try to rekindle the blogging flame by posting more about my work on the Ember project. I feel that it would be nice to have a record of the development process, mainly for myself. So...

I'm currently preparing an Ember 0.5.0 release. The main reason for the jump from 0.4.3 is that I'm now using Ogre 1.4 and CEGUI 0.5.0. Mainly the latter includes some main improvements.

Apart from those upgrades, I've been focusing heavily on creating intuitive authoring widgets. Ember, and indeed the whole Worldforge project, has been lacking good authoring tools. Currently, if you want to set up your own world, you need to hack the python and xml files of cyphesis by hand. And then there's no easy way to alter existing entities. Sure, there's the cycmd tool which allows for console type editing, but that's not really a smooth experience.
Thus I've now incorporated a new framework for editing entities into Ember. It's using adapter classes to bridge between CEGUI and Eris, with Ember handling the connections. I would really have wanted to go MVC all out, but there are some things in CEGUI that prevent that. Instead I've tried to take it far enough.
One problematic area has been to properly divide the structure into C++ and Lua code. The former is much more powerful, and adds compile time checks, while the latter is more dynamic and allows for run time editing. The overarching idea is that all layout related code should be handled by Lua, while most of the other logic should be handled by C++. I think I've succeeded in part, but there are still some rough edges. And it's never a clear cut case where some part of the functionality should go.

Yesterday I needed to take a break away from the work on the editor (mainly to get my bearings on where the code is heading) and work on some smaller issues. One of the things that have been bugging me for some time is that wielded entities aren't correctly aligned. This is because the hand bone to which wielded entities are attached to isn't really meant to be used for attaching entities, and thus attached entities needs to be adjusted for that. I've planned to add a rotation adjustment setting to the modeldef xml format for some time, but I never had the time. I was also hoping that this would be one of those things that would be excellent as an introductory task for a new developer. But sad to say so far no new developers have contributed anything to the project. It's been a one man show since it's start, with the exception of some greatly appreciated help from Hagen with the console.
So yesterday I finally took care of this issue. It's now possible to specify the rotation needed to get a wielded entity to align properly. The only thing that's missing now is an easy way to edit this in the ModelEditor.

My Zaurus doesn't love me

2005-02-28T14:20:00.000+01:00

About a year ago I bought a Zaurus SL-C700. Tt's a truly wonderful machine; without doubt the most powerful PDA you can get your hands on (well, actually, the Zaurus line is, the SL-C700 model is a bit dated, though I have yet to find a machine in Sweden which can match this two year old model when it comes to performance). And there are a _lot_ of apps to it. With something like the pdaXrom it's extremely powerful, since all X applications are available. So you do not only get access to PDA applications, you also get access to the whole Linux "product catalog". And with a 640x480 screens, it works very well.
My problem is that I suspect that there's some kind of hardware malfunction with my unit. It often crash without warning, often many times a day. And applications will often lock up or make the whole system slow, util it all just crashes. Very, very sneaky. And since I've bought it from Japan, I can't really just send it back. It's really annoying, because it prevents me from using the PDA.
Hmm, perhaps I should get one of these instead?

Regression testing, the new black

2004-10-27T11:53:00.000+02:00

It's taken me at least three days to convert Ember to Eris 1.3. In the meantime Ogre 0.15.0 was released. And glory! Ember compiles without modification against the newest Ogre. I'll try to stick to the point release as far as I can. It itches to change to PagingTerrainManager though...
Back to the Eris woes though. For some reason it's impossible to change to orientation of a Ogre node which is a child of another Ogre node. The solution is to let all nodes be children of the root node. But this is just an interim solution. Took me some days to find out though. I still can't understand how the change in eris affects ogre in such a way. Is has probably something to do with some code path not being follow in the previous Eris. I don't know.
Damnit!
And that's why we all love regression testing, and hate the lack of it.

Anyways, here's a screen of Ember in it's current (pre Eris 1.2, pre Ogre 0.15.0) state.