Saturday, June 19, 2010

Simplify tolua++ with autotools

In Ember we use Lua for our scripting needs. The bindings to the C++ parts of the client are provided by the tolua++ library. Tolua++ works by generating C++ source code from .pkg files, which are simplified .h files. This works out extremely well; the developer only needs to copy the .h file to a similarly named .pkg file and remove those methods and fields that shouldn't be exported. The command line tool "tolua++" is then run to produce the C++ source.

Previously in Ember we handled all of this by keeping the generated C++ source checked into the source, and requiring that each developer had to run the tolua++ command to regenerate these sources each time a lua binding was added or changed. This however had a couple of downsides, chief amongst them that the code generated differs a bit depending on the version of tolua++ used. It also resulted in some often very large Git commits of generated code, which tended to pollute the Git history.
A much better solution would be to instead automate the generation of the binding source, and bake it into the normal build system. Fortunately the Autotools already provides all the facilities for making this happen.

An example of how this is handled in Ember though a Makefile.am can be found here:

SUFFIXES: .cxx .pkg .lo .la .cpp .o .obj

.pkg.cxx:
cd $(srcdir) && TOLUAXX=${TOLUAXX} $(abs_top_srcdir)/scripts/update_lua_bindings.sh `basename $@ .cxx` `basename $@ .cxx`.pkg $(abs_builddir)/`basename $@` $<

INCLUDES = -I$(top_srcdir)/src -I$(srcdir) -I$(top_builddir)/src -DPREFIX=\"@prefix@\"

noinst_LIBRARIES = liblua_EmberServices.a
liblua_EmberServices_a_SOURCES = EmberServices.cxx

CLEANFILES = EmberServices.cxx
TOLUA_PKGS = ConfigService.pkg EmberServices.pkg IInputAdapter.pkg Input.pkg InputService.pkg LoggingService.pkg MetaserverService.pkg ScriptingService.pkg ServerService.pkg
EXTRA_DIST = $(TOLUA_PKGS)
EmberServices.cxx: $(TOLUA_PKGS)

The files references in TOLUA_PKG are the .pkg files which define the bindings. These will be fed through the update_lua_bindings.sh script to generate the file EmberServices.cxx, which is then compiled and added to the liblua_EmberService.a archive. Note that we need to add EmberServices.css to the CLEANFILES variable to make sure that it's deleted when we're cleaning up. Since it's generated through the tolua++ tool Automake can't keep track of it itself.
The update_lua_bindings.sh script looks like this:

#! /bin/sh
#tolua++ will for some reason translate "const std::string" into "const,std::string", so we need to remove these incorrect commas from the final code
#some versions will also for some unexplainable reason not correctly remove the tolua specific directive tolua_outside, so we need to clean that out also
#We'll also replace the inclusion of "tolua++.h" with our own version which has better support for building on win32.
echo "Updating lua bindings."

#If the TOLUAXX environment variable isn't set default to using the command "tolua++".
if [ x${TOLUAXX} = x ]; then
TOLUAXX=tolua++
fi
${TOLUAXX} -n $1 $2 > $3
grep -q '** tolua internal error' $3 && cat $3 && exit 1
sed -i -e 's/const,/const /g' -e 's/tolua_outside//g' -e 's/tolua++\.h/components\/lua\/tolua++\.h/' $3

This script basically runs the tolua++ command, as defined in the TOLUAXX environment variable (with "tolua++" as fallback) and then applies some replacement to fix some issues we've been having with the generated code.
The TOLUAXX environment variable is set at configuration time. The default is "tolua++", but we'll provide the option to use an alternative command. Our acinclude.m4 file has this snippet (which is called from configure.ac):

AC_DEFUN([AM_CHECK_TOLUAXX],
[
AC_ARG_WITH(tolua++,AS_HELP_STRING([--with-tolua++=CMD],[Tolua++ command (default=tolua++)]),
toluaxx_command="$withval", toluaxx_command="tolua++")

AC_CHECK_TOOL(TOLUAXX, $toluaxx_command)

if test "x$TOLUAXX" = "x"; then
AC_MSG_ERROR([Could not find a working tolua++ command (tried '$toluaxx_command'). Use the --with-tolua++ switch to set the proper command to use.])
fi
])

This setup will allow the script bindings to be generated at compile time, but only the first time compilation occurs, or if any of the .pkg files have changed. More examples of how this is used can be found in the Ember sources.

Tuesday, June 15, 2010

Using automake to generate ChangeLog from git

When deploying a GPL application it's required that you provide a change log containing information on all the changes to the code base. This file is normally named ChangeLog.
In the olden days, before today's fancy revision control systems, people often had to edit the ChangeLog files themselves for each and every change to the code. This is what we did in the Worldforge project when we used CVS for our source code needs.

However, for the modern developer used to distributed revision control systems such as Git this seems archaic. Why provide a separate list of all changes when the Git repository already contains a complete log? Instead of keeping the ChangeLog file updated with each commit, wouldn't it make more sense to generate it from the repository history when a new release is made?

When a new release needs to be created in a project using the autotools the "make dist" make target is invoked (or in reality the "make distcheck" target). This will package the source and produce tar archive of it all. The Makefile target "dist-hook" is provided to allow for us to hook into this process. So what we want to do is to provide some shell scripting which generates a ChangeLog file from the git history. It will look something like this:
cd ${top_srcdir} && git log --stat --name-only --date=short --abbrev-commit > ${distdir}/ChangeLog
This will generate a log of all the changes, in a condensed format not unlike the ones suggested by GNU.

We could be done here, but there are some things that could need improvement. For one thing, this will only work when the "make dist" target is invoked in the original git directory. Some distributions might want to take the dist release, add some distro specific patches, and then make a new dist from that. It would therefore be better to add some logic which can recognize whether there already exists a generated ChangeLog, and if so won't try to generate a new one from the git log. This will need some more logic so we'll be splitting this functionality out to a separate script and call that from the dist-hook target. We'll also add a ChangeLog with exactly one line:
This file is autogenerated from the Git history when a the "dist" make target is invoked. If you find this file in an official release something has gone wrong and you should contact [maintainer_email]. It needs to be exactly one line long in order for the ChangeLog generating script to work.
Our script will check the length of the ChangeLog file. If it's exactly one line, we know we should generate it from the git log, else we know it's already been generated and we shouldn't do anything. Our script (generate-ChangeLog.sh) will look something like this:
#! /bin/sh
top_srcdir=$1
distdir=$2
if [ `cat ${distdir}/ChangeLog | wc -l` = "0" ]; then
chmod u+w ${distdir}/ChangeLog && cd ${top_srcdir} && git log --stat --name-only --date=short --abbrev-commit > ${distdir}/ChangeLog
fi
The dist-hook will now look like this:
dist-hook:
sh $(top_srcdir)/generate-ChangeLog.sh $(top_srcdir) $(distdir)
Now we're pretty set. With Worldforge there's however still an issue with the older CVS ChangeLog. When the source was migrated from CVS to Git the log messages from CVS were directly imported into Git. These however were in a quite verbose format with the date and authors included. Since this data also is available as meta data in the Git repo, the result will be that these log entries will have a lot of redundant data, making the ChangeLog both very large and confusing. To prevent this, we'll store a copy of the old ChangeLog with the CVS entries with the source (as ChangeLog-CVS) and combine this with the Git log history. After looking at the Git history we can see that at the commit with id "f12012e7616c191a8926432faf866c8e43854062" marks where the transition from CVS to Git happened. We'll also replace the ChangeLog-CVS with a notice about it's previous use (as it's not needed anymore, but must be present in the dist) Our script will then look like this:
#! /bin/sh
top_srcdir=$1
distdir=$2
if [ `cat ${distdir}/ChangeLog | wc -l` = "0" ]; then
chmod u+w ${distdir}/ChangeLog && cd ${top_srcdir} && git log f12012e7616c191a8926432faf866c8e43854062..HEAD --stat --name-only --date=short --abbrev-commit > ${distdir}/ChangeLog && cat ${top_srcdir}/ChangeLog-CVS >> ${distdir}/ChangeLog
chmod u+w ${distdir}/ChangeLog-CVS && echo "This file was needed for generating the proper ChangeLog as an aggregate of the code held in git and older code in CVS. It's now empty, but needs to be included in the source distribution to not upset automake." > ${distdir}/ChangeLog-CVS

fi
And now we're done. We have an automated system which will generate the ChangeLog from a combination of both the old CVS provided ChangeLog and the Git log history. It will also make sure that if the "make dist" target is run again on an already generated dist the ChangeLog will be left as it is. Just remember to also include all relevant files in the EXTRA_DIST target of the Makefile.am.
Up to date versions of how this is used in Ember can be found in the Ember repository.

Thursday, May 20, 2010

Freedom of Speech

Today I'll be posting about something other than FOSS game development, something far more important (yes, such a thing exists!). Today's topic is Freedom of Speech and how we never must take that for granted.

I live in Sweden where the last couple of years there has been a couple of incidents where islamic fascists have tried through violence and threats of violence to impose their world view on other people. Central to that world view is an active aggression against the principle of Freedom of Speech which underpins our whole democratic and parliamentary system. The most prominent examples are the threats and active murder attempts on the Danish cartoonists who drew pictures of Muhammed, or the threats made against South Park. In Sweden the artist Lars Vilks has been both physically assaulted and subjected to arson as a result of his drawing of Muhammed in 2007.

It's easy to take the Freedom of Speech for granted and to think that those people that want to bring it down only operate in chaotic countries such as Pakistan or Somalia. That is unfortunately not the case, as shown in this video of Lars Vilks presenting a lecture on the topic of "Art and Freedom of Speech". Vilks suffered some bruises and a broken pair of glasses. The day after someone tried to kill him by setting his home on fire.


Do not be tempted to draw the conclusion from this video that Sweden, or any other European country is under any threat of "being overrun by Islam" or similar. That is absolutely not the case; it is in fact the opposite. When faced with a stable society and a high standard of living most people tend to distance themselves from religion and totalitarian ideologies, even if it might take a generation or two.
But it does show that Freedom of Speech is something we never can take for granted, and not something we can barter with. It is the firmament on which our whole society is built on, and we cannot let fascist or totalitarian ideologies or groups make us infringe on it one iota. As long as people feel threatened by violence for expressing their opinions there is no real Freedom of Speech. For every Lars Vilks who has the guts to criticize Islam and as a consequence gets assaulted and beaten, there are thousands of other people who hold their tongue. Some of them in Western countries such as Sweden, but most of them in countries where Freedom of Speech already is curtailed, such as Iran or Pakistan.

Therefore I'm today celebrating the annual Draw Muhammed Day by presenting my own drawing of Muhammed, the founder of Islam. By providing my own drawing of Muhammed I'm showing my support and solidarity with the artists currently under death threats. Note that the drawing is pretty tame (and badly drawn); I know this blog is syndicated so I'm putting a little restraint on myself. To more properly show my complete and total outrage of having islamic fascists trying to impose their warped totalitarian world view on me it would be more proper to show Muhammed being raped by a pack of dogs. But I'll wait with that picture for another day (not that the Internet is lacking in such depictions).


Muhammed, shown here during one of his schizophrenic episodes trying to come up with another verse for the Qur'an.

Friday, May 14, 2010

Ember 0.5.8 released

I've just released Ember 0.5.8. The official release announcement can be found on the main Worldforge site.

The main new feature is the multi threading framework which I've written about before on this blog. It's a pretty Big Thing, as it we're so utterly dependent on having dynamic data.
Another thing which went into this release, but which won't be apparent for any user, is a large restructuring of the media repository. It's mainly Jayr's work, but I've also helped out. Basically it amounts to defining common naming and structuring standards for the media repository so that it becomes easier to both contribute and use the assets therein. I've updated some of the media guidelines to reflect this.
And for the first time in a couple of years we now provide win32 binaries. Getting Ember to build on win32 (through msys) was a real chore, but the plan is to keep the code base compatible enough so that each subsequent release of Ember will have win32 binaries.



There's been some other developments in the FOSS game world too.
Ryzom recently released all of their source code under the GPL and all of their media under the CC-BY-SA license. This is a huge boost for FOSS gaming, as there's now a boatload of new media assets available. A quick comparison of the Ryzom media contra the Worldforge media:
* Ryzom has a much more assets.
* Ryzom textures are smaller (between 256 and 512 pixels whereas most of Worldforge's are 1024 - 2048 pixels).
* Ryzom textures are only available as png whereas Worldforge provides source assets for all textures (usually .psd).
* Ryzom only provides diffuse textures whereas Worldforge usually provides specular and normal/heightmap versions too.
* Ryzom assets are made in a particular art style, whereas Worldforge has opted for a natural style. This is however just for the Mason world. I would love to see a world using Ember and Cyphesis with the Ryzom assets.

Some have noted that the source meshes in Ryzom only are available in .max format. That's no big deal though as they can be converted to .blend (a process which is always underway). As long as the source is available it's always possible to convert it; it's harder when the only available asset is an exported format. (Worldforge provides .blend files for most of the meshes, though we have some .max versions too.)

Another big thing is the Indie Humble Bundle, which is a collection of six excellent indie games, available for a user set price. It's truly an excellent deal, especially since part of the money paid is donated to charities. And not only that, since they so far received more than 1 million USD they've now opened up the source code for four of the games as FOSS. It's really a remarkable thing.

Thursday, April 01, 2010

New animals from Jayr

This is a short post about some of the new media recently added by our extremely talented artist Jayr to the Worldforge media repository.
Jayr keeps his own blog (unfortunately missing rss) over at Hourglass3d where he posted about some of the new animals he added to Worldforge.
Head over there and take a look.

Tuesday, March 30, 2010

GSoC student applications are now open

Google has now opened the student applications for Google Summer of Code 2010.
This year, in order to help you figure out if you have the skills needed to work on Ember, and in order to help us to filter out non-serious applications, we ask students who are applying for ember to find and fix two errors in this special code drop. Also see the Ember prerequisites on this.
We encourage students to apply as soon as possible, so that there's time for us to give feedback and work out the details. The sooner the better.
If you have any questions please use the general mailing list.

Monday, March 22, 2010

Worldforge in Google Summer of Code 2010

Worldforge has for the third year in a row been accepted into the Google Summer of Code 2010 program. This means that students now have an opportunity where they can spend the summer writing code for free and open virtual worlds rather than flipping burgers and mowing lawns.
Student's applications will open on March 29, but if you are interested in applying the sooner you contact us the better.
We prefer that students use the general mailing list for discussing project ideas, or send mail to the mentors directly.
More information about the program, and a list of project suggestions can be found at the Worldforge GSoC page. Note that the suggestions are just that; suggestions. Any interested student is encouraged to also build on the suggestions, or propose their own project.

Wednesday, March 10, 2010

New world

For quite some time we've had our default world Mason. While Worldforge is a project aimed to provide basic tech for any kind of virtual world, we still need to have a showcase world in which we can try out different things. The default world has had some issues however.
  • It's not using much of the new media provided by Jayr, and the media used is often outdated.
  • It's confusing for newcomers. There's not clear direction of what new players should do in the world.
  • The steep mountains surrounding the area are hard to manage in a client, as the steep cliffs makes level-of-detail optimized rendering hard.
  • It's a quite small world.

All of this is changing as I've been working on a new default world. There are a couple of improvements to both the server and client which allows for this.
  • Persistence on the server. As cyphesis is a work in progress, it's possible that bugs are uncovered when developing the world which will cause a crash. Having the whole world disappear when the server crashes is quite unproductive. With constant database persistence there's no risk of that.
  • The ability to take a snapshot of the world on the server. This snapshot is saved as an xml file, which can then be added to a versioning system. This way it's possible to try things out, and revert to older versions if needed.
  • Freezing the world on the server. When designing the world it's important that it doesn't change. That means that trees shouldn't grow, animals shouldn't move, living things shouldn't starve. Through a new feature which allows the server to be frozen any world author can first design the world, and once that's done unfreeze it.
  • Advanced entity editing on the client. It's important that it's easy to edit all aspects on the fly on the client. The advanced entity widgets available in Ember allows for this. Aspects such as terrain areas or terrain modifications are easily editable through a point and click interface.
  • Easy entity creation. Just as important as editing entities is making sure it's simple and quick to create new ones. The Entity Recipe system in Ember allows for entities to be created with point and click, with a powerful scripting environment in the backend.
The new world is therefore created in a series of steps. The first step involves working with a frozen world, where all entities are placed, and all terrain is sculptured. Once the world is complete it needs its npcs and creatures to be given behaviour and goals. This should be done using the existing python scripting in cyphesis. Once that's done the world should be available as both an xml file defining all entities and an accompanying python script file which defined behaviour.

When creating the new world there are a couple of goals I've had in mind.
  • Use as much of the new media as possible, while clearing out any out of date media (i.e. media which simply doesn't look so good any more).
  • Provide a large enough world that it will take some time to traverse it.
  • Make the terrain friendly to any clients. That means avoiding too steep cliffs as that brings havok to any LOD mechanism.
  • Provide a clear path for any new player. There should never be any confusion of what the player is supposed to do once he or she has entered into the world.
  • Provide new game play. Apart from our current pig herding and fishing game play we have many ideas of additional tasks. These tasks also require a larger terrain, and new media.
So far things have gone very smoothly. Working with a much larger world is also good for Ember since it really pushes the engine in terms of memory usage and performance. Our current world is simply too small to really take advantage of the many dynamic paging features in Ember.

Below are a couple of screenshots of the work in progress. Some of them show the new starting area, which is on a peninsula with a clear path to the main area. Together with a couple of helpful signs and a greeting npc it should be much more clearer for any newcomer where they should go.