Support us on Patreon to keep GamingOnLinux alive. This ensures all of our main content remains free for everyone. Just good, fresh content! Alternatively, you can donate through PayPal. You can also buy games using our partner links for GOG and Humble Store.
We do often include affiliate links to earn us some pennies. See more here.

An Update On The Open Source Project 'Xoreos' Concerning Dragon Age

By - | Views: 15,269
xoreos is a FLOSS project aiming to reimplement BioWare's Aurora engine (and derivatives), covering their games starting with Neverwinter Nights and potentially up to Dragon Age II. This post gives a short update on the current progress.

Note: This is a cross-post of a news item on the xoreos website.

Yet further down the path of getting all targeted games to show areas, it seems like I reached the end with Dragon Age: Origins and Dragon Age II. Similar to my posts about my progress with Sonic Chronicles: The Dark Brotherhood (Part 1, Part 2, Part 3), The Witcher, Jade Empire and Neverwinter Nights 2, this will be a short description of what I did. This time: Dragon Age: Origins and Dragon Age II.

Models

Lucky for me, the Dragon Age model format is reasonably well documented in the Dragon Age toolset wiki. tazpn even created standalone model viewers for Dragon Age: Origins and Dragon Age II, and released them with sources under the terms of the 3-clause BSD license. :)

And since the model format is based on GFF4, missing pieces of information are relatively easy to decipher too. So I quickly had a loader capable of reading the skeleton whipped up for both Dragon Age: Origins and Dragon Age II (since they are nearly identical in format).

image image
With a bit of fiddling, the meshes were there too. There's two types of meshes within the models: static meshes, directly hanging at one specific bone, and dynamic meshes that include weights for several bones for each vertex. Similar to models in Sonic Chronicles, this would deform the mesh according to those weights when the bones are animated. Unlike Sonic Chronicles, the default vertex positions of those meshes create a valid, unanimated pose. This means I could just completely ignore the bone weights for now, and load the meshes as if they were static. In the future, a vertex shader would combine those weights with the bone position to create the fully animatable model meshes.

image image image
Only thing missing now were the textures. For that, I needed to read the MAO (material object) files, which contains the material file (MAT), various textures (diffuse, lightmap, etc.) and a number of optional parameters. The material file in turn contains several different "semantics", which is basically the name of a shader and how to map the MAO values onto the shader input. The original game takes all these, looks for the most fitting semantic in the material file (depending on number of parameters, graphics card capability and user settings), and then tells the graphics card which shader to use to render the mesh.

Now, since we don't actually support any shaders yet (and we can't use the game's Direct3D shaders directly anyway), we simple read the MAO (which can be either in GFF4 or XML format), take the diffuse texture, and apply it to the mesh directly.

image image image
Campaigns

With the models done, I turned to reading the Dragon Age: Origins campaign files. A campaign, that is either the default single player campaign (which is defined in a CIF file), or a DLC package (with both a CIF file and a manifest.xml) that doesn't extend another campaign (those would be add-ons).

There's several caveats involved here:

First of all, most of the DLC packages are encrypted. The original game queries a BioWare server for the decryption key, asking whether its a legitimate copy. While the encryption method is known (Blowfish in ECB mode), xoreos does not include any of the keys. So the only campaigns apart from the main one loadable right now are the unencrypted ones, namely Dragon Age: Awakening, and any custom ones you might have downloaded (including the PC Gamer promo DLC A Tale of Orzammar).

Then, we don't load any add-ons. So no Shale or Feastday Gifts, even if they weren't encrypted (which they are). It's not like xoreos could do anything with them yet anyway.

Finally, we have no way to install .dazip packages yet, so those need to be installed using the original game for now, or manually extracted and put in the right places. In the future, something that install them would be nice. Or maybe we could support loading of packed .dazip files, but that could be slow.

In either case, I implemented the loading of standalone campaign files.

Areas and rooms

Next up were areas (ARE) and environment layouts (ARL) with room definitions (RML). The ARE contains dynamic room information, like what music to play, and the placeables and creatures (more of those later). The ARL defines what rooms are in the area (as well as pathing information, weather, fog, etc.), each of them being a RML file with models. They are all, again, GFF4 files, making them nice and easy to understand.

image image image
There was one problem, though. The orientations of the models were given in quaternions, and as I said in the blog post about my The Witcher progress, a combination the automatic world rotation xoreos does, and our Model class wanting Euler angles instead leads to them not being correctly evaluated for whole models.

I was getting sick of that not being correct. I bit the bullet and removed the world rotation (which meant I had to rejigger the placement code in all engines, as well as the camera system, which was especially painful in Sonic Chronicles). And then I changed the Model class to take axis-angle rotations instead; those can be more easily calculated from quaternions, and can still be directly fed into OpenGL.

As a result, the area room models in Dragon Age: Origins were correctly oriented. And the placeable models in The Witcher as well.

image image
You might notice that the ground mesh in outdoor areas looks very blurry and low-res. That's because the original game doesn't specify a single texture for those, but instead combines several textures together in a shader. We don't support that yet, so instead we apply the replacement texture of the lowest LOD which is normally used for meshes that are far away.

Placeables

On to the placeables, the objects within areas. They are defined within a list in the ARE file (giving position, orientation, name, etc.), each with a template. The template is a UTP file, a GFF3, that contains common properties for all instances of this placeable. This includes an appearance, which is an index into a GDA (a GFF'd 2DA, a two-dimensional table), which specifies, among other things, the model to use.

So far, so usual for BioWare games.

One difference, though. In the Dragon Age games, the GDA files do not stand alone. Instead, each is a combination of potentially several GDA files with the same prefix (defined in m2da.gda). This is used for DLCs, which then can simply add rows to a GDA, instead of overwriting the whole file. Consequentially, the appearance index is not a direct row number, but corresponds to a value in the "ID" column.

A bit fiddly, but still relatively easy to implement.

image image
Creatures

The creatures were more difficult. There's several types of creatures: type S (simple) are just a single model; type H (head) are split into a body model and several models for the head (base, eyes, hair, beard); type W (welded) are similar to H, but already include weapons in the body model; and "P" (player-type) creatures are segmented into head (with base, eyes, hair, beard), chest, hands (gloves) and feet (boots).

image image image
Moreover, creatures of type P also switch model parts depending on the equipped items. So armor changes the chest model, gloves and boots change the hands/feet models and a helmet replaces the hair. Which models to use depends on several factors, and includes look-ups in several different GDA files, as well as UTC (creature template) and UTI (items) files.

Another problem is the tinting. The original game uses a shader to tint hair, skin and armor parts custom, user-selectable colors. To do that, their textures just contain intensity values in two color channels, while the two other channels are used as a bump map and something else (which I'm not sure yet). If we just apply the texture to those body parts, they are suddenly mostly transparent. To work around that for now, we manually modify each of those textures to remove the transparency. That leaves the weird coloring, but you can at least see all the body parts then.

image image image image image image
Dragon Age II

I then applied all this to Dragon Age II. Just a few minor changes to the resource loading was necessary, and nearly everything worked out of the box.

image image
Only the P-type creatures needed a bit more work, since how the body part models are constructed changed.

image
Similar to Sonic Chronicles, Dragon Age II is also missing many of the GDA headers; they're only stored as CRC hashes. With a dictionary attack, I did manage to crack about half of them, but that still leaves about 450 unknown. Something to watch out for in the future.

Music

I also investigated how music works in the two games. Dragon Age: Origins uses FMOD, and Dragon Age II uses Wwise. Both work similarily: the area specifies an event group, and the scripts then tell the library to play a specific event list from that group at certain times. The library does the rest, evaluating the events in the event list (which range from "play sound X", over "set volume to Y", to "add Z% reverb"). And while I do have adequately licensed code to read the sounds from both libraries' soundbanks, figuring out the events is a massive undertaking. And we don't have a script system for the Dragon Age games in place anyway, so this is nothing that can be done right now.

What's next

So... All games xoreos cares about now show areas. What's next, then?

Well, first of all, I'd like to do some cleanup of the engines code. Sync them up, make them more similar to each other. Right now, many things are done slightly different in each engine, because the games changes something around and the old concept suddenly didn't fit anymore. If possible, I'd like to unify the concepts again.

There's also a few potential portability issues I want to investigate. For example, I read that using fopen() on Windows with filenames containing non-ASCII characters won't work at all. Instead, I'll probably have to change xoreos' File stream class to use Boost's fstreams, and convert our UTF-8 strings to UTF-16 on file open. I hope that's something I can test with Wine, otherwise I'll have to bug somebody with access to a real Windows.

After those things have been cleared, I'd like to prepare for our very first release. I plan to include both xoreos and xoreos-tools, with sources (of course) and pre-compiled binaries for GNU/Linux, Mac OS X (>= 10.5) and Windows, each for both x86 and x86_64. I have cross-compilers for those, and they all should work. Yes, xoreos is still not really useful for end-users, but a release can't hurt, and might give us some publicity and/or get people interested. Who knows.

I could use some testers for those binaries, though, to make sure I get the library dependencies correctly. And that the GNU/Linux binaries work on other systems than just mine.

I'm also open for other platforms. Would it make sense to have xoreos pre-compiled for Free/Net/OpenBSD? Other architectures than just x86/x86_64? Anybody with insights there, and capable of compiling those binaries (or pointers to cross-compilers), please, contact us. :)

As for how to continue the actual xoreos development, I think it would be useful to transfer the script system that's currently hooked up to Neverwinter Nights onto the other engines. It would need to be rewritten, though. When I first wrote it, I wanted to have engine functions with signatures that mirrored the signatures of what the scripts call. I couldn't get it to work, though, and settled on a context that contained an array of parameters. For some reason, I still used boost::bind for all the functions, which, at that point, was not necessary. boost::bind compiles really, really slow, and so now the files containing the Neverwinter Nights engines functions take ages to compile. This needs to go.

There, that's the current short-term roadmap for me: cleanup, release, script system. Article taken from GamingOnLinux.com.
0 Likes
About the author -
Geek. Atheist+. Leftist. Metal-Head. Discordian. Lefty.
ScummVM dev, xoreos lead.
Free software zealot.
See more from me
The comments on this article are closed.
12 comments
Page: 1/2»
  Go to:

km3k Jul 10, 2015
Thanks for the update. I don't use Xoreos myself, but I enjoy following the development and seeing what gets added with each of these articles.
Kristian Jul 10, 2015
Thanks. A very very interesting read. Sounds like shader support should be pretty high on the priority list.
drmoth Jul 10, 2015
Keep up the good work. Looks very promising.
minj Jul 10, 2015
Even though I don't have an interest in those games this was an interesting perspective into reverse-engineering as usual, thank you for the update.
GustyGhost Jul 10, 2015
How long do you think you would need to leave cracking software running to get some more of those GDA headers?
minj Jul 10, 2015
I wonder if encrypted headers could be hacked. AFAIK blowfish is not weak if good keys are used, however ECB mode has an inherent problem with identical plaintext leading to identical cyphertext and is advised against.
DrMcCoy Jul 10, 2015
Quoting: AnxiousInfusionHow long do you think you would need to leave cracking software running to get some more of those GDA headers?

The problem is not really the cracking as such, the problem is finding the right dataset. I could run a stupid brute-force on it, and it would thousands of matches for each hash, all false positives. The hash is a CRC of the string in all lower-case, encoded as UTF-16LE, that gives a lot of easy collision.

For example, "bungholebehaviortype_" is a valid match for one of them, but obviously not what I want. As is "skimpilyshowmaterial", which shares a hash with "idtag". The latter is obviously the more correct one.

The way I cracked the one I did find is by taking strings found in the game, strings related to the game and strings found in the Dragon Age: Origins toolset files (which already include headers of most GDA files of Origins), and then combine them with each other and with words from a normal dictionary. I hash those combinations, compare them with the hashes we still need and spit out matches. Then I manually go through all these matches and remove the ones not fitting the game (like "properlyunsmoked", "rareuniquemummifications" and "impressionablyrooftops" ).


Last edited by DrMcCoy on 10 July 2015 at 3:07 pm UTC
DrMcCoy Jul 10, 2015
Quoting: minjAFAIK blowfish is not weak if good keys are used, however ECB mode has an inherent problem with identical plaintext leading to identical cyphertext and is advised against.

Oh, getting the keys for the encrypted ERF archives is not that hard. Dragon Age: Origins keys are very weak, and can be brute-forced easily. And the Dragon Age II keys are cached by the game after querying the server once. But I'm not sure if xoreos should/could include the keys, from a legal standpoint...
scaine Jul 10, 2015
View PC info
  • Contributing Editor
  • Mega Supporter
Man, that's absolutely fascinating. I take it that no-one at Bioware or who was involved in the coding is willing to share even definitions with you? Or even if they could, would you be able to accept without compromising your white-room approach?
DrMcCoy Jul 10, 2015
Quoting: scaineI take it that no-one at Bioware or who was involved in the coding is willing to share even definitions with you?

Yeah, I don't think they could. And I don't think I could either. That would just spell legal troubles for all involved.
While you're here, please consider supporting GamingOnLinux on:

Reward Tiers: Patreon. Plain Donations: PayPal.

This ensures all of our main content remains totally free for everyone! Patreon supporters can also remove all adverts and sponsors! Supporting us helps bring good, fresh content. Without your continued support, we simply could not continue!

You can find even more ways to support us on this dedicated page any time. If you already are, thank you!
The comments on this article are closed.