You can sign up to get a daily email of our articles, see the Mailing List page.
We do often include affiliate links to earn us some pennies. See more here.

An Update On The Open Source Project 'Xoreos' Concerning The Witcher

By - | Views: 13,791
xoreos is a FLOSS project aiming to reimplement BioWare's Aurora engine (and derivatives), covering their games starting with Neverwinter Nights and potentially up to Dragon Age II. This post gives a short update on the current progress.

Note: This is a cross-post of a news item on the xoreos website.

Further continuing on the path to make all the engines display areas (see my posts on Jade Empire and Neverwinter Nights 2), I looked at The Witcher next.

As some people might know, CD Projekt RED licensed BioWares Aurora engine for the first The Witcher game (and only the first; the later parts use their own REDengine). And in many aspects, it's very obvious that they spawn off directly from the Neverwinter Nights codebase, not from later BioWare titles. There have been quite some changes, though, to accommodate for The Witcher's requirements.

Models

Due to their similarity to Neverwinter Nights' MDL model files (and with light help from Michael_DarkAngel's twMax code), I had already a partially working loader for The Witcher's MDB files. Or so I thought. Turns out, my quick & dirty hack was just about enough to get simple object and character models to show, but it totally failed on area geometry models. Those turned up completely invisible.

Textures and lightmaps

Closely looking at what my code does made me realize that the geometry itself loaded, but the textures failed to load. In fact, the textures it tried to load didn't exist. However, there were similarly named ones in the resources. In fact, for the requested texture "foo", there were texture resources "foo!d", "foo!n" and several others.

Well, these are, it seems, lightmaps, and several different ones depending on the time of day (named after their Polish words). There's:

  • !d, for dzień (day)
  • !r, for rano (morning)
  • !p, for południe (noon)
  • !w, for wieczór (evening)
  • !n, for noc (night)

Since not all of them might exist for a given texture, I settled on just loading the first one available. And yes, that gave me textured area geometry. With just the lightmap applied, it still looked a bit low-res, however. No wonder, there needs to be a base texture as well. Unlike Neverwinter Nights, which just straight up names the textures and has simple TXI files for some texture properties, The Witcher has full-fledged material definitions integrated into the model. And they're shader-based. For the object models, it was enough to just take the texture names and run with it, but for the area geometry models, I had to extend this a bit. Granted, this is still a hack (we still don't support shaders), and fails occasionally, only less so than before.

The result was this:

image image image
There's bits missing, I hear you say? Correct. Unfortunately.

TexturePaint nodes

More debug printouts on the model loader clued me in: there's a new kind model node! The twMax author also noticed this; he calls them "TexturePaint". No support for them in twMax, though. My Google-fu didn't uncover anything else helpful either.

With no existing tools to help me, I had to do the dirty work myself. I pulled out my trusty friend the disassembler. Luckily, CD Projekt RED kept the tradition of supporting ASCII representation of model files, and so I was able to map out the loader code relatively quickly, for the most part.

First, I filled out my loader to stub and comment more MDB fields I previously just ignored. Then, I started implementing the TexturePaint nodes. Thanks to a brief email conversation with someone who's also playing around with The Witcher models, I already knew what these nodes probably represented: geometry textured by blending several distinct textures together, to create more realistic terrain. I.e. similar to what I found Neverwinter Nights 2 does for its terrain geometry.

This turned out to be exactly the case, only with in-node weightmaps instead of the color channel approach in Neverwinter Nights 2. Additionally, these nodes too have a lightmap applied. Without shaders, this mixing is awfully slow and memory-consuming to do, therefore I instead just apply the lightmap for now. The result looks like this:

image
It's not exactly pretty, but it at least shows something.

Level of Detail (LOD)

After having implemented the TexturePaint nodes, I found a curious issue: certain nodes appeared twice, overlapping each other:

image
This is the result of the LOD information in the node headers. Some nodes are supposed to be displayed when you're near, some when you're far. Simpler textured geometry nodes are displayed instead of the more complex TexturePaint nodes when far enough away to not notice the texture blending anyway. As a workaround, I rigged it to only display the highest LOD for now.

Areas

With the area models correctly loading, I went on to actually load the area. Owing to its origin, it's again very similar to Neverwinter Nights, with one difference: no tiles and tilesets. Instead, the "tileset" value specifies the singular area geometry model to display.

I quickly implemented loading said area geometry model and simple area objects. And was baffled. The area geometry's position didn't match up with the objects' positions. The area was so far in the distance, you couldn't even see it. I looked and searched for a "tile" position in the area description files...nothing. As an experiments, I bound moving the area model to keyboard keys, and played around with it until it fit. The correct position, for some reason, is {1500.0, 1500.0, 0.0}. Don't ask me why, but this works for all areas in the game. ¯\_(ツ)_/¯

image image image
Object orientation

There was another thing I noticed, though: the orientation of the area objects is wrong. Unlike Neverwinter Nights, which only specifies one angle, the "bearing", for each object, The Witcher lets you rotate all objects in all three axes. The orientation is described as a quaternion. Now, Neverwinter Nights 2 does the same and because we need the object orientation in Euler angles instead, we convert them. Unfortunately, that code doesn't seem to work correctly in The Witcher. I assume it's connected to the fact that The Witcher actually fully rotates the objects (while Neverwinter Nights 2 only, in effect, rotates around two axes), combined with how xoreos changes the axes around for world objects and additionally needs to translate the coordinate system from Direct3D orientation to OpenGL's.

Try as I might, I couldn't get the correct orientation. After way too much banging my head against the wall, I caved and put that onto the TODO pile. That's something I have to revisit another day.

Creatures

"Only objects? Where are the creatures, the NPCs, the people?", I hear you ask, imaginary reader. Well, The Witcher doesn't directly describe creatures in the area files. Instead, there are spawn points, and I think the rest is handled by the game scripts. Not something I can do now. No NPCs for now, I'm afraid.

Miscellaneous

A few bits and pieces I found out or did during this endeavour:

  • I renamed the engine and namespace in xoreos from "thewitcher"/"TheWitcher" to "witcher"/"Witcher"
  • I noticed that The Witcher, unlike other Aurora games, encodes all strings in UTF-8. The language IDs are also wildly different. To get these together under one big tent, I changed how xoreos handles languages and encodings
  • The Witcher uses Lua scripts. I already knew that previously. But what I didn't know: it also, additionally, uses the traditional NWScript. That'll be a lot of "fun" to reimplement...
  • I found references to the LuaCOM library in the The Witcher disassembly, which provides Lua bindings for ActiveX components. I really, really, really hope that's unused

In conclusion, area loading in The Witcher is now in a similar state to area loading in Neverwinter Nights, Neverwinter Nights 2, Knights of the Old Republic, Knights of the Old Republic II and Jade Empire. It's not flawless, and there's still a lot of things missing, but it's a start. :)

What's next? There's three games left without area support: Dragon Age: Origins, Dragon Age 2 and Sonic Chronicles: The Dark Brotherhood. The latter is a Nintendo DS game, and is missing basic support for a lot of DS-specific file formats. In fact, I'm still not 100% sure this game even belongs in xoreos... The two Dragon Age games are somewhat further along: resource loading works and the texture format is known. They are, however, completely missing model support, as well as support for the new GFF version. Since that is at least something to go by, it's possible I'll tackle Dragon Age: Origins next. It could take a while until I have something worthwhile to report, though. Article taken from GamingOnLinux.com.
0 Likes
About the author -
Geek. Atheist+. Leftist. Metal-Head. Discordian. Lefty.
ScummVM dev, xoreos lead.
Free software zealot.
See more from me
The comments on this article are closed.
14 comments
Page: 1/2»
  Go to:

minj Apr 14, 2015
That was an interesting read but it seems xoreos.org can't handle the image traffic.
Anorelsan Apr 14, 2015
It's fantastic. Great work as always! Now wait to see the next update :P
Maquis196 Apr 14, 2015
View PC info
  • Supporter Plus
Apologies if this in a FAQ that I read ages ago but forgotten, but are you planning on getting some basic grounding in (like the areas) then going all out to make at least game work?

I admit, I can't wait to play kotor on this but curious if you'll have a "each game is about 60-75% playable" thing going on?

either way, keep up the great work. Much appreciated!
drmoth Apr 14, 2015
Great work, interesting read...keep it up and keep digging!!
DrMcCoy Apr 14, 2015
Quoting: minjThat was an interesting read but it seems xoreos.org can't handle the image traffic.
Yeah, seems my server didn't like the initial surge of requests. It's calmer now, though.
Maybe I should start putting the images up on imgur next time, like other articles here do.

Quoting: Maquis196but are you planning on getting some basic grounding in (like the areas) then going all out to make at least game work?
Well, right now, I am trying to get areas to show for all games, to have something to show for myself, so to speak.

And in general, if I already know how to do one thing, one concept for one game, it's easier getting that thing adapted to the other games. It also helps in making sure these subsystems are flexible enough: if I implement something for one game only, and then continue and build upon it, it's of course harder to change later, should the need arise (because a different game might use that thing slightly differently).

So I do expect that I will probably have all games completed up to a certain point until one single game is 100% done, yes. Otherwise, I would probably break the game that's declared as 100% done a lot while focusing on another game.
Kristian Apr 14, 2015
Got any thoughts on what features should be included to get to that "certain point"?
DrMcCoy Apr 14, 2015
Well, I'm thinking developing all gameplay commonalities together for each game makes sense: scripting system [1], walking characters, pathfinding, proper ingame camera perspective, the notion of inventories, dialogue [2], combat mode, .... Basically, each game would be somewhat playable, the gameplay recognizable as such.

That doesn't mean that the games would each be very close to finished. There's a lot of script functions that need to be implemented that are mostly independant from this.

Of course, this all is just my current view and might change when I'm further in, as I discover more. More people joining in can also shift around things a lot. And yes, I can always use more people joining in! :)

[1] NWN already has the bindings, but ImperatorPrime is currently working on rewriting that, because the code is a bit awkward. Once that's done, it should be neatly applyable to all games [3]
[2] NWN already has that. Dialogue working is closely tied to having an ingame GUI (the GUIs in the games are wildly different) and the script system (because dialogue nodes call scripts to check and set states).
[3] Well, except The Witcher, because NWScript + Lua...
oldrocker99 Apr 14, 2015
View PC info
  • Supporter Plus
Holy cow, this is grand news ^_^ . I am still agog at the gaming possibilities now , and certainly coming, for Linux .
DrMcCoy Apr 14, 2015
Hold your horses, this is all still very early progress. There's a lot of work left to do here. :P
oldrocker99 Apr 14, 2015
View PC info
  • Supporter Plus
Quoting: DrMcCoyHold your horses, this is all still very early progress. There's a lot of work left to do here. :P

Oh, I read the article; I am hornswoggled at the dedication being applied!.
While you're here, please consider supporting GamingOnLinux on:

Reward Tiers: Patreon. Plain Donations: PayPal.

This ensures all of our main content remains totally free for everyone! Patreon supporters can also remove all adverts and sponsors! Supporting us helps bring good, fresh content. Without your continued support, we simply could not continue!

You can find even more ways to support us on this dedicated page any time. If you already are, thank you!
The comments on this article are closed.