BB2 Data format

[tags]

So hey guys, since there is no BBM2 yet, I've started looking into parsing the stuff. Now, this may not lead to anything, but I would love to see if we could share knowledge here and maybe I'll be able to piece something halfway useful together.

First off, after my match today, I see the following files in C:\Users\tags\Documents\BloodBowl2\Profiles\D57AC45256849D9B13E2422D91580FB9\Replays:

Code Select Expand


be3115b9a2ad9a6720a026bab0cce0ae_2016-01-08_07_12_29.bbrz
Coach-1117-2955d2ead36ed965a3fff024988702f0_2016-02-13_10_39_23.bbrz
ReplayIndex.xml


The last two are modified today, the other one is one of my previous games I guess.

Anyway, ReplayIndex.xml not only contains a list of the replay files (.bbrz) in the directory, but also seemingly all the stats highlights in readable XML. Sweet.

Now, the bbrz files were binary, so I was worried there for a sec, but on a hunch I renamed one of them to .zip, and lo and behold, WinRAR was able to show me an XML file inside, weee!

This is where the news get _very_ good, I think. The XML files are HUGE. Ok, so that means it'll take some work, but what I also THINK it means at this point is that we can actually faithfully recreate the game exclusively with the replay files. If that is the case, it's possible to make something that is better than BBM ever could be for BB1.

So, I've just scratched the surface of this so far, but I encourage everyone with a technical inkling to take a look and see what useful info they can dig out. If/when I make any progress, I'll be sure to post it here.

[Barmution]

Wow, nice work!

[Mr B]

Wow, nice work!

Cool..
I guess you know this but huge XML are no problem if you use 'First Object ' a very good xml-reader!
I'll take a look at my files also after my first game.
My first ever game online will be on Sunday.
And it is quite easy  to load replays etc.
NO lobby chat is awful but....  It might improve later on.

[Mr B]

Also.. A friend at work used Zorba to get all wanted data from big XML to xls.. And then  we could sort the data captured with ease.

If you know these two if sounds possible anyhow.

[tags]

I'm currently working on a C# client that at least fills some internal structures based on the XML data. I don't think converting to a spreadsheet with ~10MB XML data, around 15MB when properly formatted, is very useful unfortunately. Anyway, it's a lot of work for sure, but I am making progress. Just posting here to say that I haven't given up on this yet :D

For now, there's only a client as I said (which is also far from done), but I guess I/we would want some kind of server solution as well. I think that I would want most of the processing to be done client-side, so that the server is only a thin database interface to sync replays against, but all of that will come later I guess. I'm mostly a low-level programmer, so I'm not really proficient with either C#/.NET or web development, so I may have to involve someone else for some help/viewpoints once I get the basics down. If anyone has any strong viewpoints here, I'd be happy to hear them.

For now, there are a few question marks with the XML, and that is what apparent enum values represent. I'd appreciate any help to fill in the gaps here.

For example, there's <IdRace>, where I've figured out these through replays:
                case 3: return "Skaven";
                case 4: return "Orcs";
                case 5: return "Lizardmen";
                case 8: return "Chaos";
                case 24: return "Bretonnians";

There's skills in the <ListSkills> tag, and I have these so far:
                case  1: return "Strip Ball";
                case  6: return "Catch";
                case  7: return "Dodge";
                case  8: return "Sprint";
                case 12: return "Extra Arms";
                case 13: return "Mighty Blow";
                case 15: return "Horns";
                case 17: return "Stand Firm";
                case 26: return "Dauntless";
                case 27: return "Dirty Player";
                case 30: return "Block";
                case 35: return "Fend";
                case 36: return "Frenzy";
                case 38: return "Guard";
                case 44: return "Loner";
                case 47: return "Pass";
                case 49: return "Prehensile Tail";
                case 56: return "Side Step";
                case 57: return "Tackle";
                case 58: return "Strong Arm";
                case 59: return "Stunty";
                case 61: return "Sure Hands";
                case 67: return "Wild Animal";
                case 68: return "Wrestle";
                case 71: return "Kick";
                case 75: return "Claw";

And there's <IdPlayerTypes>, where I have these:
                case 16: return "Skaven Lineman";
                case 17: return "Skaven Thrower";
                case 18: return "Gutter Runner";
                case 19: return "Storm Vermin";
                case 20: return "Rat Ogre";

                case 21: return "Orc Lineman";
                case 23: return "Orc Thrower";
                case 24: return "Black Orc";
                case 25: return "Orc Blitzer";

                case 27: return "Skink";
                case 28: return "Saurus";

                case 32: return "Beastman";
                case 33: return "Chaos Warrior";

                case 139: return "Bretonnian Lineman";
                case 140: return "Bretonnian Blitzer";
                case 141: return "Bretonnian Blocker";

And then there's the absolute mystery of <TeamColor>. I can't get it to make sense. For example, Da Free Pink Brotherz (which are... well... pink) has the value <TeamColor>400039</TeamColor>. Converting it to hex and treating it as channels yield ‭0x061AA7‬, which make no sense in either RGB, HSV, HSL or anything else I can think of. It also looks like it's too big a number to be an internal enum in the engine. Anyone got any ideas?

[Barmution]

Team color could be already coded in hex, ie. purple that gets tinted to pink, see http://www.colorhexa.com/400039.

[tags]

Team color could be already coded in hex, ie. purple that gets tinted to pink, see http://www.colorhexa.com/400039.

Ah, thanks, CMYK looks promising, I'll definitely have a look at that. I don't _think_ it's already hex, since I've looked at a few different replays now and they're all decimal. But the values may make sense after conversion in CMYK. Anyway, I'll have a look at a few different replays and cross-reference them with what I see in-game, cheers.

Not that team colour is really the most important thing... but it's been bugging me :D

[Mr B]

www.onesandskulls.com
This can also be used.
Feel free to check it out.
I'll do when I'm home.
Cheers

[tags]

Cool, I'll have a look at that tomorrow (no time today)

[tags]

Hey guys, I've been quiet about this for a while, but I have been working on it.

Turns out, this project is/was a LOT more ambititous than the optimistic engineer in me estimated, proving once again that I should not pursue a career in project management, but also, that I am at least not a fully developed cynic just yet.


TL;DR (short version):

- I've built the first beta prototype of BB2Replayer for (not-quite-public) distribution, please give it a go: www.bb2replayer.com/beta
- It should be able to sync your replays with the server at bb2replayer.com, and give you some basic stats
- It is ugly, slow, and probably has quite a few bugs in it. It should, however, be perfectly safe to run, although I can't guarantee it's useful in its current state
- I appreciate any and all feedback on it, either here, or to bb2replayer@gmail.com (or tagsemb@gmail.com). Mail might be easier if you are including the log or replay files (which is probably useful if you report a bug)
- If technical details bore you, you can safely skip the rest of this post :)


Long version:

The project consists of a C# client (which is the file distributed now) and a C++ server, which for now I'm currently running and managing on bb2replayer.com. The plan is to open-source all this at some point in the long term, but in the slightly shorter term, distribute the server binaries as well so that everyone can set up servers if they want to. The client can already configure multiple servers, although the default is set up to sync with bb2replayer.com. The server compiles for both Linux and Windows targets at the moment. bb2replayer.com runs Linux, but I'm doing a lot of testing locally with the Windows build. The communication model is a custom protocol on top of a TCP socket, which enables me to run both the client and server on localhost, which is handy for development and debugging.

It is a client/server solution where the server is actually quite thin in logic, and merely serves as a semi-smart repository for .bbrz files. The client is responsible for pulling out all the stats and interpreting the results, as well as building a database. The idea is that the client can subscribe to any number of leagues, and the server will automatically feed the client all the replay files it has matching that league. The client will at all times also send all its local replay files to the server (that it doesn't already have, of course, based on md5 sum checking) IF they belong to a league. Matches that do not belong to a league are typically single-player matches, which are not interesting for this purpose. Since the server-side is relatively simple, I think that part of the project is close to finished.

Now, the client-side is where things get messy. Here's a run-down of the current state of affairs:

- GUI
There are a lot of things to do in a world I don't really know that well (C#/.NET), and I've just realized I've already made a pretty big mistake in starting a WinForms project. Being ignorant of GUI stuff, I didn't really know WinForms was dead in the water, and didn't notice the problems until populating views with several tables and stuff. Bottom line is: This is the last release using WinForms (barring small hotfixes). I will probably have to re-create the GUI more or less from scratch in WPF. This will take some time, but hopefully I can create something that is a) more responsive, and b) less ugly.

- Reverse-engineering
So, .bbrz files are basically .zip files (around 150-200KB), and they each contain exactly one huge .xml file (around 15MB or so). This is good news, in that all the information about the game is in there, and being XML, it is technically readable. For those that had a look at the Blood Bowl 1 replay files, this is a million times better. There are, however, quite a few challenges as well. There are lots of references to internal enums in the code, so it's not obvious for example what "<PlayerType>7</PlayerType>" means. I've figured out most of the races, player types and skills at this point, but there are still unknowns. One of the features I want to have, and one that is partially implemented, is the ability to skip back and forth through the replay in the C# client, and be able to analyze the game in a more accessible manner than you can from the actual BB2 client. This is quite difficult, though, as I have to understand almost all of the events in the XML. The challenge is more about size and scope than difficulty, but I will be chipping away at this. It is, however, not the most important part of the project. Luckily, the replay files contain <Statistics> tags for both teams and players, making stats mining from replay files quite easy given that you are happy with what the game provides. It does provide most of the basic stuff like amount of touchdowns, tackles, injuries, casulties, KOs, kills, passes, rushing etc. In the long run, I want to introduce more stats based on understanding the event log, but all that is less important than getting the basics up.

- Syncing
Upon starting, the client now searches your typical BB2 replay path and tries to automatically import all replay files it find there. BB2Replayer will take copies of all replay files it finds (with a valid league) and store them in its app directory (which is basically <x>/users/<user>/bb2replayer). I hope this isn't a problem size-wise for anyone, as the replay files are about 150KB each, but it _may_ become one later on. I'll have to rethink that later if that is the case. After the client is up and it has assessed all the replay files it knows about locally, it will send its file list to the server and ask if the server is missing any of those. If it is, then all files that the server doesn't already have will be sent. This should typically just be your last game, so hopefully that's not a big pain for anyone except those sitting on ISDN lines :D
After that, the client will request all replay files from server that matches any of the leagues it knows about locally. In the future, you will be able to see a list of all the servers' leagues and subscribe to them, but for now, you are limited to whatever leagues you have locally.

Part of the reason why I'm releasing this now in a quite unfinished state, is that merely by running the client, you should automatically be uploading all the NUL replay files to the server, which gives me a broader data set to work with, and I won't have to bug you constantly about sending them.

[tags]

Released version 0.2.3.0, it has a fix for an issue where your local games may not get synchronized (thanks Barmution for the debugging help).

http://www.bb2replayer.com/beta/

[Barmution]

This looks really promising, kvasi! Great work! ;D

[J-Reedy]

Wow. I'm most impressed on the work you've done here Kvasi. Kudos for your hard work. I'll try to find time to test drive the system during weekend so you'll get the missing games in.

[tags]

Cheers, I appreciate that. I'm releasing (minor) fixes in new versions pretty rapidly at this point, so be sure to check the web for the latest build. I've fixed a couple of issues that Barm ran into while guinea pigging and implemented them in 0.2.4.

My next immediate goal now is to make proper web reports that we can link to from the statistics page, that will probably be 0.3.0

[tags]

Hi guys, I just released BB2Replayer 0.5.0, and it's a doozy of an update!

Do check out the new stats page: http://norscaleague.com/index.php?topic=586.0