Re: [Adonthell-devel] Charedit File Format

[Kai Sterker]

On 8/14/07, Andrew Phillips <address@hidden> wrote:

> Rather than perform thread necromancy, I'm starting a new thread. I'm
> both asking for suggestions and trying to work things out for myself.
> I don't think I've reached any firm conclusions just yet.

Before commenting on your ideas below, some general thoughts on the
whole topic of file formats, saving and loading. It kind of goes along
with the requirement you mention way down:

> Finally, it seems to me that whatever record format is chosen, it
> should ideally be easily convertible to well-formed XML


Most classes of the Adonthell engine that can be serialized provide
the methods put_state(base::flat &out) and get_state(base::flat &in).
That will get them into a common format that can then be stored either
in human readable XML or gz compressed binary format. Optionally, some
classes provide put_state(std::string filename) and
get_state(std::string filename) to save a single class instance to a
single file.

I'm pretty sure that this design isn't the best possible solution, but
it offers a reasonable error checking mechanism (i.e. it will fail
gracefully on corrupted or missing data) and achieves at least some
compatibility across versions (i.e. it's possible to change order of
fields or to add new fields without breaking stuff).


Now, all this does not really matter for an editor, as long as it
finally produces something the engine can read. And that's where a
decision has to be made: Does an editor use its own file format and
conversion is required before it can be used, or does an editor use
the engine file format so that it can be used directly?

The charedit prototype for 0.3 used the first approach. It saved each
character into individual files and a second progam, mergechars, read
all those and created the character.data file required by the engine.
So this is perfectly okay, and under the current circumstances I
believe it would be a good idea to.do it that way.

Actually, the final conversion could be performed by a little python
script, which would make all those string operations much simpler.


The only real benefit of the second method would be that one could
reuse parts of the engine for the editor (not so tricky as with 0.3,
with the nice shared libs we now have for the different modules), it
could simplify the editor code somewhat. It's what I'll try to do with
questedit, once I get around to it ...


With that out of the way, and assuming that we'll have one file per
character and a little conversion script, lets get to your
suggestions.

> Last time, we reached two conclusions (possibly three)
> 1) No specific (file|record) format guidelines exist
> 2) Form follows function
> 3) Eventually, whatever format we decide upon might need to be
> workable within the context of a single, monolithic character file.

Yes.


> The simplest useful format seems to be
>
> Name:Race:$Label:Strength:$Label:Hardness:$Label:Agility:$Label:Willpower
>
> However, this design leaves out alignment, level, and all property
> values. However, as a basic template for things that will be built and
> leveled when they are instantiated (Wolf359, for example) it might
> work.

When keeping it to one file per character, you can probably get away
with putting everything into one line. Nobody will probably be able to
edit it manually, but that is what we have the editor for :-).
Compatibility will be an issue, however, unless you only append stuff
at the end. This however will probably result in a complete mess, as
far as the file parsing is concerned. So maybe it is not the best
choice if you are concerned about future enhancements. (Think for
example about equipment or inventory required for NPCs, what spells
and feats they know, etc).


> There seem to be two flaws in this design. The
> first is that because every field after Name and Race is subject to
> change, any changes would overwrite the previous values. The character
> would contain no history of itself and provide no chance of restoring
> to a previous state should data become corrupted.

It's an issue, but it could be mitigated by keeping the character data
files in CVS.  You could even go so far and create the final
character.data XML file used by the engine during the installation
process from the raw editable character files.


> The second flaw is
> that properties are stored as a single value. A given property value
> could likely be factored to derive its multiplier and modifier, but
> this seems either non-trivial or less trivial than I would like. Also,
> it breaks the database normalization rule about storing values that
> can be calculated from other fields.

I agree. It would be better to store the base value of a property
(given a fixed multiplier/modifier), not the final value itself. If
either of multiplyer/modifier is not fixed, it needs to be stored
together with the base value.


> The other basic format contains no labels and (at least initially)
> recorded each new level and its attribute values on a separate line.
[...]
> The flaw in this record format is that it would be very, and
> arbitrarily, long, particularly if I roll in the property modifier and
> multiplier stuff from the other basic design. Also, it contains no
> labels, so navigating through the record would be non-intuitive
> without good documentation of the format. Labels would certainly help
> make everything more navigable, particularly where finding information
> on previous levels is concerned.

Not sure if it really makes sense to store any kind of character
history. During game creation, character data can be version
controlled in CVS. During game play, a new character data file will be
written for each saved game. Reverting would be as simple as loading
an older save.


> Finally, it seems to me that whatever record format is chosen, it
> should ideally be easily convertible to well-formed XML, since we are
> already using XML for character storage in worldtest.

Right. But unless the format is overly complicated, the conversion
shouldn't be a big deal (see above).


If we really want a clean format that can be expanded fairly easily in
the future, maybe something like an .ini file would work:

[General]
Name =
Gender =
...

[Attributes]
Strength =
Willpower =
...

[Skills]
...

It's a bit cumbersome to parse line by line, but by using the right
data structures it might not be too bad. (i.e. keep all attributes,
skills, etc. in a hashtable, so you can access the corresponding
object by the identifier left of "=" and let it parse whatever is on
the right side.) You'll probably have to handle the [General] section
manually, but for the other sections you'll know what type of property
they contain and only need to iterate over each line. Adding entries
to a section later on (or completely new sections like [Equipment] or
[Spells]) will not break the whole format.

Not sure if that is really the best solution for the problem at hand,
but at least it is simple, expandable, practically self-documenting
and still fairly easy to implement. It may not be as efficient as a
completely custom file format (in respect to both file size and
parsing), but I figured some time ago that being more explicit helps
making things (be it code or file formats) easier to understand.

Kai