Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  

Author Topic: Version Control System as a game save/backup system  (Read 1773 times)

awilson

  • Escaped Lunatic
    • View Profile
Version Control System as a game save/backup system
« on: June 17, 2010, 01:13:28 pm »

I've seen a few threads mentioning the idea of using a VCS as a method to backup files, but didn't see any offering any results or tips.  I thought I'd add in what I found out over the past few days.

The idea is to use a VCS to manage backing out game files so that I can revert to an early save if you need to, like if my frakking Mayor executes my best craftsdwarf because he can't make something of clear crystal glass (which is bugged) Fun happens.  I've been just zipping up the DF folder, but storing and renaming all those zip files is a pain. A zip of the DF directory would come in just over 70mb.  I've got 3 different computers I'm playing on, so being able to move it all around is a huge plus.

I started out looking at both Mercurial and Git, as they seemed like the top contenders.

Mercurial was working lovely (very easy to set up and use) until I looked at the sizes.  Creating the initial repository results in a (to be expected) rough doubling of the space needed. I world-gen'd a large region, and the world.sav file clocks in at around 50mb (compression on).  Each commit ends up adding around 50mb to the repository.  That's gonna grow pretty fast -- and in the end be worse than the zip file option -- as the repository starts to get as big as my USB stick, I'm not going to be able to move it around between machines very well.

The size growth is expected, though.  With compressed saves, the world.sav file is going to look very different between revisions, and thus most likely end up with a full (or nearly full) copy needed for each delta.  It dawns on me that I make actually see a reduced file size (in the long run) if I turn off save compression.

But here's were Mercurial breaks down.  The uncompressed world.sav is 266mb.  Mercurial can't commit that without puking up an out of memory error.  Mercurial tries to use a single string comparison to even diff large binary files, so any file more than a few dozen megs will break it.

So, to Git, keeping the save compression off with the idea that it can hopefully diff the changes.  I init the repository, and do the initial check in.  I fire DF up, play for a for a minute, save, exit, and do another check in.  I run the Git database compression.  The whole DF directory (including the .git repository) is about 370mb.  I play a bit more, do then do another commit.   The  size goes up by 50mb, but after I run the database compression, that change goes down to just 1mb.  Awesome!  Only 1mb extra space for a save!

If I need to zip up the directory to copy it onto my USB, it's now just under 140mb.  Larger, but manageable.

I think Git is the answer, but it requires that you turn off save compression, and that don't mind the bit of extra time to run Git's database compression.  You need a bit of extra space running the game, but after about 4-5 saves (compared to zipped directories) you make that difference back, and any more than that, the saving is huge.  Compared to moving it all around, you can have a huge archive of a region's saves using Git and zip for the same space that just two regular zipped directories would use.

Has anyone found a better tool or way of doing their backups?

If there's any interest in how to do this, let me know and I can post my steps.
Logged

MP2E

  • Bay Watcher
    • View Profile
Re: Version Control System as a game save/backup system
« Reply #1 on: June 17, 2010, 02:02:05 pm »

Wow I didn't realize git would be good for this too! Excellent :)

Another reason to add to the lists why I prefer git to mercurial.

EDIT: Also you can just zip up the .git folder, it contains all the files necessary for git to reconstruct the project's entire history! This will make it loads smaller for you :P
« Last Edit: June 17, 2010, 02:12:47 pm by MP2E »
Logged

darius

  • Bay Watcher
  • ^^
    • View Profile
Re: Version Control System as a game save/backup system
« Reply #2 on: June 17, 2010, 02:45:03 pm »

I'm using bazaar for my code. Love it for gui. But as for professional opinion don't ask me :D (although they say they are simpler then git and faster in some operations)
Logged

Urist McDepravity

  • Bay Watcher
    • View Profile
Re: Version Control System as a game save/backup system
« Reply #3 on: June 17, 2010, 03:04:56 pm »

Same here, tried both hg and git, and stopped after repo grew past 5GB.
Git can delete revisions tho, so i'm thinking about using this feature and delete unneeded saves later.

Plus we need to finish python bindings for dfhack so i can finally automate saving, committing and writing commit logs.
Logged

chmod

  • Bay Watcher
  • I get by with a little help from my friends
    • View Profile
    • UDP Viper
Re: Version Control System as a game save/backup system
« Reply #4 on: June 17, 2010, 06:33:31 pm »

Distributed VCS are going to be problematic for such large sets of data. Mercurial, Bazaar, Git etc... are designed to handle changes to ASCII files. Which tend to be very small in content. And rely on easily differentiable data. Large blobs of compressed data is not what they are good at.

You may be better off with SVN.
Logged

Urist McDepravity

  • Bay Watcher
    • View Profile
Re: Version Control System as a game save/backup system
« Reply #5 on: June 17, 2010, 11:02:14 pm »

Distributed VCS are going to be problematic for such large sets of data. Mercurial, Bazaar, Git etc... are designed to handle changes to ASCII files. Which tend to be very small in content. And rely on easily differentiable data. Large blobs of compressed data is not what they are good at.
You may be better off with SVN.
SVN will be worse size-wise, since it keeps each frame instead of changes. Plus it is client-server, which is annoying.
DVCS like git behave same way as svn in case of binaries - they just store complete file instead of changes in each revision.

Uncompressed with git seems to be best solution, it was designed with performance as top priority after all.
Logged