Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1] 2 3

Author Topic: Obtaining a dump of the wiki  (Read 31340 times)

telamon

  • Bay Watcher
    • View Profile
Obtaining a dump of the wiki
« on: February 18, 2012, 01:35:42 am »

I spend a lot of time without internet access, and DF is such a hardcore game that I find myself severely hobbled without access to the glorious wiki to provide me with the raw information I need. I'm almost unable to play DF without either immediate access to the wiki, or printouts of the pages I require. Since I'm getting back into the game right after a new version's release (and I stopped playing around 31.18, so there are a lot of features I have yet to learn), I'm in constant need of the wiki to familiarize myself, so I basically can't play DF offline.

Wikipedia has the facility to dump a massive number of pages, minus talks, users and old revisions, to a reader's hard drive, and the WikiTaxi tool is capable of interpreting those dumps (ie the mediawiki database format) and displaying them in a convenient reader for offline use. This would totally solve my problem, and we can assume that I have enough hard drive space to handle the volume. Is it at all possible to get this dump service for an entire namespace of the DF wiki, or is that too big a task for the admin team to worry about right now?
« Last Edit: February 18, 2012, 01:44:50 am by telamon »
Logged
Playing DF on Windows 98 since.... ?
At 55 frames per minute.

Locriani

  • Bay Watcher
  • Locriani == Briess
    • View Profile
    • dwarf fortress wiki
Re: Obtaining a dump of the wiki
« Reply #1 on: February 18, 2012, 01:50:28 pm »

We recently added a toolserver to the mix.  I can't promise when it will get done, but we will look into providing automated dumps for the wiki for download, as soon as we get image uploads working again.
Logged
I am one of many administrators of the wiki.  Please use my user page (http://dwarffortresswiki.org/index.php/User_talk:Briess) on the wiki to contact me, as I check that more often than these forums.

umaxtu

  • Escaped Lunatic
    • View Profile
Re: Obtaining a dump of the wiki
« Reply #2 on: February 24, 2012, 11:12:53 pm »

A dump of the wiki would be nice.  8) But its not urgent so no rush :D

Edit: What is a toolserver? (Just curious)
« Last Edit: February 25, 2012, 02:22:15 pm by umaxtu »
Logged

GalenEvil

  • Bay Watcher
    • View Profile
    • Mac-Man Games
Re: Obtaining a dump of the wiki
« Reply #3 on: March 15, 2012, 02:35:26 am »

I think this is a pretty good idea as well, would prevent me from having to go to each individual page and then save it via CTRL+S and save as a whole webpage... I wonder though, would links be functional between pages within the dumped wiki?
Logged
Fun is Fun......Done is Done... or is that Done is !!FUN!!?
Quote from: Mr Frog
Digging's a lot like surgery, see -- you grab the sharp thing and then drive the sharp end of the sharp thing in as hard as you can and then stuff goes flying and then stuff falls out and then there's a big hole and you're done. I kinda wish there was more screaming, but rocks don't hurt so I guess it can't be helped.

Kogut

  • Bay Watcher
  • Next account: Bulwersator
    • View Profile
Re: Obtaining a dump of the wiki
« Reply #4 on: March 15, 2012, 02:37:57 am »

Edit: What is a toolserver? (Just curious)
Server hosting bots and various useful tools. Description of Wikipedia toolserver: http://en.wikipedia.org/wiki/Wikipedia:Toolserver
Logged
The worst bug - 34.11 poll
Tired of going decades without goblin sieges? Try The Fortress Defense Mod
Kogut, the Bugfixes apostle of Bay12forum. Every posts he makes he preaches about the evil of Bugs.

umaxtu

  • Escaped Lunatic
    • View Profile
Re: Obtaining a dump of the wiki
« Reply #5 on: March 17, 2012, 01:05:51 pm »

Quote
Server hosting bots and various useful tools. Description of Wikipedia toolserver: http://en.wikipedia.org/wiki/Wikipedia:Toolserver

Thank you. :D
« Last Edit: March 20, 2012, 02:39:44 pm by umaxtu »
Logged

badwin

  • Bay Watcher
    • View Profile
Re: Obtaining a dump of the wiki
« Reply #6 on: March 30, 2012, 12:06:35 am »

Is this a possible thing to do now? I don't really know much about wikis, so sorry if it's still not an implemented feature.
Logged

telamon

  • Bay Watcher
    • View Profile
Re: Obtaining a dump of the wiki
« Reply #7 on: March 30, 2012, 12:18:39 am »

afaik, it's very poor etiquette to spider a website and download all the pages manually since you take up a lot of the site's bandwidth if it's big (tools exist that can do that, check out httrack or the classic wget). since the wiki, to the best of my knowledge, has not yet added a page to access this feature, i can only assume that it's not yet available.
Logged
Playing DF on Windows 98 since.... ?
At 55 frames per minute.

Locriani

  • Bay Watcher
  • Locriani == Briess
    • View Profile
    • dwarf fortress wiki
Re: Obtaining a dump of the wiki
« Reply #8 on: May 03, 2012, 09:50:21 am »

Your wish is my command.
http://dwarffortresswiki.org/images/dump.xml.gz

http://dwarffortresswiki.org/images/dump.xml.bz2
This dump is automagically updated daily.
« Last Edit: May 04, 2012, 09:44:39 am by Locriani »
Logged
I am one of many administrators of the wiki.  Please use my user page (http://dwarffortresswiki.org/index.php/User_talk:Briess) on the wiki to contact me, as I check that more often than these forums.

telamon

  • Bay Watcher
    • View Profile
Re: Obtaining a dump of the wiki
« Reply #9 on: May 03, 2012, 04:29:52 pm »

Thank you so much! Works perfectly with wikitaxi.

Note for other people trying to use the dump: because it's gzipped, wikitaxi cannot interpret it (I was surprised that it couldn't... really if you can support one of the archive formats, another one shouldn't be too far away. But whatever, it works). On unix systems you probably know of a compression utility to convert to bz2; on windows you'll need 7zip.

Download the dump and open it in 7zip, then extract the xml file inside. Create a new archive in 7zip and make sure the archive format is bzip2. Add the dump xml to this archive (not the gz file you downloaded, but the xml file itself).

You need wikitaxi to read the dump. Taxi leaves some things to be desired and it's a bit rudimentary, but it's basically the only program available that's capable of doing this job. Extract wikitaxi into any folder of your choice (it's a self contained prog and needs no installation) then run the wikitaxi importer. Select the bz2 archive you just made with 7zip. Wikitaxi importer will convert the bz2 archive into a .taxi file that wikitaxi uses to interpret the wiki database, so you also need to tell the importer where to output the taxi file. Then run the importer to create the taxi file itself.

Finally, open up wikitaxi and go to options, then select "open .taxi file" and select the taxi file you just created. You will now have access to the DF wiki offline! You can search the wiki using the bar at the top of wikitaxi window. Because the DF wiki is so reliant on redirecting general links to a particular namespace (for example when you go to wiki/noble, it's actually going to redirect you to wiki/DF2012:Noble), those redirects don't seem to play perfectly with wikitaxi, so you'll have to rely on the search bar for most of your reading. For example, if you're looking for the nobles page, your best bet is to search DF2012:noble. Nevertheless, I can confirm that it works!

EDIT: some alternatives to taxi can be found here but I have yet to try any of them. In general most of these programs listed are server frontends for the mediawiki software, so they're not exactly introductory to install. I have a bit of experience with XAMPP so I could try most of them (I bet I could even load up mediawiki on a local server and just run the dump out of that somehow), but wikitaxi is still prob the easiest solution since it wraps the code in a pretty little reader interface.
« Last Edit: May 03, 2012, 08:48:10 pm by telamon »
Logged
Playing DF on Windows 98 since.... ?
At 55 frames per minute.

Quietust

  • Bay Watcher
  • Does not suffer fools gladly
    • View Profile
    • QMT Productions
Re: Obtaining a dump of the wiki
« Reply #10 on: May 03, 2012, 07:51:05 pm »

If wikitaxi needs the file to be bzip2-compressed, I'm sure Locriani can switch the daily process to use bzip2 instead of gzip - it'll probably be smaller, too, saving a bit of bandwidth.
Logged
P.S. If you don't get this note, let me know and I'll write you another.
It's amazing how dwarves can make a stack of bones completely waterproof and magmaproof.
It's amazing how they can make an entire floodgate out of the bones of 2 cats.

Locriani

  • Bay Watcher
  • Locriani == Briess
    • View Profile
    • dwarf fortress wiki
Re: Obtaining a dump of the wiki
« Reply #11 on: May 04, 2012, 09:17:02 am »

It turns out we have a problem with our toolserver, I reverted the dump back to a version from late march.  It will probably be a week before I can get it corrected again.

Dump is back up at http://dwarffortresswiki.org/images/dump.xml.bz2
« Last Edit: May 04, 2012, 09:44:21 am by Locriani »
Logged
I am one of many administrators of the wiki.  Please use my user page (http://dwarffortresswiki.org/index.php/User_talk:Briess) on the wiki to contact me, as I check that more often than these forums.

umaxtu

  • Escaped Lunatic
    • View Profile
Re: Obtaining a dump of the wiki
« Reply #12 on: May 05, 2012, 03:40:10 pm »

I found another program like WikiTaxi that looks better called Kiwix. Problem is it only accepts .zim dumps. I'm currently setting up a local MediaWiki with the dump of this wiki and will use the Collection Extension to convert it into a .zim dump. I will report back here with results.
Logged

telamon

  • Bay Watcher
    • View Profile
Re: Obtaining a dump of the wiki
« Reply #13 on: May 05, 2012, 03:50:15 pm »

It looks really good, but I'm personally too lazy to change my dump format. Format implementation details here for anyone who is interested
Logged
Playing DF on Windows 98 since.... ?
At 55 frames per minute.

NYDwarf

  • Bay Watcher
    • View Profile
Re: Obtaining a dump of the wiki
« Reply #14 on: July 10, 2012, 12:18:30 pm »

Did you ever make the .zim file?

Thanks!
Logged
Pages: [1] 2 3