Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1] 2 3

Author Topic: [DATABASE] Dwarf Fortress  (Read 10942 times)

JodGap

  • Bay Watcher
    • View Profile
[DATABASE] Dwarf Fortress
« on: September 19, 2013, 07:14:40 pm »

Dwarf Fortress Database
http://dfdb.info
http://dfdb.info/old

New design
http://i.imgur.com/yQX83YA.png
http://i.imgur.com/SXnUE0v.png
http://i.imgur.com/ZZmwu2m.png

Done
- Items parser

Development
- Interactive dwarf
- Raw import - upload - export for modders
- Import Masterwork manual into the website

Wanted
- Icon font designer
- PHP developer (We use Laravel PHP Framework, Semantic UI CSS Framework, jQuery and mySQL)
- Modders to receive a beta tester account and upload their raws to beta test the raw uploader/parser

Team members
SQL/PHP: Jodgap
Perl: MagmaMcFry GitHub
RAWS: Meph
« Last Edit: May 26, 2014, 02:44:22 pm by JodGap »
Logged

MagmaMcFry

  • Bay Watcher
  • [EXISTS]
    • View Profile
Re: Dwarf Fortress mySQL Database
« Reply #1 on: September 19, 2013, 07:17:20 pm »

Hello, I am a Perl coder.
Logged

Meph

  • Bay Watcher
    • View Profile
    • worldbicyclist
Re: Dwarf Fortress mySQL Database
« Reply #2 on: September 20, 2013, 10:51:01 am »

You know where to find me. Any additional info you need, just ask.
Logged
::: ☼Meph Tileset☼☼Map Tileset☼- 32x graphic sets with TWBT :::
::: ☼MASTERWORK DF☼ - A comprehensive mod pack now on Patreon - 250.000+ downloads and counting :::
::: WorldBicyclist.com - Follow my bike tours around the world - 148 countries visited :::

JodGap

  • Bay Watcher
    • View Profile
Re: Dwarf Fortress mySQL Database
« Reply #3 on: September 21, 2013, 06:33:29 am »

Related to armor tokens, STRUCTURAL_ELASTICITY_WOVEN_THREAD, STRUCTURAL_ELASTICITY_CHAIN_METAL, STRUCTURAL_ELASTICITY_CHAIN_ALL can be used together or you can only use one?
What is the max number TILE can have?
What is the max number VALUE can have?
What is the max number SIZE can have?
Logged

Meph

  • Bay Watcher
    • View Profile
    • worldbicyclist
Re: Dwarf Fortress mySQL Database
« Reply #4 on: September 21, 2013, 08:53:53 am »

the structual ones can be used all together.
tile:255 is max
value:open ended, probably some 2^10 or 2^12 or so. very large. just go with 1,000,000 and you should be fine. It can also be negative though, so smaller then 0 is valid as well.
size: same as above, but negatives dont work. But size is open ended as well, but at some point in the very, very large numbers it breaks.
Logged
::: ☼Meph Tileset☼☼Map Tileset☼- 32x graphic sets with TWBT :::
::: ☼MASTERWORK DF☼ - A comprehensive mod pack now on Patreon - 250.000+ downloads and counting :::
::: WorldBicyclist.com - Follow my bike tours around the world - 148 countries visited :::

JodGap

  • Bay Watcher
    • View Profile
Re: Dwarf Fortress mySQL Database
« Reply #5 on: September 21, 2013, 11:43:09 am »

Q: What happens when you create a weapon, trapcomp or ammo and do not fill it with an ATTACK? Is it Mandatory to includy ATTACK on them or is optional?
Q: Can you create a RANGED that uses no AMMO or requires no SKILL? (In the wiki they are not marked as Required)
« Last Edit: September 21, 2013, 12:05:36 pm by JodGap »
Logged

Meph

  • Bay Watcher
    • View Profile
    • worldbicyclist
Re: Dwarf Fortress mySQL Database
« Reply #6 on: September 21, 2013, 01:05:44 pm »

Are these necessary? Because they dont exist, neither in mods nor vanilla. If you want to know, test it yourself. :)

I assume this:
Weapon/trapcomp/ammo will either crash the game or take some default value, or pass through the target.
Ranged with no ammo: sure thing, but its a melee weapon then, because it will never load any ammo.
ranged with no skill: not sure if dwarves would use it.
Logged
::: ☼Meph Tileset☼☼Map Tileset☼- 32x graphic sets with TWBT :::
::: ☼MASTERWORK DF☼ - A comprehensive mod pack now on Patreon - 250.000+ downloads and counting :::
::: WorldBicyclist.com - Follow my bike tours around the world - 148 countries visited :::

Putnam

  • Bay Watcher
  • DAT WIZARD
    • View Profile
Re: Dwarf Fortress mySQL Database
« Reply #7 on: September 21, 2013, 01:53:49 pm »

You can create one, but it'd be pointless. No skill will almost surely work.

JodGap

  • Bay Watcher
    • View Profile
Re: Dwarf Fortress mySQL Database
« Reply #8 on: September 21, 2013, 01:59:20 pm »

I ask this because when we import the RAWs into mySQL we can add some Rules. Right now i am creating if it must be mandatory for a ranged to have skill and ammo.
If Our perl parser reads a raw about a weapon with RANGED token that does not include SKILL and AMMO then an alert pops up, and we can inform the modder that something is wrong.
Altough we will not be able to add an incomplete ranged weapon to the DB if we enable that. Do you follow me?

You as modders what would prefer?
Logged

Putnam

  • Bay Watcher
  • DAT WIZARD
    • View Profile
Re: Dwarf Fortress mySQL Database
« Reply #9 on: September 21, 2013, 02:00:07 pm »

It isn't mandatory, but it should be considered such.

MagmaMcFry

  • Bay Watcher
  • [EXISTS]
    • View Profile
Re: Dwarf Fortress mySQL Database
« Reply #10 on: September 21, 2013, 08:00:44 pm »

I'd say that the modder would find stuff like that out by himself. How about you just don't make anything mandatory that could ever possibly be omitted, so we can load any kind of raws into memory that don't have a tag format error when initially parsed. I'd actually suggest that you don't make melee weapons and ranged weapons different tables at all (it seems like you're doing that), more generally I'd say that you always use the same table for objects that have the same object constructor (object constructors are [ITEM_WEAPON:], [REACTION:], [BODY:], [ATTACK:], [SYNDROME], etc.).
Logged

MagmaMcFry

  • Bay Watcher
  • [EXISTS]
    • View Profile
Re: Dwarf Fortress mySQL Database
« Reply #11 on: September 24, 2013, 07:01:53 pm »

Alright, I put my current progress up on Github here. So far I can fully read (and reproduce, for testing) all [OBJECT:BODY] raws.
Logged

Mr S

  • Bay Watcher
    • View Profile
Re: Dwarf Fortress mySQL Database
« Reply #12 on: September 24, 2013, 07:14:11 pm »

This is relevant to my interests.

However, it may be outside the scope of my ability to contribute in any material way.  I would, however, be happy to participate in discussion and/or advisement for this.
Logged

Maklak

  • Bay Watcher
    • View Profile
Re: Dwarf Fortress mySQL Database
« Reply #13 on: October 03, 2013, 05:17:33 am »

Thank you for the invite. When (if?) this works, it should make my own scripts (almost?) obsolete, which is fine. After I'm done with all of them (and am doing other stuff IRL now, so no progress), a logical step was to do something to bind them all, like a master CGI script or somesuch and I just mentally cringe at all the work it would take to rip my scripts to modules, glue them back together, add links everywhere and write a manual explaining how to set up a web server on a home computer.

I won't be joining your team, but by the looks of it, you're better off with MagmaMcFry anyway.
I didn't use databases in my scripts and kept everything in memory in lots of arrays and hashes, but I imagine at some point using a database pays off by actually being less complex and faster, even for non-persistent data. 
I didn't use objects because I wanted all my scripts to be single files. It lead to some redundancy and while I'm fine with procedural programming, McFry's approach of making lots of small modules is more professional. Personally I'd go for fewer and bigger modules, but "there is more than one way to do it".
Overall I think that I suck at perl, am unfamiliar with the rest of your toolchain and don't have anything meaningful to contribute to this project (at least not with very significant effort on my part), but if you need some code from my scripts, just take it. I had to solve some problems to make them work properly, so looking at my code may help when you bump into the same problems. Maybe. http://www.bay12forums.com/smf/index.php?topic=126953.0 If you have any doubts, just ask your programmer how horrible my code is and how much !fun! it would be for him to work with me :D

> - upload a php website so visitors can query information from the database
I'm surprised you're planning to use perl for reading data and php to show it. I think it would be more consistent to just use perl for both. CGI has been around for a long time.

> I ask this because when we import the RAWs into mySQL we can add some Rules. Right now i am creating
> if it must be mandatory for a ranged to have skill and ammo.
My approach would be not to stop reading data on errors such as negative VALUE (even if it can be negative, what's the point? cursed items?) or missing attack values, but to produce a list of problems and present it to the user. In case of say, a weapon with no ATTACK, add that to warnings and display " " or "ERROR" in the attack column. Or just have on top of the first page "Readnng done: NO ERRORS" or "ERRORS" with a link to a page with the list of issues.
Use perl and not the database for error detection (or php, whatever), because database will just refuse to insert / edit something that won't pass validation and you probably still want to show even incomplete data, so it is easier for the modder to figure out what went wrong. So basically, this:
> I'd say that the modder would find stuff like that out by himself. How about you just don't make
> anything mandatory that could ever possibly be omitted, so we can load any kind of raws
> into memory that don't have a tag format error when initially parsed.

I'd say focus on getting it to work at all and not on robust error detection at this point. Because working on something that works and you can see the results of small changes instantly is invaluable for programming. Don't be overly ambitious. Even getting some of the functionality to work is more useful than getting bogged down by code and quitting.

I think Dragons have SIZE 25M, so the limit would be a 32 bit integer. In general, assume most numbers are 32 bit integers. So the limit would be 2^31-1 = 2147483647.

> I want easy access to information, and reading over 30 wiki pages is not fast.
Even more to the point, you can have instant information for any mod. And data in tabular form. That was my motivation for writing my scripts.

> What people would like to search
Plants (mainly crops), creatures, items (mainly weapons and armour), syndromes and interactions, reactions (preferably sorted by building), buildings. Bonus points for adding links with IDs, so a table of plants has links to detailed information about them, chains of syndromes and interactions are linked, you can click for details on syndromes or interactions from creatures and so on. Reactions can have links to products and reagnets, items to reactions they're made with. It gets tangled pretty quickly if you try to do it right.

> People telling me what they are going to search most ( so i can create better indexes )
I don't think speed is going to be a concern with the size of a database from DF RAWs. At least for me the slowest thing was reading the data.

> I'd say that you always use the same table for objects that have the same object constructor
> (object constructors are [ITEM_WEAPON:], [REACTION:], [BODY:], [ATTACK:], [SYNDROME], etc.).
I second this and this is what I did. For example IIRC the only difference between Ranged and Melee weapons are presence or absence of RANGED and AMMO. So just have those columns, fill them with NULLs for melee and when you need to query for just ranged or just melee weapons, return only those that have / don't have a NULL.
Furthermore for some things you may want to not read all the tokens, because there are just so many that aren't relevant for displaying data in a table. In those cases instead of a white list (that is a list of all the valid tokens), I used a blacklist (that is read tokens, process relevant ones from a whitelist, stop processing, store result and unget token on any from the blacklist. If you make your blacklist contain things like EOF, ITEM_, REACTION and so on, this works.

> I can fully read (and reproduce, for testing) all [OBJECT:BODY] raws.
Hm, not the thing I'd start with, but it may be kind of useful to see how all the body parts are connected. I guess.

Oh and this tread might be better off in the "Utilities and 3rd party applications" subforum.
Logged
Quote from: Omnicega
Since you seem to criticize most things harsher than concentrated acid, I'll take that as a compliment.
On mining Organics
Military guide for FoE mod.
Research: Crossbow with axe and shield.
Dropbox referral

JodGap

  • Bay Watcher
    • View Profile
Re: Dwarf Fortress mySQL Database
« Reply #14 on: October 03, 2013, 05:00:39 pm »

Update:

MySQL Workbench added to the tool list.

http://i.imgur.com/Wy2CK9v.png
This is the basic items DB, it will be anexaded to skills and other stuff later.

The SQL code for that .png is done, the DB can be uploaded to MySQL and start to be filled right now.
Logged
Pages: [1] 2 3