Bay 12 Games Forum
Dwarf Fortress => DF Modding => Topic started by: JodGap on September 19, 2013, 07:14:40 pm
-
Dwarf Fortress Database
http://dfdb.info
http://dfdb.info/old
New design
http://i.imgur.com/yQX83YA.png
http://i.imgur.com/SXnUE0v.png
http://i.imgur.com/ZZmwu2m.png
Done
- Items parser
Development
- Interactive dwarf
- Raw import - upload - export for modders
- Import Masterwork manual into the website
Wanted
- Icon font designer
- PHP developer (We use Laravel PHP Framework, Semantic UI CSS Framework, jQuery and mySQL)
- Modders to receive a beta tester account and upload their raws to beta test the raw uploader/parser
Team members
SQL/PHP: Jodgap (http://www.bay12forums.com/smf/index.php?action=profile;u=90864)
Perl: MagmaMcFry (http://www.bay12forums.com/smf/index.php?action=profile;u=29635)GitHub (https://github.com/MagmaMcFry/RawsReader)
RAWS: Meph (http://www.bay12forums.com/smf/index.php?action=profile;u=61165)
-
Hello, I am a Perl coder.
-
You know where to find me. Any additional info you need, just ask.
-
Related to armor tokens, STRUCTURAL_ELASTICITY_WOVEN_THREAD, STRUCTURAL_ELASTICITY_CHAIN_METAL, STRUCTURAL_ELASTICITY_CHAIN_ALL can be used together or you can only use one?
What is the max number TILE can have?
What is the max number VALUE can have?
What is the max number SIZE can have?
-
the structual ones can be used all together.
tile:255 is max
value:open ended, probably some 2^10 or 2^12 or so. very large. just go with 1,000,000 and you should be fine. It can also be negative though, so smaller then 0 is valid as well.
size: same as above, but negatives dont work. But size is open ended as well, but at some point in the very, very large numbers it breaks.
-
Q: What happens when you create a weapon, trapcomp or ammo and do not fill it with an ATTACK? Is it Mandatory to includy ATTACK on them or is optional?
Q: Can you create a RANGED that uses no AMMO or requires no SKILL? (In the wiki they are not marked as Required)
-
Are these necessary? Because they dont exist, neither in mods nor vanilla. If you want to know, test it yourself. :)
I assume this:
Weapon/trapcomp/ammo will either crash the game or take some default value, or pass through the target.
Ranged with no ammo: sure thing, but its a melee weapon then, because it will never load any ammo.
ranged with no skill: not sure if dwarves would use it.
-
You can create one, but it'd be pointless. No skill will almost surely work.
-
I ask this because when we import the RAWs into mySQL we can add some Rules. Right now i am creating if it must be mandatory for a ranged to have skill and ammo.
If Our perl parser reads a raw about a weapon with RANGED token that does not include SKILL and AMMO then an alert pops up, and we can inform the modder that something is wrong.
Altough we will not be able to add an incomplete ranged weapon to the DB if we enable that. Do you follow me?
You as modders what would prefer?
-
It isn't mandatory, but it should be considered such.
-
I'd say that the modder would find stuff like that out by himself. How about you just don't make anything mandatory that could ever possibly be omitted, so we can load any kind of raws into memory that don't have a tag format error when initially parsed. I'd actually suggest that you don't make melee weapons and ranged weapons different tables at all (it seems like you're doing that), more generally I'd say that you always use the same table for objects that have the same object constructor (object constructors are [ITEM_WEAPON:], [REACTION:], [BODY:], [ATTACK:], [SYNDROME], etc.).
-
Alright, I put my current progress up on Github here (https://github.com/MagmaMcFry/RawsReader). So far I can fully read (and reproduce, for testing) all [OBJECT:BODY] raws.
-
This is relevant to my interests.
However, it may be outside the scope of my ability to contribute in any material way. I would, however, be happy to participate in discussion and/or advisement for this.
-
Thank you for the invite. When (if?) this works, it should make my own scripts (almost?) obsolete, which is fine. After I'm done with all of them (and am doing other stuff IRL now, so no progress), a logical step was to do something to bind them all, like a master CGI script or somesuch and I just mentally cringe at all the work it would take to rip my scripts to modules, glue them back together, add links everywhere and write a manual explaining how to set up a web server on a home computer.
I won't be joining your team, but by the looks of it, you're better off with MagmaMcFry anyway.
I didn't use databases in my scripts and kept everything in memory in lots of arrays and hashes, but I imagine at some point using a database pays off by actually being less complex and faster, even for non-persistent data.
I didn't use objects because I wanted all my scripts to be single files. It lead to some redundancy and while I'm fine with procedural programming, McFry's approach of making lots of small modules is more professional. Personally I'd go for fewer and bigger modules, but "there is more than one way to do it".
Overall I think that I suck at perl, am unfamiliar with the rest of your toolchain and don't have anything meaningful to contribute to this project (at least not with very significant effort on my part), but if you need some code from my scripts, just take it. I had to solve some problems to make them work properly, so looking at my code may help when you bump into the same problems. Maybe. http://www.bay12forums.com/smf/index.php?topic=126953.0 If you have any doubts, just ask your programmer how horrible my code is and how much !fun! it would be for him to work with me :D
> - upload a php website so visitors can query information from the database
I'm surprised you're planning to use perl for reading data and php to show it. I think it would be more consistent to just use perl for both. CGI has been around for a long time.
> I ask this because when we import the RAWs into mySQL we can add some Rules. Right now i am creating
> if it must be mandatory for a ranged to have skill and ammo.
My approach would be not to stop reading data on errors such as negative VALUE (even if it can be negative, what's the point? cursed items?) or missing attack values, but to produce a list of problems and present it to the user. In case of say, a weapon with no ATTACK, add that to warnings and display " " or "ERROR" in the attack column. Or just have on top of the first page "Readnng done: NO ERRORS" or "ERRORS" with a link to a page with the list of issues.
Use perl and not the database for error detection (or php, whatever), because database will just refuse to insert / edit something that won't pass validation and you probably still want to show even incomplete data, so it is easier for the modder to figure out what went wrong. So basically, this:
> I'd say that the modder would find stuff like that out by himself. How about you just don't make
> anything mandatory that could ever possibly be omitted, so we can load any kind of raws
> into memory that don't have a tag format error when initially parsed.
I'd say focus on getting it to work at all and not on robust error detection at this point. Because working on something that works and you can see the results of small changes instantly is invaluable for programming. Don't be overly ambitious. Even getting some of the functionality to work is more useful than getting bogged down by code and quitting.
I think Dragons have SIZE 25M, so the limit would be a 32 bit integer. In general, assume most numbers are 32 bit integers. So the limit would be 2^31-1 = 2147483647.
> I want easy access to information, and reading over 30 wiki pages is not fast.
Even more to the point, you can have instant information for any mod. And data in tabular form. That was my motivation for writing my scripts.
> What people would like to search
Plants (mainly crops), creatures, items (mainly weapons and armour), syndromes and interactions, reactions (preferably sorted by building), buildings. Bonus points for adding links with IDs, so a table of plants has links to detailed information about them, chains of syndromes and interactions are linked, you can click for details on syndromes or interactions from creatures and so on. Reactions can have links to products and reagnets, items to reactions they're made with. It gets tangled pretty quickly if you try to do it right.
> People telling me what they are going to search most ( so i can create better indexes )
I don't think speed is going to be a concern with the size of a database from DF RAWs. At least for me the slowest thing was reading the data.
> I'd say that you always use the same table for objects that have the same object constructor
> (object constructors are [ITEM_WEAPON:], [REACTION:], [BODY:], [ATTACK:], [SYNDROME], etc.).
I second this and this is what I did. For example IIRC the only difference between Ranged and Melee weapons are presence or absence of RANGED and AMMO. So just have those columns, fill them with NULLs for melee and when you need to query for just ranged or just melee weapons, return only those that have / don't have a NULL.
Furthermore for some things you may want to not read all the tokens, because there are just so many that aren't relevant for displaying data in a table. In those cases instead of a white list (that is a list of all the valid tokens), I used a blacklist (that is read tokens, process relevant ones from a whitelist, stop processing, store result and unget token on any from the blacklist. If you make your blacklist contain things like EOF, ITEM_, REACTION and so on, this works.
> I can fully read (and reproduce, for testing) all [OBJECT:BODY] raws.
Hm, not the thing I'd start with, but it may be kind of useful to see how all the body parts are connected. I guess.
Oh and this tread might be better off in the "Utilities and 3rd party applications" subforum.
-
Update:
MySQL Workbench added to the tool list.
http://i.imgur.com/Wy2CK9v.png
This is the basic items DB, it will be anexaded to skills and other stuff later.
The SQL code for that .png is done, the DB can be uploaded to MySQL and start to be filled right now.
-
Experimental version:
http://jodgap.kd.io/dfdb/ DOWN
-
This VM has been turned off
-
Will be online for the next 15 minuts http://jodgap.kd.io/dfdb
-
http://dfdb.info/
i bought a domain and rented a host. Now is online 24/7 and needs feedback and new ideas.
-
Heh, this is pretty similar to what my scripts do, http://www.bay12forums.com/smf/index.php?topic=126953.0 except your work is geared more towards users than modders. For example it uses database on the server and can't be quickly updated for local changes, unlike my scripts. It has more eye candy and I like it that unlike with my scripts, you can sort by columns, but overall it is cluttered and not as readable as my tables. Rows having alternating colours is a nice touch.
* Consider adding check-boxes for what to display. I've run into the problem of having too many columns and solved it with command line options, but with a www server, check-boxes and a button are a more obvious solution. We don't need all the information at the same time, just a way to select the columns we want.
* More human-readable names to columns. Keep them short and maybe add some explanatory text at the bottom.
* Some columns can be combined without impacting readability. For example instead of three structural elasticity columns that display "1", have one named "Elasticity" that displays a combination of "Ch. All", "Ch. Met.", "Thread", then explain it below the table.
* Modders need IDs for things.
* By default, sort everything alphabetically by name.
* For boolean things "Yes" and "No" should be more readable than "1" and "".
I seriously consider resuming work on my scripts, then finding a way to integrate them into one tool that can output a set of locally linked web-pages. It would be especially useful for navigating syndrome and interaction chains. Well, I think I know how to do that, but I need a better architecture than "put everything in one file and have like 5k lines of code and a hundred global variables".
-
@Maklak I really appreciate your extended feedback, i was not able to integrate all your tips this time as i was more focused on improving the design.
DFDB with new style: http://dfdb.info/
Please feel free to repeat again whatever suggestion that can be done to this new model :D
-
Adding a value calculator would be incredible.
-
One word: workshops.
Workshops and furnaces (with lists of what they can make) would be an incredible help to new players (or just people new to a mod)
I assume that they would be rather difficult to add though...
-
> I assume that they would be rather difficult to add though...
Not that difficult. The relevant building data are just these tokens:
[BUILDING_WORKSHOP:(.*)], [NAME:(.*)], [BUILD_ITEM:1:BLOCKS:NONE:NONE:NONE][CAN_USE_ARTIFACT]
Then there are reactions which need to remember their ID, building, name, skill and keep the lists of reagents and products. Their scripts already read item data too, so most of reagents and products can even use IDs instead of names. It is just some butt in the chair work to implement all this. About the only complication I see is how to present the data clearly.
-
About the only complication I see is how to present the data clearly.
That's sorta what I meant ;)
-
Going through all threads, and this is rather awesome. :)
Is the source code available anywhere? (Please say github)
~ T
-
Source it not available as far as I know, but I want this integrated into MDF somehow. Either as link in the manual/gui (if it is permanently online), or as downloaded utility which is added into the GUI.
This database is too good not to be included. :) It also shows the hundrets (thousands?) of unused and unneeded item-leftovers in the mod, which helps me clean it up.
-
The website domain and server was already paid for a year, so it is going to be online.
It will be improved eventually and store more things like reactions buildings creatures...
The source code will not be open for public until the website is finished, it can only be shared with other coders that join our team to improve it.
-
In that case I will add a button to the GUI that opens the website. Simple. :)
-
I'm not exactly sure if this the right time or place to say this, but in the masterwork section, loads of (maybe) weapons and (definitely) armor can be made of more materials than are currently listed. This is due to the Temple of Armok's gear-sacrificing reactions, where the product inherits the reagent's material. I've made leather versions of pretty much every high-level armor with that reaction. Granted, I guess that, except for possible applications in the inventor's workshop,the products aren't as useful as metal versions, but nevertheless . . .
-
It reads out directly from the raws, the items raws. It does not include the custom reactions, that would be near impossible to read out.
-
looking pretty good so far!
i don't trust the masterwork manual anymore and raw hunting is tedious, so it would be really nice to see more armor information. properties like COVERAGE (150% coverage? does that even work?), LAYER, and SHAPED tokens would be a huge help. size, permit, UB/LB/UP/STEP could also be helpful to more experienced players, i'd imagine. bonus points for adding grouping ;)
-
According to the wiki, 100+% do have an affect, but not much science has been done.
-
Remake in the process. Proyect is still alive and being developed so it can have more features:
Colaborators still wanted.
http://www.bay12forums.com/smf/index.php?topic=138865.0