Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 [2] 3

Author Topic: DF Modloader mockup (and discussion on how to load mods in Steam DF)  (Read 7366 times)

voliol

  • Bay Watcher
    • View Profile
    • Website
Re: DF Modloader mockup
« Reply #15 on: August 27, 2021, 01:15:17 pm »

I have some questions about the details, but first I should mention that I think creature variations should be generalized for all objects, otherwise token removal and token converts can't be done to them. In the mockup I call these object variations, but the name doesn't really matter. In the rest of this post, everything I say about creature variations will be equally about the generalized form. Creature variations by name (and all their tokens) can still remain as a synonym for the sake of backwards compatibility.
Maybe so; I'm not 100% sure that creature variations truly need to be generalized to everything; I mean, the functionality of removing and converting tokens of course should be generalized naturally, but for instance, descriptor color patterns don't really require the complexity of object variations (same goes for many other objects).

I think a unified approach is the way to go, rather than having "plant variations" and "entity variations" etc., having one new object type with all the powers of creature variations, but which can be used for any object, not just creatures. So the syntax would be e.g. APPLY_OBJECT_VARIATION:EXAMPLE_OV and OV_REMOVE_TOKEN:token for both a plant and an entity (and a creature).
Again, the reason we want these is that removing tokens and changing their arguments is possible when manually changing files, but with the current propositions for the mod loader, this is only possible through CV_REMOVE_TAG and CV_CONVERT_TAG. If hypothetical EDIT-level removal/conversion tokens are allowed to remove/convert any tokens, and not just the special+cv ones, then they can do it too, but as I argue below I still think they should be restricted.
Of course, if the handling of the objects is separated enough in Toady's code that this wouldn't be possible then yeah, the descriptors and colors and word definition probably don't need a creature variation counterpart of their own. On the other hand, there's a chance that adding an generalized/unified "object variation" is no harder than adding a single "plant variation" or "entity variation". It wasn't that difficult for the mockup.

Quote
For the first question, why are both points 1&2 and 3&4 needed? They seem to me like they're doing the same thing, and it doesn't need to be read twice if all the creature variations etc. are processed at the end, right?
The point of having the separate steps was to illustrate how it works with different mods; step 1 is basically the same thing as step 2, but step 2 makes additional overwrites/changes to the data collected from step 1.

In other words, step 1 is gathering the vanilla data, and step 2 is gathering the mods contents and editing what was gathered in step 1. Technically "step 2" would be repeated for every mod.

No, I mean, what's the difference between the pairs of 1&2, and 3&4. I might be missing something really obvious here, in which case I'm sorry.

Quote
The second question is about the syntax for changing creature variations (and special tokens like APPLY_CREATURE_VARIATION). Is it optimal to use CV_CONVERT_TOKEN and CV_REMOVE_TOKEN? I don't think so. If the systems are applied separately, they should be delineated when it comes to the tokens used as well. Basically, add copies of all the CV_ tokens to act at the EDIT-level.
Code: [Select]
[EDIT:CREATURE:BY_ID:GIANT_TOAD]
[REMOVE_SPEC_TAG:CV_REMOVE_TAG:CHANGE_BODY_SIZE_PERC]
These may seem redundant, as they could in theory be used for doing all creature variation action at EDIT-level. This is not something we want people to do, because then we've lost the point of EDIT not mimicking what can be done through creature variations. I think the natural solution would be to lock these tokens to only work with CV_ tokens and special tokens, so this example above is possible, but this isn't:
Code: [Select]
[EDIT:CREATURE:BY_ID:GIANT_TOAD]
[REMOVE_SPEC_TAG:PET_EXOTIC]
Hm, you're right in a sense, but I'm not 100% sure on the tag being restricted to special tokens. The idea I had in mind for EDIT is to be like putting code directly in the actual raw text, exactly as if you were typing it in manually, but only in small pieces (so other mods can make their own changes).
...
But I'm not sure this is the right approach; I mean, maybe it is good enough, but I feel uneasy about it for some reason. Could be complications with mods further on in the load order making their own changes maybe.

Please try to come up with some examples of it breaking stuff. I am thinking it is better to do all removals/conversions at step 6 (except for removals/conversions of special tokens, of course), because that is a simpler system and I don't know for what you would need to do removals/conversions any sooner. To me, it seems like an unneeded opportunity to do missteps.

Quote
Edit: I realized that if EDIT goes first, then the EDIT selection won't be able to target depending on tokens added/removed/converted by special tokens. I'm not sure how big an issue this is, though. For one, COPY_TAGS_FROM would mean that even if you're only able to target the base creature (by e.g. a creature class), the changes made by EDIT would be copied over to the animal man and giant versions of the creature.
And you could select by the creature variation tokens as well. Say you want to select all intelligent creatures, then you wouldn't be able to just use [CREATURE:EDIT:BY_TOKEN:INTELLIGENT] because animal people don't have the INTELLIGENT token before their creature variation is applied. But you could do
Code: [Select]
[CREATURE:EDIT:BY_TOKEN:INTELLIGENT]
[CREATURE:PLUS_SELECT:BY_TOKEN_PRECISE:APPLY_CREATURE_VARIATION:ANIMAL_PERSON]
[CREATURE:PLUS_SELECT:BY_TOKEN_PRECISE:APPLY_CREATURE_VARIATION:ANIMAL_PERSON_LEGLESS]
Still, this depends on the creature variations adding said tokens being known, so it might be grounds for rethinking the order.
That's a good point; I mean, if you were to look around the raws with CTRL+F you wouldn't be able to find INTELLIGENT inside animal people, that's something you just have to know and expect with animal people.

Maybe in the same vein that if you want to edit CRAB and CRAB_MAN and GIANT_CRAB all together, you can just select the original CRAB and rely on COPY_TAGS_FROM to do the rest for you, you could edit just the ANIMAL_PERSON creature variation, and use [APPLY_CREATURE_VARIATION:ANIMAL_PERSON] to carry the changes over?

That would definitely be safer than your example with PLUS_SELECT, because what if another mod were to change ANIMAL_PERSON_LEGLESS to not be intelligent anymore?

It's tricky; most tokens are pure inert data (ie declarative), but the creature variation tokens are non-data commands that make changes to the data, and the game ultimately loads in the pure data left over after these commands (this thread complains about the raws mixture of declarative and non-declarative code: http://www.bay12forums.com/smf/index.php?topic=63200.0).

You could even do [CREATURE_VARIATION:EDIT:BY_TOKEN_PRECISE:CV_ADD_TAG:INTELLIGENT], to catch all custom animal people variations. I'm feeling more safe with this system now. It is finicky for sure, but as long as you can do all things you'd want to (in this case making changes to all intelligent creatures), that's less of a problem.

And of course someone noticed the weird mixture back in 2010. There's always a thread from 2010 :P.

Mr Crabman

  • Bay Watcher
  • A person with the head and pincers of a crab.
    • View Profile
Re: DF Modloader mockup
« Reply #16 on: August 27, 2021, 04:32:24 pm »

I think a unified approach is the way to go, rather than having "plant variations" and "entity variations" etc., having one new object type with all the powers of creature variations, but which can be used for any object, not just creatures. So the syntax would be e.g. APPLY_OBJECT_VARIATION:EXAMPLE_OV and OV_REMOVE_TOKEN:token for both a plant and an entity (and a creature).
Again, the reason we want these is that removing tokens and changing their arguments is possible when manually changing files, but with the current propositions for the mod loader, this is only possible through CV_REMOVE_TAG and CV_CONVERT_TAG. If hypothetical EDIT-level removal/conversion tokens are allowed to remove/convert any tokens, and not just the special+cv ones, then they can do it too, but as I argue below I still think they should be restricted.
Of course, if the handling of the objects is separated enough in Toady's code that this wouldn't be possible then yeah, the descriptors and colors and word definition probably don't need a creature variation counterpart of their own. On the other hand, there's a chance that adding an generalized/unified "object variation" is no harder than adding a single "plant variation" or "entity variation". It wasn't that difficult for the mockup.

I think there needs to be some kind of specialization, even if just [OBJECT_VARIATION:CREATURE:ANIMAL_PERSON] or something like that, because what happens when you try to OV_ADD_TAG tokens from 2 different object types in the same variation? What if you try to apply an invalid variation to something?

Also, this is a bit of a pipe dream/unlikely hope, but if they're kept separate/specialized, one could conceivably scrap the CV_NEW_TAG syntax and just directly write in the tags, so instead of this:
Code: [Select]
[CV_NEW_TAG:CAN_DO_INTERACTION:PET_ANIMAL]
[CV_NEW_TAG:CDI:ADV_NAME:Pet animal]
[CV_NEW_TAG:CDI:USAGE_HINT:GREETING]
[CV_NEW_TAG:CDI:BP_REQUIRED:BY_TYPE:GRASP]
[CV_NEW_TAG:CDI:VERB:pet:pets:pets]
[CV_NEW_TAG:CDI:TARGET:A:SELF_ONLY]
[CV_NEW_TAG:CDI:TARGET:B:TOUCHABLE]
[CV_NEW_TAG:CDI:TARGET_RANGE:B:1]
[CV_NEW_TAG:CDI:MAX_TARGET_NUMBER:B:1]
[CV_NEW_TAG:CDI:WAIT_PERIOD:20]

You'd have this:
Code: [Select]
[CAN_DO_INTERACTION:PET_ANIMAL]
[CDI:ADV_NAME:Pet animal]
[CDI:USAGE_HINT:GREETING]
[CDI:BP_REQUIRED:BY_TYPE:GRASP]
[CDI:VERB:pet:pets:pets]
[CDI:TARGET:A:SELF_ONLY]
[CDI:TARGET:B:TOUCHABLE]
[CDI:TARGET_RANGE:B:1]
[CDI:MAX_TARGET_NUMBER:B:1]
[CDI:WAIT_PERIOD:20]

Which would just be so much nice and neater, allowing the likes of easily copy/pasting things into an object variation from an actual object (instead of having to add in CV_NEW_TAG one by one).

No, I mean, what's the difference between the pairs of 1&2, and 3&4. I might be missing something really obvious here, in which case I'm sorry.

1&2 is for CREATURE_VARIATION objects, and 3&4 is for CREATURE objects (because every single CREATURE_VARIATION needs to be edited by all mods, before being applied to even vanilla creatures).

Please try to come up with some examples of it breaking stuff. I am thinking it is better to do all removals/conversions at step 6 (except for removals/conversions of special tokens, of course), because that is a simpler system and I don't know for what you would need to do removals/conversions any sooner. To me, it seems like an unneeded opportunity to do missteps.

My worry is that the mods need to be applied in order; a top-level token needs to be able to reliably re-add a token that was removed, without knowing necessarily that it was removed (rather than just not present to begin with). I'm worried kind of that something like GO_TO_TAG could be used in ways that would place a token like say, PET_VALUE to somewhere high up in the object, and another mod places REMOVE:PET_VALUE at the end, such that no matter what load order the mods are placed in, you would get the tag deleted, which is bad....

.... And now mentioning that has brought to mind/gotten me worried about how REMOVE_SPEC_TAG and adding in tags again would work with the 3 `GO_TO` tokens, since they're not meant to be processed until step 6.

sharp exhale of frustration

Maybe I'm overthinking this, or am I underthinking it....

EDIT: I think I should probably write out some mockups and try to actually follow through the logic of what would happen step by step; maybe that will make it all clearer.

You could even do [CREATURE_VARIATION:EDIT:BY_TOKEN_PRECISE:CV_ADD_TAG:INTELLIGENT], to catch all custom animal people variations. I'm feeling more safe with this system now. It is finicky for sure, but as long as you can do all things you'd want to (in this case making changes to all intelligent creatures), that's less of a problem.

That's right, good catch. Frankly, for a Steam modloading experience that allows neatly combining mods, only the BY_ID function for EDIT and REMOVE are strictly needed, as they allow us to have all the power we have now without having to modify the "base" raws. The rest of PLUS_SELECT and UNSELECT and all that are just bonuses really.
« Last Edit: August 28, 2021, 03:32:07 am by Mr_Crabman »
Logged

voliol

  • Bay Watcher
    • View Profile
    • Website
Re: DF Modloader mockup (and discussion on how to load mods in Steam DF)
« Reply #17 on: August 30, 2021, 09:44:51 am »

I think there needs to be some kind of specialization, even if just [OBJECT_VARIATION:CREATURE:ANIMAL_PERSON] or something like that, because what happens when you try to OV_ADD_TAG tokens from 2 different object types in the same variation? What if you try to apply an invalid variation to something?

The same thing as when you manually enter a token meant for a different object type I imagine, a little error message warning about an unrecognized token. Unless modders are unable to tell them apart, or want to be able to at a glance, specializing them for different object types shouldn't be needed.

Quote
Frankly, for a Steam modloading experience that allows neatly combining mods, only the BY_ID function for EDIT and REMOVE are strictly needed, as they allow us to have all the power we have now without having to modify the "base" raws. The rest of PLUS_SELECT and UNSELECT and all that are just bonuses really.

True, but I think their usefulness should not be understated. Being able to select by more criteria than just BY_ID makes it possible for mods to bridge together without modders having full knowledge of all other mods out there, including future ones. "Compatibility" addons will still be required in some cases, but many fewer.


For the rest of my replies, see below.

voliol

  • Bay Watcher
    • View Profile
    • Website
Re: DF Modloader mockup (and discussion on how to load mods in Steam DF)
« Reply #18 on: August 30, 2021, 09:49:30 am »

Mr_Crabman, I replied to some of your previous comment in the comment above, but the other point were complicated enough that I felt I needed better tools to get my head around them, and it lead to this. Those replies are at the bottom of this comment, using the model described. I'm sorry for dropping this bomb on you, but at the same time a more complete description like this can be used to base an eventual suggestions-board post on, so I hope you bear with it...


Below is a model for a mod loader for Dwarf Fortress, either one integrated into the game, or a third-party tool. The core parts are the same for either, so in most cases the distinction won't be made. I will assume a familiarity with DF modding.

The player is presented with a list of select-able mods. This list is extracted from a mods folder, the mods presumably coming from Steam Workshop or DFFD. Unselected mods may also be grouped together as "mod packs", or just "mods with multiple components". This is so the player can easily recognize which mods are related, if they have downloaded many of them. Players may select mods, moving them into a separate list, and they may also freely change the order of mods in this "selected" list. The order is guided by the description of each mod, which may include dependencies, other mods that should be higher up in the list.

The mods are loaded top-down. If a list of selected mod looks like this:
Code: [Select]
- Mod A
- Mod C
- Mod B
then they are loaded as
Code: [Select]
1. Mod A
2. Mod C
3. Mod B
Within each mod, files are parsed in order according to their first line. This is how it currently works in vanilla, see https://dwarffortresswiki.org/index.php/DF2014:Raw_file#Parsing. The only difference is that the new "object variations" have priority, so files with "o_variation" are parsed first in each mod. This has less importance than in current DF, but it is still worth mentioning.
If the mod list (expanded so we can see the files in each mod) looks like this:
Code: [Select]
- Mod A
- creature_a
- entity_a
- Mod B
- creature_b
- inorganic_b
- o_variation_b
- Mod C
- entity_c
Then the parsing order is:
Code: [Select]
- Mod A
1. creature_a
2. entity_a
- Mod B
3. o_variation_b
4. inorganic_b
5. creature_b
- Mod C
6. entity_c
Each file contains one or multiple objects, and each object contains tokens. The order of objects withing files, and tokens withing objects, is the natural order within a text file.
An object/token written above is parsed before one written below. The parsing/loading of mods is done in three steps.

  • First, going by parsing order, each object is read.
    If the object is an object variation, it is just saved to memory. That is also the case if the object is anything but an "EDIT" object, so creatures, materials, entities etc. are just saved to memory for now.
    If the object is an EDIT object, then it is not saved to memory. Instead, the EDIT object selects one or more of the already encountered objects, and makes changes to them. Here, it is relevant to divide the tokens into 4 categories: "EDIT-level" tokens, "special" tokens, "object variation" tokens ("ov" tokens for short), and "normal" tokens.

    EDIT-level tokens do something in this first step, and are only found in EDIT objects. PLUS_SELECT and UNSELECT allow you to further specify which objects should be edited. The rest of the EDIT-level tokens (ADD_SPEC_TOKEN, REMOVE_SPEC_TOKEN, CONVERT_SPEC_TOKEN, CST_MASTER, CST_TARGET, and CST_REPLACEMENT) add/remove/convert special tokensand ov tokens from/to/in the selected object(s).

    Special tokens are the three GO_TO tokens GO_TO_END, GO_TO_START, GO_TO_TAG, as well as COPY_FROM_OBJECT, REMOVE_OBJECT (which is a new token), and APPLY_OBJECT_VARIATION and APPLY_CURRENT_OBJECT_VARIATION.
    When these are in EDIT objects, they are copied to the end of the selected object(s).

    Object variation tokens all start with "OV"; OV_ADD_TAG (and the synonymous OV_NEW_TAG), OV_REMOVE_TAG, OV_CONVERT_TAG, OVCT_MASTER, OVCT_TARGET, OVCT_REPLACEMENT, as well as the conditional "_CTAG" counterparts to the first three listed.
    When these are in EDIT objects, they are copied to the end of the selected object(s).

    Normal tokens are the rest, like NAME and BRAG_ON_KILL, but also ones that may have syntax meaning simply not relevant to the mod loading process, like SELECT_CASTE and SYNDROME.
    When these are in EDIT objects, they are not simply copied over, instead they are added using an OV_ADD_TAG with said token as the argument. This is because actually copying over normal tokens complicates things, so OV_ADD_TAGs are disguised so you don't have to type them out. The "disguise" also makes it practical to move tokens from normal objects to EDIT objects.

    When all tokens in the EDIT object have been read, if it added any ov tokens, APPLY_CURRENT_OBJECT_VARIATION is added to the end of the selected object(s).
    Also, if an object has the same object ID as one that has already been saved to memory, then the later object overwrites the former.

  • In step two, all the objects saved to memory are read once more, except the object variations. For each in-memory object, an "output object" is prepared. Then, each token of the object is read, in order. If the token is a special token, then it executes a special command.

    The three GO_TO tokens change the insertion index for new tokens in the output object, moving it to the start, the end, or a specified token. The GO_TO tokens only affect the output object, not the in-memory object currently being parsed.

    COPY_FROM_OBJECT copies tokens from another output object. If that output object has not been created yet, the corresponding in-memory object is given priority, and parsed before copying over the tokens. A loop of COPY_FROM_OBJECT causes an error (message).

    REMOVE_OBJECT prevents the eventual writing down of the output object. This is so COPY_FROM_OBJECT can still copy from an object slated for removal. That technicality aside, what REMOVE_OBJECT does is removing the current object.

    APPLY_OBJECT_VARIATION applies an in-memory object variation. Again, these work like current DF creature variations, but generalized to work not only with creatures. See https://dwarffortresswiki.org/index.php/DF2014:Creature_variation_token
    APPLY_OBJECT_VARIATION also takes arbitrary arguments which can be used with the "_CTAG" tokens.

    APPLY_CURRENT_OBJECT_VARIATION applies the ov tokens that have directly preceded it in the current object.
    If the token is an ov token, it is saved up (with others like it) and applied at the next APPLY_CURRENT_OBJECT_VARIATION.
    If the token is a normal token, it is inserted into the output object's list of tokens. Initially, this means they are added to the end, but the GO_TO tokens can change this.

  • Step three is taking all the output objects, except the ones slated for removal with REMOVE_OBJECT, and writing them into "compiled" text files (as compared to the original list of mods, they are not compiled per-say. If we imagine the mod loader to be integrated into Dwarf Fortress, these compiled files are likely what a save uses, so it won't have to parse the more involved list of mods each time it is opened. If we imagine the mod loader to be third-party, these compiled files are what are dragged into the game's raws folder, as one would install a total conversion mod in current Dwarf Fortress.

Some further thoughts:

Body detail plans could be made into OBJECT_VARIATIONs. As far as I can tell, they already work the same as diminished creature variations only able to add tokens. Body details plan are older than the latter, I believe, but the overlap still seems strange.

All tokens related to creature variations could be kept as synonyms for their corresponding object variation-related token, for the sake of backwards compability. At the same time, this could be problematic if targeted by REMOVE_SPEC_TOKEN/CONVERT_SPEC_TOKEN, if a search for e.g. OV_ADD_TAG doesn't find instances of CV_ADD_TAG. There is a similar problem with OV_ADD_TAG and OV_NEW_TAG, and to some extent the "_CTAG" tokens. Perhaps some repetition will be needed in the EDIT objects to cover all cases.

As APPLY_CURRENT_OBJECT_VARIATION is only added automatically at the end of the EDIT object, this may lead to some misleading EDIT objects. E.g.
Code: [Select]
[EDIT:CREATURE:BY_ID:TOAD]
[GO_TO_START]
[ADOPTS_OWNER]
[GO_TO_END]
gives
Code: [Select]
[CREATURE:TOAD]
...
[GO_TO_START]
[OV_ADD_TAG:ADOPTS_OWNER]
[GO_TO_END]
[APPLY_CURRENT_OBJECT_VARIATION]
which adds ADOPTS_OWNER at the end of TOAD, not at the start. Maybe APPLY_CURRENT_OBJECT_VARIATION should be added not only at the end of the block made using EDIT, but before any GO_TO tokens it adds.

I'm thinking of ways that [IF]-blocks or other conditionals could be added. I think the way might be something similar to the current creature variation arguments, where all occurrences of a text string are replaced by a conditional value, at the very start of loading a mod. Lets call these "mod arguments". Then, the definitions for the mod arguments is found in a text found of their own. Something like:
Code: [Select]
...
[CONDITIONAL:CAN_DWARVES_TAME_DRAGONS]
[DESCRIPTION:Allow dwarves to tame dragons?]
[COND_TYPE:BOOLEAN]
[CURRENT_VALUE:TRUE]
...
or maybe
Code: [Select]
...
Allow dwarves to tame dragons?
[CAN_DWARVES_TAME_DRAGONS:BOOLEAN:TRUE]
...
Either way, I hope that some of the GUI work here can be repurposed from reading init or advanced world generation settings files,
in an integrated version. They should have similar requirements, and having to know whether a gui element is supposed to be a boolean, an integer, a multi-choice option, or an arbitrary string (like the world name in advanced world gen).
And also there should be some token that checks whether a mod argument has a certain value, and allows/restricts other tokens based on that. I'm not sure when this would be done, or how it would work with the other parts of the mod loader described, there's a lot to figure out here.



Now, to finally making those replies using this model:

Also, this is a bit of a pipe dream/unlikely hope, but if they're kept separate/specialized, one could conceivably scrap the CV_NEW_TAG syntax and just directly write in the tags, so instead of this:
Code: [Select]
[CV_NEW_TAG:CAN_DO_INTERACTION:PET_ANIMAL]
[CV_NEW_TAG:CDI:ADV_NAME:Pet animal]
[CV_NEW_TAG:CDI:USAGE_HINT:GREETING]
[CV_NEW_TAG:CDI:BP_REQUIRED:BY_TYPE:GRASP]
[CV_NEW_TAG:CDI:VERB:pet:pets:pets]
[CV_NEW_TAG:CDI:TARGET:A:SELF_ONLY]
[CV_NEW_TAG:CDI:TARGET:B:TOUCHABLE]
[CV_NEW_TAG:CDI:TARGET_RANGE:B:1]
[CV_NEW_TAG:CDI:MAX_TARGET_NUMBER:B:1]
[CV_NEW_TAG:CDI:WAIT_PERIOD:20]

You'd have this:
Code: [Select]
[CAN_DO_INTERACTION:PET_ANIMAL]
[CDI:ADV_NAME:Pet animal]
[CDI:USAGE_HINT:GREETING]
[CDI:BP_REQUIRED:BY_TYPE:GRASP]
[CDI:VERB:pet:pets:pets]
[CDI:TARGET:A:SELF_ONLY]
[CDI:TARGET:B:TOUCHABLE]
[CDI:TARGET_RANGE:B:1]
[CDI:MAX_TARGET_NUMBER:B:1]
[CDI:WAIT_PERIOD:20]

Which would just be so much nice and neater, allowing the likes of easily copy/pasting things into an object variation from an actual object (instead of having to add in CV_NEW_TAG one by one).
If we allow EDIT objects to include disguised OV_ADD_TAGs, it could also be done with the object variations; when reading an ov object in step 1,
replace all non-ov tokens with OV_ADD_TAG:whatever-token-that-was.
They don't need to be separate/specialized for this though; ov objects will never include any non-ov tokens so its obvious that any token that appears that way must
be a disguised OV_ADD_TAG (or OV_NEW_TAG, the synonymous tokens still weird me out tbh).

Quote
No, I mean, what's the difference between the pairs of 1&2, and 3&4. I might be missing something really obvious here, in which case I'm sorry.

1&2 is for CREATURE_VARIATION objects, and 3&4 is for CREATURE objects (because every single CREATURE_VARIATION needs to be edited by all mods, before being applied to even vanilla creatures).
This is not needed in the above model, because APPLY_OBJECT_VARIATION is a special token processed in step 2. All edits to object variations and objects such as creatures alike
are done in step 1.

Quote
My worry is that the mods need to be applied in order; a top-level token needs to be able to reliably re-add a token that was removed, without knowing necessarily that it was removed (rather than just not present to begin with). I'm worried kind of that something like GO_TO_TAG could be used in ways that would place a token like say, PET_VALUE to somewhere high up in the object, and another mod places REMOVE:PET_VALUE at the end, such that no matter what load order the mods are placed in, you would get the tag deleted, which is bad....

.... And now mentioning that has brought to mind/gotten me worried about how REMOVE_SPEC_TAG and adding in tags again would work with the 3 `GO_TO` tokens, since they're not meant to be processed until step 6.
This is why the EDIT-level tokens can only make changes to special tokens, and why the edit objects add OV_ADD_TAGs instead of directly adding tokens.
During step 1 (i.e. when edits are being made), all edits done to normal tokens remain as unprocessed ov tokens in the in-memory objects, and can thus be reverted.

As the GO_TO tokens are special-only and have no EDIT-level counterparts, edits can't place ov tokens anywhere but the end of the selected object(s),
and so they will be processed in step 2 in the same order as the edits were parsed, top-down in the list of selected mods.
Say we have the raws of the TOAD, and a mod A:
Code: [Select]
[CREATURE:EDIT:BY_TOKEN:TOAD]
[GO_TO_END]
[OV_REMOVE_TAG:PET_VALUE]
and mod B:
Code: [Select]
[CREATURE:EDIT:BY_TOKEN:TOAD]
[GO_TO_START]
[PET_VALUE:50]
loaded in that order. After step 1, we would have:
Code: [Select]
[CREATURE:TOAD]
...
[CREATURE_TILE:249]
[COLOR:2:0:0]
[PET_VALUE:10]
...
[GO_TO_END]
[OV_REMOVE_TAG:PET_VALUE]
[APPLY_CURRENT_OBJECT_VARIATION]
[GO_TO_START]
[OV_ADD_TAG:PET_VALUE:50]
[APPLY_CURRENT_OBJECT_VARIATION]
. In step 2, this would be processed line-by-line, and as we can see REMOVE_TOKEN would be processed/applied first. As REMOVE_TOKEN does not add a token,
it is unaffected by the change of insertion index made by GO_TO_END. GO_TO_START still affects where PET_VALUE is inserted. Ultimately, we would get the desired result:
Code: [Select]
[CREATURE:TOAD]
[PET_VALUE:50]
...
[CREATURE_TILE:249]
[COLOR:2:0:0]
...

Mr Crabman

  • Bay Watcher
  • A person with the head and pincers of a crab.
    • View Profile
Re: DF Modloader mockup (and discussion on how to load mods in Steam DF)
« Reply #19 on: August 30, 2021, 06:20:48 pm »

The same thing as when you manually enter a token meant for a different object type I imagine, a little error message warning about an unrecognized token. Unless modders are unable to tell them apart, or want to be able to at a glance, specializing them for different object types shouldn't be needed.

I'm gonna be upfront, a major part of the reason I want them to somehow be specialized/differentiable on a syntax-level is so that it's actually practical/possible for the DF Raw Language Server extension to operate on them usefully; the raws are already in general way too lightweight on syntax (and heavy on context/semantics) already.

True, but I think their usefulness should not be understated. Being able to select by more criteria than just BY_ID makes it possible for mods to bridge together without modders having full knowledge of all other mods out there, including future ones. "Compatibility" addons will still be required in some cases, but many fewer.

True, though it does still seem like a lower priority (particularly since Tarn apparently doesn't have much time to work on mod support for the initial Steam release).

The player is presented with a list of select-able mods. This list is extracted from a mods folder, the mods presumably coming from Steam Workshop or DFFD. Unselected mods may also be grouped together as "mod packs", or just "mods with multiple components". This is so the player can easily recognize which mods are related, if they have downloaded many of them. Players may select mods, moving them into a separate list, and they may also freely change the order of mods in this "selected" list. The order is guided by the description of each mod, which may include dependencies, other mods that should be higher up in the list.

The mods are loaded top-down. If a list of selected mod looks like this:
Code: [Select]
- Mod A
- Mod C
- Mod B
then they are loaded as
Code: [Select]
1. Mod A
2. Mod C
3. Mod B

Assuming this is to be posted as a suggestion, I would be a bit clearer about the nature of modpacks and how they are packaged possibly; this reading here gives the impression that grouping mods is done by the end-user as a purely visual thing, and doesn't suggest that it can be done with Steam Workshop or DFFD files by putting multiple mods into a single modpack folder (which then goes inside the top-level mod folder).

Also, you should note that vanilla files would be in this list of mods as well (though it shouldn't be present in the actual "mods" folder presumably), able to be enabled or disabled.

However, something to look out for is that apparently just disabling vanilla and not having equivalently named files (and with some similarly named objects?) will break the game; I'm not sure what to do about this: http://www.dwarffortresswiki.org/index.php/DF2014:Required_Raw_Files

The mods are loaded top-down...

Unnecessarily picky here since it doesn't really matter, but I feel like the mod list should be in reverse; mods on the bottom of the selected list are loaded first (maybe it's just Minecraft resource packs forging this intuition of mine).

Body detail plans could be made into OBJECT_VARIATIONs. As far as I can tell, they already work the same as diminished creature variations only able to add tokens. Body details plan are older than the latter, I believe, but the overlap still seems strange.

Funny enough, Tarn seems to have had a similar idea: https://www.bay12games.com/dwarves/dev_2009.html

All tokens related to creature variations could be kept as synonyms for their corresponding object variation-related token, for the sake of backwards compability. At the same time, this could be problematic if targeted by REMOVE_SPEC_TOKEN/CONVERT_SPEC_TOKEN, if a search for e.g. OV_ADD_TAG doesn't find instances of CV_ADD_TAG. There is a similar problem with OV_ADD_TAG and OV_NEW_TAG, and to some extent the "_CTAG" tokens. Perhaps some repetition will be needed in the EDIT objects to cover all cases.

I kind of disagree with this (unless this is to be how); I'm struggling to imagine any mods that would be able to just use this new modloading system without requiring changes, and at that point why not toss out redundant things that also cause issues with future mods since it can't be backwards compatible anyway?

Especially since updating to the new standard would for the most part just consist of "find and replace CV_ with OV_".

As APPLY_CURRENT_OBJECT_VARIATION is only added automatically at the end of the EDIT object, this may lead to some misleading EDIT objects. E.g.
Code: [Select]
[EDIT:CREATURE:BY_ID:TOAD]
[GO_TO_START]
[ADOPTS_OWNER]
[GO_TO_END]
gives
Code: [Select]
[CREATURE:TOAD]
...
[GO_TO_START]
[OV_ADD_TAG:ADOPTS_OWNER]
[GO_TO_END]
[APPLY_CURRENT_OBJECT_VARIATION]
which adds ADOPTS_OWNER at the end of TOAD, not at the start. Maybe APPLY_CURRENT_OBJECT_VARIATION should be added not only at the end of the block made using EDIT, but before any GO_TO tokens it adds.

Maybe so; an alternative to that solution could be to just scrap APPLY_CURRENT_OBJECT_VARIATION and make immediate changes for each OV_ADD and OV_REMOVE, and to come up with some other way to deal with OV_CONVERT.

It would also probably make sense (assuming you haven't already implicitly included it), to have an implicit "GO_TO_END" at the start of every new EDIT object.

This is why the EDIT-level tokens can only make changes to special tokens, and why the edit objects add OV_ADD_TAGs instead of directly adding tokens.
During step 1 (i.e. when edits are being made), all edits done to normal tokens remain as unprocessed ov tokens in the in-memory objects, and can thus be reverted.

As the GO_TO tokens are special-only and have no EDIT-level counterparts, edits can't place ov tokens anywhere but the end of the selected object(s),
and so they will be processed in step 2 in the same order as the edits were parsed, top-down in the list of selected mods.

But if edits can't place GO_TO tokens don't have an EDIT counterpart, how do we handle cases where we need to add a special tag near the start? That is, how does the likes of ADD_SPEC_TOKEN actually place them where desired, in cases where that's relevant (especially if body detail plans were converted to object variations)? Because it seems many tokens are sensitive to token ordering.

This also brings up another question for me; can OBJECT_VARIATIONs be EDITed? (I assume yes, but just in case)

Code: [Select]
[CREATURE:EDIT:BY_TOKEN:TOAD]
[GO_TO_END]
[OV_REMOVE_TAG:PET_VALUE]
and mod B:
Code: [Select]
[CREATURE:EDIT:BY_TOKEN:TOAD]
[GO_TO_START]
[PET_VALUE:50]
loaded in that order. After step 1, we would have:
Code: [Select]
[CREATURE:TOAD]
...
[CREATURE_TILE:249]
[COLOR:2:0:0]
[PET_VALUE:10]
...
[GO_TO_END]
[OV_REMOVE_TAG:PET_VALUE]
[APPLY_CURRENT_OBJECT_VARIATION]
[GO_TO_START]
[OV_ADD_TAG:PET_VALUE:50]
[APPLY_CURRENT_OBJECT_VARIATION]
. In step 2, this would be processed line-by-line, and as we can see REMOVE_TOKEN would be processed/applied first. As REMOVE_TOKEN does not add a token,
it is unaffected by the change of insertion index made by GO_TO_END. GO_TO_START still affects where PET_VALUE is inserted. Ultimately, we would get the desired result:
Code: [Select]
[CREATURE:TOAD]
[PET_VALUE:50]
...
[CREATURE_TILE:249]
[COLOR:2:0:0]
...

So this all works out nicely in this example, but how does this interact with the likes of REMOVE_SPEC_TOKEN?



Other than all that, this looks okay, though I'd have a suggestion to change the syntax of tag conversion, which to me seems unnecessarily long; to make one single change, you need 4 different tokens, when with different syntax at most you'd need 3, and could go as low as 1 + 2 for every replacement you wish to make to the same "master" token.

Current:
Code: [Select]
[CV_CONVERT_TAG]
[CVCT_MASTER:BODY]
[CVCT_TARGET:QUADRUPED]
[CVCT_REPLACEMENT:HUMANOID]
[CV_CONVERT_TAG]
[CVCT_MASTER:BODY]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
[CVCT_REPLACEMENT:HUMANOID]

1st idea:
Code: [Select]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED]
[CVCT_REPLACEMENT:HUMANOID]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
[CVCT_REPLACEMENT:HUMANOID]

2nd idea (maybe a little semantic/context heavy though):
Code: [Select]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED]
[CVCT_REPLACEMENT:HUMANOID]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
[CVCT_REPLACEMENT:HUMANOID]

It's a small difference, but it can add up to a big difference when making a lot of conversions, and the 2nd one makes a big difference when handling many conversions to the same "master" token.

Also, if this is for a suggestion and it wouldn't be too bloated, remember the way of selecting what to be converted by token and argument number instead of value that I mentioned before: http://www.bay12forums.com/smf/index.php?topic=178868.msg8306433#msg8306433
« Last Edit: August 31, 2021, 04:56:36 am by Mr_Crabman »
Logged

voliol

  • Bay Watcher
    • View Profile
    • Website
Re: DF Modloader mockup (and discussion on how to load mods in Steam DF)
« Reply #20 on: August 31, 2021, 11:22:51 am »

I'm gonna be upfront, a major part of the reason I want them to somehow be specialized/differentiable on a syntax-level is so that it's actually practical/possible for the DF Raw Language Server extension to operate on them usefully; the raws are already in general way too lightweight on syntax (and heavy on context/semantics) already.

Thanks for being upfront. Your Raw Language Server is something I’m looking forward to, especially the text-editor markup, and I think many other modders would agree with me. I believe a compromise like e.g. [OBJECT_VARIATION:CREATURE:ANIMAL_PERSON] was suggested earlier in this thread, it should do the trick, right?

Quote
The player is presented with a list of select-able mods. This list is extracted from a mods folder, the mods presumably coming from Steam Workshop or DFFD. Unselected mods may also be grouped together as "mod packs", or just "mods with multiple components". This is so the player can easily recognize which mods are related, if they have downloaded many of them. Players may select mods, moving them into a separate list, and they may also freely change the order of mods in this "selected" list. The order is guided by the description of each mod, which may include dependencies, other mods that should be higher up in the list.

The mods are loaded top-down. If a list of selected mod looks like this:
Code: [Select]
- Mod A
- Mod C
- Mod B
then they are loaded as
Code: [Select]
1. Mod A
2. Mod C
3. Mod B

Assuming this is to be posted as a suggestion, I would be a bit clearer about the nature of modpacks and how they are packaged possibly; this reading here gives the impression that grouping mods is done by the end-user as a purely visual thing, and doesn't suggest that it can be done with Steam Workshop or DFFD files by putting multiple mods into a single modpack folder (which then goes inside the top-level mod folder).

Also, you should note that vanilla files would be in this list of mods as well (though it shouldn't be present in the actual "mods" folder presumably), able to be enabled or disabled.

However, something to look out for is that apparently just disabling vanilla and not having equivalently named files (and with some similarly named objects?) will break the game; I'm not sure what to do about this: http://www.dwarffortresswiki.org/index.php/DF2014:Required_Raw_Files

Aye, many things to polish up to make it a proper suggestion post. I’ll probably have another look at it once we have figured out some more details.

Quote
The mods are loaded top-down...

Unnecessarily picky here since it doesn't really matter, but I feel like the mod list should be in reverse; mods on the bottom of the selected list are loaded first (maybe it's just Minecraft resource packs forging this intuition of mine).

The more you know. I could have sworn the resource packs too were loaded top-down, but the MC wiki backs you up... I wonder why they do that? My intuition (based on programming, I reckon) goes against that, and I also have the argument that the raws themselves have some kind of top-down linearity. At the same time, I have limited knowledge of ”mod loaders”, with the only example I know of being top-down are the custom scripts in RPG Maker VXA, and the interface for that isn’t strictly a ”mod loader” either. (Others I haven’t explored, or am pretty sure they are unordered (for the end-user)) If Minecraft does it, maybe going bottom-up is not as unusual as I assume?
Also, I did intentionally design the mockup to look like the Minecraft resource pack screen, so if that look is kept there it wouldn’t be unexpected for people to assume they work the same.

I’m feeling unsure about this, though it doesn’t matter outside of the weirdness factor regardless on one’s stance.

Quote
Body detail plans could be made into OBJECT_VARIATIONs. As far as I can tell, they already work the same as diminished creature variations only able to add tokens. Body details plan are older than the latter, I believe, but the overlap still seems strange.
Funny enough, Tarn seems to have had a similar idea: https://www.bay12games.com/dwarves/dev_2009.html
Huh. That post also contains some other points of interest. One is CVCT_MASTER maybe working differently than we thought, being able to look for multiple tokens (or likely it’s just a string-search). I’ll do some science on that, as I’ll have to update my tools if that’s the case.

Also in that post there is a mention of nesting creature variations (called ”templates”). I don’t know what to think about it, or even the extent that some nesting can be done already, both in current DF and in the ”current” mod loader model. I think Toady is more concerned in the post about how they can be used as templates to reduce copy-pasting (and possibly some procgen myth-stuff?), while we are more concerned to making edits to ”already defined”.

Quote
All tokens related to creature variations could be kept as synonyms for their corresponding object variation-related token, for the sake of backwards compability. At the same time, this could be problematic if targeted by REMOVE_SPEC_TOKEN/CONVERT_SPEC_TOKEN, if a search for e.g. OV_ADD_TAG doesn't find instances of CV_ADD_TAG. There is a similar problem with OV_ADD_TAG and OV_NEW_TAG, and to some extent the "_CTAG" tokens. Perhaps some repetition will be needed in the EDIT objects to cover all cases.

I kind of disagree with this (unless this is to be how); I'm struggling to imagine any mods that would be able to just use this new modloading system without requiring changes, and at that point why not toss out redundant things that also cause issues with future mods since it can't be backwards compatible anyway?

Especially since updating to the new standard would for the most part just consist of "find and replace CV_ with OV_".

Mods that only add new objects instead of making changes would be able to use the mod loader system unchanged. But you are right, fixing these shouldn’t be difficult - if no more syntax changes are introduced then it should just be a matter of searching and replacing.

More importantly (and this I realized first after reading your comment), if the old tokens are deprecated this will break all old saves, because they will still be using the deprecated tokens. If the Steam release breaks saves either way (probably due to changes in how graphics are stored) then this is fine, but less so if it doesn’t. The Steam update won’t add much in the way of new features either, so the incentives for genning a new world will be low. People will want to keep their old saves and some of them will be people who don’t play with mods. Mods are optional and so the framework for them changing should not be forced on the players by breaking saves.

Quote
As APPLY_CURRENT_OBJECT_VARIATION is only added automatically at the end of the EDIT object, this may lead to some misleading EDIT objects. E.g.
Code: [Select]
[EDIT:CREATURE:BY_ID:TOAD]
[GO_TO_START]
[ADOPTS_OWNER]
[GO_TO_END]
gives
Code: [Select]
[CREATURE:TOAD]
...
[GO_TO_START]
[OV_ADD_TAG:ADOPTS_OWNER]
[GO_TO_END]
[APPLY_CURRENT_OBJECT_VARIATION]
which adds ADOPTS_OWNER at the end of TOAD, not at the start. Maybe APPLY_CURRENT_OBJECT_VARIATION should be added not only at the end of the block made using EDIT, but before any GO_TO tokens it adds.

Maybe so; an alternative to that solution could be to just scrap APPLY_CURRENT_OBJECT_VARIATION and make immediate changes for each OV_ADD and OV_REMOVE, and to come up with some other way to deal with OV_CONVERT.

It would also probably make sense (assuming you haven't already implicitly included it), to have an implicit "GO_TO_END" at the start of every new EDIT object.

Do you also suppose separate object variations should be applied line-by-line then, when called by APPLY_OBJECT VARIATION? I am charmed by this idea, as the current way they are applied (REMOVEs bottom-up, then CONVERTs bottom-up, finally ADDSs top-down) is very strange and unintuitive. At the same time, them being so strange makes me think there is some secret behind it which I have failed to grasp. Also the old creature variations would need to have their CONVERTs reordered, this is a small issue if the saves are to break anyways, but otherwise a save-breaking (big) issue.

I forgot about the GO_TO_END, you’re right about that.

Quote
This is why the EDIT-level tokens can only make changes to special tokens, and why the edit objects add OV_ADD_TAGs instead of directly adding tokens.
During step 1 (i.e. when edits are being made), all edits done to normal tokens remain as unprocessed ov tokens in the in-memory objects, and can thus be reverted.

As the GO_TO tokens are special-only and have no EDIT-level counterparts, edits can't place ov tokens anywhere but the end of the selected object(s),
and so they will be processed in step 2 in the same order as the edits were parsed, top-down in the list of selected mods.

But if edits can't place GO_TO tokens don't have an EDIT counterpart, how do we handle cases where we need to add a special tag near the start? That is, how does the likes of ADD_SPEC_TOKEN actually place them where desired, in cases where that's relevant (especially if body detail plans were converted to object variations)? Because it seems many tokens are sensitive to token ordering.

This also brings up another question for me; can OBJECT_VARIATIONs be EDITed? (I assume yes, but just in case)

I’m trying to think of cases where you need to add special tokens to certain places. If body detail plans are converted, then they would consist of OV_ADD_TAGs, and these could be directed using the normal GO_TO tokens to wherever they are needed. I don’t think there are any uses for moving the GO_TO tokens themselves from the end. COPY_FROM_OBJECT should also be inserting the tokens according to the GO_TOs, so should APPLY_OBJECT_VARIATION and APPLY_CURRENT_OBJECT_VARIATION (if left/included). And for REMOVE_OBJECT the placement shouldn’t matter.

Object variations can be EDITed, yes.

Quote
Code: [Select]
[CREATURE:EDIT:BY_TOKEN:TOAD]
[GO_TO_END]
[OV_REMOVE_TAG:PET_VALUE]
and mod B:
Code: [Select]
[CREATURE:EDIT:BY_TOKEN:TOAD]
[GO_TO_START]
[PET_VALUE:50]
loaded in that order. After step 1, we would have:
Code: [Select]
[CREATURE:TOAD]
...
[CREATURE_TILE:249]
[COLOR:2:0:0]
[PET_VALUE:10]
...
[GO_TO_END]
[OV_REMOVE_TAG:PET_VALUE]
[APPLY_CURRENT_OBJECT_VARIATION]
[GO_TO_START]
[OV_ADD_TAG:PET_VALUE:50]
[APPLY_CURRENT_OBJECT_VARIATION]
. In step 2, this would be processed line-by-line, and as we can see REMOVE_TOKEN would be processed/applied first. As REMOVE_TOKEN does not add a token,
it is unaffected by the change of insertion index made by GO_TO_END. GO_TO_START still affects where PET_VALUE is inserted. Ultimately, we would get the desired result:
Code: [Select]
[CREATURE:TOAD]
[PET_VALUE:50]
...
[CREATURE_TILE:249]
[COLOR:2:0:0]
...

So this all works out nicely in this example, but how does this interact with the likes of REMOVE_SPEC_TOKEN?

I’m not sure what you’re getting at, to be honest :/. Do you mean removing GO_TO tokens with REMOVE_SPEC_TOKEN (that’s probably a bad idea)? Or do you just want an example of it being used?

Quote
Other than all that, this looks okay, though I'd have a suggestion to change the syntax of tag conversion, which to me seems unnecessarily long; to make one single change, you need 4 different tokens, when with different syntax at most you'd need 3, and could go as low as 1 + 2 for every replacement you wish to make to the same "master" token.
Current:
Code: [Select]
[CV_CONVERT_TAG]
[CVCT_MASTER:BODY]
[CVCT_TARGET:QUADRUPED]
[CVCT_REPLACEMENT:HUMANOID]
[CV_CONVERT_TAG]
[CVCT_MASTER:BODY]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
[CVCT_REPLACEMENT:HUMANOID]
1st idea:
Code: [Select]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED]
[CVCT_REPLACEMENT:HUMANOID]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
[CVCT_REPLACEMENT:HUMANOID]
2nd idea (maybe a little semantic/context heavy though):
Code: [Select]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED]
[CVCT_REPLACEMENT:HUMANOID]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
[CVCT_REPLACEMENT:HUMANOID]
It's a small difference, but it can add up to a big difference when making a lot of conversions, and the 2nd one makes a big difference when handling many conversions to the same "master" token.

Both these syntaxes are nicer than what we have now :). I gravitate towards the first one, but that preference is slight and might change when there’s more of them stacked.
Again, changes that would deprecate current vanilla raw files I think should only be done if the Steam release will break change compability even without them.

Quote
Also, if this is for a suggestion and it wouldn't be too bloated, remember the way of selecting what to be converted by token and argument number instead of value that I mentioned before: http://www.bay12forums.com/smf/index.php?topic=178868.msg8306433#msg8306433

I don’t think I’ll be writing the suggestion too soon, as Toady seems to still be a way from starting that work, but sure, I’ll include it. If the list of suggestions gets too bloated Toady will get to choose what to implement.
I’m assuming you mean something like this:
Code: [Select]
[CREATURE:RABBIT]
    ...
    [TL_COLOR_MODIFIER:BLACK:1:BROWN:1:WHITE:1:GRAY:1:LIGHT_BROWN:1:DARK_BROWN:1:TAN:1:AUBURN:1:CHESTNUT:1:SLATE_GRAY:1:CREAM:1:CINNAMON:1:BUFF:1:BEIGE:1:CHOCOLATE:1:CHARCOAL:1:ASH_GRAY:1:RUSSET:1:IVORY:1:FLAX:1:PUMPKIN:1:GOLD:1:GOLDEN_YELLOW:1:GOLDENROD:1:COPPER:1:SAFFRON:1:AMBER:1:MAHOGANY:1:OCHRE:1:PALE_BROWN:1:RAW_UMBER:1:BURNT_SIENNA:1:BURNT_UMBER:1:SEPIA:1:DARK_TAN:1:PALE_CHESTNUT:1:DARK_CHESTNUT:1:TAUPE_PALE:1:TAUPE_DARK:1:TAUPE_SANDY:1:TAUPE_GRAY:1:TAUPE_MEDIUM:1:ECRU:1]
    ...
    [OV_CONVERT_TAG:TL_COLOR_MODIFIER]
        [OVCT_TARGET_ARGNUM:0]
        [OVCT_REPLACEMENT:GOLD]
    [OV_CONVERT_TAG:TL_COLOR_MODIFIER]
        [OVCT_TARGET_ARGNUM:1]
        [OVCT_REPLACEMENT:1000]
=>
Code: [Select]
[CREATURE:RABBIT]
    ...
    [TL_COLOR_MODIFIER:GOLD:1000:BROWN:1:WHITE:1:GRAY:1:LIGHT_BROWN:1:DARK_BROWN:1:TAN:1:AUBURN:1:CHESTNUT:1:SLATE_GRAY:1:CREAM:1:CINNAMON:1:BUFF:1:BEIGE:1:CHOCOLATE:1:CHARCOAL:1:ASH_GRAY:1:RUSSET:1:IVORY:1:FLAX:1:PUMPKIN:1:GOLD:1:GOLDEN_YELLOW:1:GOLDENROD:1:COPPER:1:SAFFRON:1:AMBER:1:MAHOGANY:1:OCHRE:1:PALE_BROWN:1:RAW_UMBER:1:BURNT_SIENNA:1:BURNT_UMBER:1:SEPIA:1:DARK_TAN:1:PALE_CHESTNUT:1:DARK_CHESTNUT:1:TAUPE_PALE:1:TAUPE_DARK:1:TAUPE_SANDY:1:TAUPE_GRAY:1:TAUPE_MEDIUM:1:ECRU:1]
    ...
   

Mr Crabman

  • Bay Watcher
  • A person with the head and pincers of a crab.
    • View Profile
Re: DF Modloader mockup (and discussion on how to load mods in Steam DF)
« Reply #21 on: August 31, 2021, 01:16:27 pm »

I believe a compromise like e.g. [OBJECT_VARIATION:CREATURE:ANIMAL_PERSON] was suggested earlier in this thread, it should do the trick, right?

Yep, that would work fine.

The more you know. I could have sworn the resource packs too were loaded top-down, but the MC wiki backs you up... I wonder why they do that? My intuition (based on programming, I reckon) goes against that, and I also have the argument that the raws themselves have some kind of top-down linearity. At the same time, I have limited knowledge of ”mod loaders”, with the only example I know of being top-down are the custom scripts in RPG Maker VXA, and the interface for that isn’t strictly a ”mod loader” either. (Others I haven’t explored, or am pretty sure they are unordered (for the end-user)) If Minecraft does it, maybe going bottom-up is not as unusual as I assume?

I'd guess the reason is some intuitions about the bottom being the "base level", and how the higher ones may not necessarily replace all textures; in other words, a given resource pack may not be complete/may just be some small tweaks, and it rests on vanilla (and other resource packs below) as a "foundation", which fills in any gaps. Also, "top" pack is "top" priority, overriding what comes below.

Some other modloaders seem to go top to bottom though (https://wiki.nexusmods.com/index.php/How_to_handle_load_order), so I don't even know anymore; leaving aside intuitions, I wonder which is better UX.

Huh. That post also contains some other points of interest. One is CVCT_MASTER maybe working differently than we thought, being able to look for multiple tokens (or likely it’s just a string-search). I’ll do some science on that, as I’ll have to update my tools if that’s the case.

Also in that post there is a mention of nesting creature variations (called ”templates”). I don’t know what to think about it, or even the extent that some nesting can be done already, both in current DF and in the ”current” mod loader model. I think Toady is more concerned in the post about how they can be used as templates to reduce copy-pasting (and possibly some procgen myth-stuff?), while we are more concerned to making edits to ”already defined”.

Hm, didn't see any mention of CVCT_MASTER (maybe you're referring to "The swap can be restricted to a leading tag list (say, BODY:HUMANOID)"), but I hope to see the results of the science.

As for templates, in a way nesting can probably be done already partially, if we consider BODY_DETAIL_PLAN to be a primordial form of CREATURE_VARIATION, you can add those in with creature variations I'm pretty sure.

Yeah, our focus is a bit different, but reducing copy-pasting is very useful as well (after all, this is the very purpose of body detail plans and creature variations isn't it?).

More importantly (and this I realized first after reading your comment), if the old tokens are deprecated this will break all old saves, because they will still be using the deprecated tokens. If the Steam release breaks saves either way (probably due to changes in how graphics are stored) then this is fine, but less so if it doesn’t. The Steam update won’t add much in the way of new features either, so the incentives for genning a new world will be low. People will want to keep their old saves and some of them will be people who don’t play with mods. Mods are optional and so the framework for them changing should not be forced on the players by breaking saves.

Ah, that's a good point, but I'd imagine such a change would be possible to write a pretty simple update save converter for; I'm sure I could easily enough write a python script to make this change to already existing save files raws (it's just a find and replace for 2-4 things using regexes).

Do you also suppose separate object variations should be applied line-by-line then, when called by APPLY_OBJECT VARIATION? I am charmed by this idea, as the current way they are applied (REMOVEs bottom-up, then CONVERTs bottom-up, finally ADDSs top-down) is very strange and unintuitive. At the same time, them being so strange makes me think there is some secret behind it which I have failed to grasp. Also the old creature variations would need to have their CONVERTs reordered, this is a small issue if the saves are to break anyways, but otherwise a save-breaking (big) issue.

I believe so; I think the reason it's ordered like this though is because it seemed more natural to Toady to do things like QUADRUPED at the top of the conversion chain, but because of the weird "find and replace text instead of token arguments" behavior that has to actually be processed last. A solution might be to change tag conversions to work over whole arguments, and not do things like convert QUADRUPED_NECK to HUMANOID_NECK if you put them in the wrong order.

The only situation where the current behavior makes sense is for replacing parts of arbitrary strings, as done in vanilla for STATE_NAME, STATE_ADJ, SYN_IMMUNE_CREATURE, and SYN_NAME. It occurs to me now that the example I gave before for why it's necessary to be able to specify the argument number to be converted was wrong, because the example I gave before (for gila monster men I think) wasn't replacing the whole argument, only a specific part of it (which is why the exact text must be specified).

But for replacing whole arguments, like is done with BODY tokens, this is a terrible implementation. If that were changed/fixed, there would be no breakage of save files because the loading order of CV_CONVERT_TAG tokens wouldn't matter (though it would be necessary to do a quick save file conversion for to change the SYN_NAME and others I mentioned above).

Also, what's this about REMOVE being bottom-up? What implications does that have currently?

I’m trying to think of cases where you need to add special tokens to certain places. If body detail plans are converted, then they would consist of OV_ADD_TAGs, and these could be directed using the normal GO_TO tokens to wherever they are needed. I don’t think there are any uses for moving the GO_TO tokens themselves from the end. COPY_FROM_OBJECT should also be inserting the tokens according to the GO_TOs, so should APPLY_OBJECT_VARIATION and APPLY_CURRENT_OBJECT_VARIATION (if left/included). And for REMOVE_OBJECT the placement shouldn’t matter.

Could you give some examples of this? The way that the ADD_SPEC_TOKEN, REMOVE_SPEC_TOKEN, CONVERT_SPEC_TOKEN all work, and the idea of modifying the special tokens directly in step 1 as opposed to just adding on special tokens at the end that get processed 1 by 1, feels really hazy and dizzying to me in how it would work out between mods.

I’m not sure what you’re getting at, to be honest :/. Do you mean removing GO_TO tokens with REMOVE_SPEC_TOKEN (that’s probably a bad idea)? Or do you just want an example of it being used?

An example I think (I was thinking more along the lines of using [REMOVE_SPEC_TOKEN:OV_REMOVE_TAG:PET_VALUE] though to be honest).

Both these syntaxes are nicer than what we have now :). I gravitate towards the first one, but that preference is slight and might change when there’s more of them stacked.
Again, changes that would deprecate current vanilla raw files I think should only be done if the Steam release will break change compatibility even without them.

Well, looking at the vanilla files:
Code: [Select]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED]
    [CVCT_REPLACEMENT:HUMANOID]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
    [CVCT_REPLACEMENT:HUMANOID]
[CVCT_TARGET:QUADRUPED_NECK]
    [CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS]
[CVCT_TARGET:QUADRUPED_NECK_FRONT_GRASP]
    [CVCT_REPLACEMENT:HUMANOID_NECK]
[CVCT_TARGET:QUADRUPED_HOOF]
    [CVCT_REPLACEMENT:HUMANOID_HOOF:3FINGERS]
[CVCT_TARGET:QUADRUPED_NECK_HOOF]
    [CVCT_REPLACEMENT:HUMANOID_NECK_HOOF:3FINGERS]
[CVCT_TARGET:HUMANOID_ARMLESS]
    [CVCT_REPLACEMENT:HUMANOID:4FINGERS]
[CVCT_TARGET:HUMANOID_ARMLESS_NECK]
    [CVCT_REPLACEMENT:HUMANOID_NECK:4FINGERS]
[CVCT_TARGET:BODY_WITH_HEAD_FLAG]
    [CVCT_REPLACEMENT:HUMANOID:3FINGERS]
[CVCT_TARGET:BASIC_1PARTBODY:BASIC_HEAD]
    [CVCT_REPLACEMENT:HUMANOID:3FINGERS]
[CVCT_TARGET:BASIC_2PARTBODY:BASIC_HEAD]
    [CVCT_REPLACEMENT:HUMANOID:3FINGERS]
[CVCT_TARGET:BASIC_1PARTBODY:BASIC_HEAD_NECK]
    [CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS]
[CVCT_TARGET:BASIC_2PARTBODY:BASIC_HEAD_NECK]
    [CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS]
[CVCT_TARGET:INSECT]
    [CVCT_REPLACEMENT:HUMANOID_4ARMS:3FINGERS]
[CVCT_TARGET:INSECT_4LEGS_2ARMS]
    [CVCT_REPLACEMENT:HUMANOID:3FINGERS]
[CVCT_TARGET:SPIDER]
    [CVCT_REPLACEMENT:HUMANOID_6ARMS:3FINGERS]
[CVCT_TARGET:5TOES_FQ_FINGERS]
    [CVCT_REPLACEMENT:5FINGERS]
[CVCT_TARGET:4TOES_FQ_FINGERS]
    [CVCT_REPLACEMENT:4FINGERS]
[CVCT_TARGET:3TOES_FQ_FINGERS]
    [CVCT_REPLACEMENT:3FINGERS]
[CVCT_TARGET:5TOES_FQ_REG]
    [CVCT_REPLACEMENT:5FINGERS]
[CVCT_TARGET:4TOES_FQ_REG]
    [CVCT_REPLACEMENT:4FINGERS]
[CVCT_TARGET:3TOES_FQ_REG]
    [CVCT_REPLACEMENT:3FINGERS]
[CVCT_TARGET:5TOES_RQ_ANON]
    [CVCT_REPLACEMENT:5TOES]
[CVCT_TARGET:4TOES_RQ_ANON]
    [CVCT_REPLACEMENT:4TOES]
[CVCT_TARGET:3TOES_RQ_ANON]
    [CVCT_REPLACEMENT:3TOES]
[CVCT_TARGET:5TOES_RQ_REG]
    [CVCT_REPLACEMENT:5TOES]
[CVCT_TARGET:4TOES_RQ_REG]
    [CVCT_REPLACEMENT:4TOES]
[CVCT_TARGET:3TOES_RQ_REG]
    [CVCT_REPLACEMENT:3TOES]
Code: [Select]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED]
[CVCT_REPLACEMENT:HUMANOID]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
[CVCT_REPLACEMENT:HUMANOID]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED_NECK]
[CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED_NECK_FRONT_GRASP]
[CVCT_REPLACEMENT:HUMANOID_NECK]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED_HOOF]
[CVCT_REPLACEMENT:HUMANOID_HOOF:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED_NECK_HOOF]
[CVCT_REPLACEMENT:HUMANOID_NECK_HOOF:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:HUMANOID_ARMLESS]
[CVCT_REPLACEMENT:HUMANOID:4FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:HUMANOID_ARMLESS_NECK]
[CVCT_REPLACEMENT:HUMANOID_NECK:4FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:BODY_WITH_HEAD_FLAG]
[CVCT_REPLACEMENT:HUMANOID:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:BASIC_1PARTBODY:BASIC_HEAD]
[CVCT_REPLACEMENT:HUMANOID:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:BASIC_2PARTBODY:BASIC_HEAD]
[CVCT_REPLACEMENT:HUMANOID:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:BASIC_1PARTBODY:BASIC_HEAD_NECK]
[CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:BASIC_2PARTBODY:BASIC_HEAD_NECK]
[CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:INSECT]
[CVCT_REPLACEMENT:HUMANOID_4ARMS:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:INSECT_4LEGS_2ARMS]
[CVCT_REPLACEMENT:HUMANOID:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:SPIDER]
[CVCT_REPLACEMENT:HUMANOID_6ARMS:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:5TOES_FQ_FINGERS]
[CVCT_REPLACEMENT:5FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:4TOES_FQ_FINGERS]
[CVCT_REPLACEMENT:4FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:3TOES_FQ_FINGERS]
[CVCT_REPLACEMENT:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:5TOES_FQ_REG]
[CVCT_REPLACEMENT:5FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:4TOES_FQ_REG]
[CVCT_REPLACEMENT:4FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:3TOES_FQ_REG]
[CVCT_REPLACEMENT:3FINGERS]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:5TOES_RQ_ANON]
[CVCT_REPLACEMENT:5TOES]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:4TOES_RQ_ANON]
[CVCT_REPLACEMENT:4TOES]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:3TOES_RQ_ANON]
[CVCT_REPLACEMENT:3TOES]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:5TOES_RQ_REG]
[CVCT_REPLACEMENT:5TOES]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:4TOES_RQ_REG]
[CVCT_REPLACEMENT:4TOES]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:3TOES_RQ_REG]
[CVCT_REPLACEMENT:3TOES]

I know which one I think looks better, and more compact.... Again, save update conversion script (built into the game itself naturally) could solve this, it only takes a couple of changes.

I’m assuming you mean something like this:

Precisely, although as noted above, I'm less sure of the utility of it now; it was primarily for replacing whole strings or numeric arguments since any given mod could make significant or small changes to them and yet you'd want to change it, but now that I've realized that the example I gave was actually only replacing part of the string, I'm less sure now of how much point it has (like, in your example there, would anyone really have a feeling of that first and second arguments being important to change regardless of what other mods have done to it?).
« Last Edit: August 31, 2021, 01:44:48 pm by Mr_Crabman »
Logged

Mr Crabman

  • Bay Watcher
  • A person with the head and pincers of a crab.
    • View Profile
Re: DF Modloader mockup (and discussion on how to load mods in Steam DF)
« Reply #22 on: August 31, 2021, 01:38:41 pm »

Ah, one more thing to throw on the suggestion pile that isn't super important; given that object variations would be replacing some stuff, it would make sense to have a different name to "OBJECT_VARIATION", given that the only reason creature variations appear to be called that is because of how they're used to create variants of creatures mostly/originally, when in reality their function is much broader (containing chunks of raw code to reduce duplication/repetition).

voliol

  • Bay Watcher
    • View Profile
    • Website
Re: DF Modloader mockup (and discussion on how to load mods in Steam DF)
« Reply #23 on: August 31, 2021, 01:58:35 pm »

Ah, one more thing to throw on the suggestion pile that isn't super important; given that object variations would be replacing some stuff, it would make sense to have a different name to "OBJECT_VARIATION", given that the only reason creature variations appear to be called that is because of how they're used to create variants of creatures mostly/originally, when in reality their function is much broader (containing chunks of raw code to reduce duplication/repetition).

Good point. What about OBJECT_TEMPLATE, as that is what they are (and creature variations were described as templates in that post by Toady)? In a way just TEMPLATE would suffice, but it feels too short.

Here's a quick update on the science: Toady's remark was about CVCT_MASTER, or at least it holds true for it. CVCT_MASTER can target arguments in addition to the token name, and it does so by checking the first arguments. So [BODY:HUMANOID_SIMPLE:3_EYES] would be targeted by CVCT_MASTER:HUMANOID_SIMPLE, but not by CVCT_MASTER:3_EYES. However, it does check for whole arguments not for a string as I had initially thought, so CVCT_MASTER:HUMANOID would not target the former.
Edit: upon further thought, this is the same as how CV_REMOVE_TOKEN works. Another argument for [CV_CONVERT_TOKEN:ARGUMENTS] being preferable over using CVCT_MASTER I guess.

I'll get back on the rest tomorrow.

Mr Crabman

  • Bay Watcher
  • A person with the head and pincers of a crab.
    • View Profile
Re: DF Modloader mockup (and discussion on how to load mods in Steam DF)
« Reply #24 on: September 01, 2021, 08:21:46 am »

Good point. What about OBJECT_TEMPLATE, as that is what they are (and creature variations were described as templates in that post by Toady)? In a way just TEMPLATE would suffice, but it feels too short.

That sounds okay.

That aside, it might be a good idea to just drop the "OT_" from OT_REMOVE_TAG and OT_CONVERT_TAG, and just do REMOVE_TAG, CONVERT_TAG, and ADD_TAG (or ideally abolish ADD_TAG altogether as a thing that modders/end users actually see); if it's all the same system, there's no need to specify what "kind" of tag is being added; the only factor that gives me any hesitation here is the idea of an autocomplete that lets you see and select from just those 3 tags from those very unique first 2 letters (but then again, the other special GO_TO tokens don't match up with that anyway).

So [BODY:HUMANOID_SIMPLE:3_EYES] would be targeted by CVCT_MASTER:HUMANOID_SIMPLE, but not by CVCT_MASTER:3_EYES.

I assume you mean [CVCT_MASTER:BODY:HUMANOID_SIMPLE]?

voliol

  • Bay Watcher
    • View Profile
    • Website
Re: DF Modloader mockup (and discussion on how to load mods in Steam DF)
« Reply #25 on: September 01, 2021, 03:15:23 pm »

More importantly (and this I realized first after reading your comment), if the old tokens are deprecated this will break all old saves, because they will still be using the deprecated tokens. If the Steam release breaks saves either way (probably due to changes in how graphics are stored) then this is fine, but less so if it doesn’t. The Steam update won’t add much in the way of new features either, so the incentives for genning a new world will be low. People will want to keep their old saves and some of them will be people who don’t play with mods. Mods are optional and so the framework for them changing should not be forced on the players by breaking saves.

Ah, that's a good point, but I'd imagine such a change would be possible to write a pretty simple update save converter for; I'm sure I could easily enough write a python script to make this change to already existing save files raws (it's just a find and replace for 2-4 things using regexes).

Hmm, you're right. I wonder if that's in the plans already, for converting the graphics in the save folders to whatever new system will be used... but that's a side track, please excuse me.

Quote
Do you also suppose separate object variations should be applied line-by-line then, when called by APPLY_OBJECT VARIATION? I am charmed by this idea, as the current way they are applied (REMOVEs bottom-up, then CONVERTs bottom-up, finally ADDSs top-down) is very strange and unintuitive. At the same time, them being so strange makes me think there is some secret behind it which I have failed to grasp. Also the old creature variations would need to have their CONVERTs reordered, this is a small issue if the saves are to break anyways, but otherwise a save-breaking (big) issue.

I believe so; I think the reason it's ordered like this though is because it seemed more natural to Toady to do things like QUADRUPED at the top of the conversion chain, but because of the weird "find and replace text instead of token arguments" behavior that has to actually be processed last. A solution might be to change tag conversions to work over whole arguments, and not do things like convert QUADRUPED_NECK to HUMANOID_NECK if you put them in the wrong order.

The only situation where the current behavior makes sense is for replacing parts of arbitrary strings, as done in vanilla for STATE_NAME, STATE_ADJ, SYN_IMMUNE_CREATURE, and SYN_NAME. It occurs to me now that the example I gave before for why it's necessary to be able to specify the argument number to be converted was wrong, because the example I gave before (for gila monster men I think) wasn't replacing the whole argument, only a specific part of it (which is why the exact text must be specified).

But for replacing whole arguments, like is done with BODY tokens, this is a terrible implementation. If that were changed/fixed, there would be no breakage of save files because the loading order of CV_CONVERT_TAG tokens wouldn't matter (though it would be necessary to do a quick save file conversion for to change the SYN_NAME and others I mentioned above).

Also, what's this about REMOVE being bottom-up? What implications does that have currently?

As both the arbitrary string searches and the full token argument searches are useful when it comes to CV_CONVERT_TAG (or really CVCT_TARGET), why not split the cases into CVCT_TARGET_STRING for the gila monsters and... probably CVCT_TARGET still for e.g. the BODY conversions, as I think that is the obvious way for it to work.

As for REMOVE being bottom-up, heh, that's something the wiki said. I'm guessing someone figured it out using memory hacking, but now that you mention it is quite silly. It should be inconsequential.

Quote
I’m trying to think of cases where you need to add special tokens to certain places. If body detail plans are converted, then they would consist of OV_ADD_TAGs, and these could be directed using the normal GO_TO tokens to wherever they are needed. I don’t think there are any uses for moving the GO_TO tokens themselves from the end. COPY_FROM_OBJECT should also be inserting the tokens according to the GO_TOs, so should APPLY_OBJECT_VARIATION and APPLY_CURRENT_OBJECT_VARIATION (if left/included). And for REMOVE_OBJECT the placement shouldn’t matter.

Could you give some examples of this? The way that the ADD_SPEC_TOKEN, REMOVE_SPEC_TOKEN, CONVERT_SPEC_TOKEN all work, and the idea of modifying the special tokens directly in step 1 as opposed to just adding on special tokens at the end that get processed 1 by 1, feels really hazy and dizzying to me in how it would work out between mods.
Quote
An example I think (I was thinking more along the lines of using [REMOVE_SPEC_TOKEN:OV_REMOVE_TAG:PET_VALUE] though to be honest).

Understood. The original TOAD raws will still be the same as before, and so would mod A;
Code: [Select]
[CREATURE:EDIT:BY_TOKEN:TOAD]
[OV_REMOVE_TAG:PET_VALUE]
but instead of mod B adding a new PET_VALUE, it uses REMOVE_SPEC_TOKEN:
Code: [Select]
[CREATURE:EDIT:BY_TOKEN:TOAD]
[REMOVE_SPEC_TOKEN:OV_REMOVE_TAG:PET_VALUE]
During step 1, after the A EDIT has been applied, we have
Code: [Select]
[CREATURE:TOAD]
...
[PET_VALUE:10]
...
[OV_REMOVE_TAG:PET_VALUE]
([APPLY_CURRENT_OBJECT_VARIATION]) <-- only if they are still used
        [GO_TO_END]
Still during step 1, when B is being parsed, REMOVE_SPEC_TOKEN removes the OV_REMOVE_TAG:PET_VALUE, giving us
Code: [Select]
[CREATURE:TOAD]
...
[PET_VALUE:10]
...
([APPLY_CURRENT_OBJECT_VARIATION])
        [GO_TO_END]
In step 2, this would be processed line-by-line, and we would just get the normal TOAD raws.

ADD_SPEC_TOKEN is even more redundant than OV_ADD_TAG (or whatever you call it), as its functionality is supposedly built into the EDIT object handling, you just have to write the special/ov token out and it assumes you're adding it. So ADD_SPEC_TOKEN could probably get scrapped.

CONVERT_SPEC_TOKEN works similar to REMOVE_SPEC_TOKEN... and I know realize they are broken and do not solve the initial problem of mod C changing APPLY_OBJECT_VARIATION:ANIMAL_PERSON to APPLY_OBJECT_VARIATION:AP_C, blocking mod D from changing it into APPLT_OBJECT_VARIATION:AP_D. And likewise, if you remove APPLY_OBJECT_VARIATION:STANDARD_SWIMMING_GAITS using REMOVE_SPEC_TOKEN, then there's no getting it back. Sigh.

Very wild and untested idea, what if REMOVE_SPEC_TOKEN and CONVERT_SPEC_TOKEN are not processed in step 1, but added to the start of the in-memory objects, in reverse order. And also they can target each other i.e. mod E:
Code: [Select]
[EDIT:CREATURE:BY_ID:TOAD]
[REMOVE_SPEC_TOKEN:APPLY_OBJECT_VARIATION:STANDARD_SWIMMING_GAITS]
and D:
Code: [Select]
[EDIT:CREATURE:BY_ID:TOAD]
[REMOVE_SPEC_TOKEN:REMOVE_SPEC_TOKEN:APPLY_OBJECT_VARIATION:STANDARD_SWIMMING_GAITS]
(loaded in that order) gives these TOAD raws at the start of step 2:
Code: [Select]
[CREATURE:TOAD]
[REMOVE_SPEC_TOKEN:REMOVE_SPEC_TOKEN:APPLY_OBJECT_VARIATION:STANDARD_SWIMMING_GAITS]
[REMOVE_SPEC_TOKEN:APPLY_OBJECT_VARIATION:STANDARD_SWIMMING_GAITS]
...
[APPLY_OBJECT_VARIATION:STANDARD_SWIMMING_GAITS:...]
...
Granted, this might not solve the aforementioned CONVERT_SPEC_TOKEN problem, and I also think these very long tokens look ridiculous.

Quote
Well, looking at the vanilla files:
Code: [Select]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED]
    [CVCT_REPLACEMENT:HUMANOID]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
    [CVCT_REPLACEMENT:HUMANOID]
etc.
Code: [Select]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED]
[CVCT_REPLACEMENT:HUMANOID]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
[CVCT_REPLACEMENT:HUMANOID]
etc.

I know which one I think looks better, and more compact.... Again, save update conversion script (built into the game itself naturally) could solve this, it only takes a couple of changes.

The tabs really make the first one, I was thinking of it like this:
Code: [Select]
[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED]
[CVCT_REPLACEMENT:HUMANOID]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
[CVCT_REPLACEMENT:HUMANOID]
[CVCT_TARGET:QUADRUPED_NECK]
[CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS]
[CVCT_TARGET:QUADRUPED_NECK_FRONT_GRASP]
[CVCT_REPLACEMENT:HUMANOID_NECK]
[CVCT_TARGET:QUADRUPED_HOOF]
[CVCT_REPLACEMENT:HUMANOID_HOOF:3FINGERS]
[CVCT_TARGET:QUADRUPED_NECK_HOOF]
[CVCT_REPLACEMENT:HUMANOID_NECK_HOOF:3FINGERS]
[CVCT_TARGET:HUMANOID_ARMLESS]
[CVCT_REPLACEMENT:HUMANOID:4FINGERS]
[CVCT_TARGET:HUMANOID_ARMLESS_NECK]
[CVCT_REPLACEMENT:HUMANOID_NECK:4FINGERS]
[CVCT_TARGET:BODY_WITH_HEAD_FLAG]
[CVCT_REPLACEMENT:HUMANOID:3FINGERS]
[CVCT_TARGET:BASIC_1PARTBODY:BASIC_HEAD]
[CVCT_REPLACEMENT:HUMANOID:3FINGERS]
[CVCT_TARGET:BASIC_2PARTBODY:BASIC_HEAD]
[CVCT_REPLACEMENT:HUMANOID:3FINGERS]
[CVCT_TARGET:BASIC_1PARTBODY:BASIC_HEAD_NECK]
[CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS]
[CVCT_TARGET:BASIC_2PARTBODY:BASIC_HEAD_NECK]
[CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS]
[CVCT_TARGET:INSECT]
[CVCT_REPLACEMENT:HUMANOID_4ARMS:3FINGERS]
[CVCT_TARGET:INSECT_4LEGS_2ARMS]
[CVCT_REPLACEMENT:HUMANOID:3FINGERS]
[CVCT_TARGET:SPIDER]
[CVCT_REPLACEMENT:HUMANOID_6ARMS:3FINGERS]
[CVCT_TARGET:5TOES_FQ_FINGERS]
[CVCT_REPLACEMENT:5FINGERS]
[CVCT_TARGET:4TOES_FQ_FINGERS]
[CVCT_REPLACEMENT:4FINGERS]
[CVCT_TARGET:3TOES_FQ_FINGERS]
[CVCT_REPLACEMENT:3FINGERS]
[CVCT_TARGET:5TOES_FQ_REG]
[CVCT_REPLACEMENT:5FINGERS]
[CVCT_TARGET:4TOES_FQ_REG]
[CVCT_REPLACEMENT:4FINGERS]
[CVCT_TARGET:3TOES_FQ_REG]
[CVCT_REPLACEMENT:3FINGERS]
[CVCT_TARGET:5TOES_RQ_ANON]
[CVCT_REPLACEMENT:5TOES]
[CVCT_TARGET:4TOES_RQ_ANON]
[CVCT_REPLACEMENT:4TOES]
[CVCT_TARGET:3TOES_RQ_ANON]
[CVCT_REPLACEMENT:3TOES]
[CVCT_TARGET:5TOES_RQ_REG]
[CVCT_REPLACEMENT:5TOES]
[CVCT_TARGET:4TOES_RQ_REG]
[CVCT_REPLACEMENT:4TOES]
[CVCT_TARGET:3TOES_RQ_REG]
[CVCT_REPLACEMENT:3TOES]
Much less charming.

Good point. What about OBJECT_TEMPLATE, as that is what they are (and creature variations were described as templates in that post by Toady)? In a way just TEMPLATE would suffice, but it feels too short.
That sounds okay.

That aside, it might be a good idea to just drop the "OT_" from OT_REMOVE_TAG and OT_CONVERT_TAG, and just do REMOVE_TAG, CONVERT_TAG, and ADD_TAG (or ideally abolish ADD_TAG altogether as a thing that modders/end users actually see); if it's all the same system, there's no need to specify what "kind" of tag is being added; the only factor that gives me any hesitation here is the idea of an autocomplete that lets you see and select from just those 3 tags from those very unique first 2 letters (but then again, the other special GO_TO tokens don't match up with that anyway).

I'm a little averse to changing up (or suggesting changing up) the syntax too much. Or maybe just my gut feelings are, these arguments make sense.

Quote
So [BODY:HUMANOID_SIMPLE:3_EYES] would be targeted by CVCT_MASTER:HUMANOID_SIMPLE, but not by CVCT_MASTER:3_EYES.
I assume you mean [CVCT_MASTER:BODY:HUMANOID_SIMPLE]?
Yes. Woops.

Mr Crabman

  • Bay Watcher
  • A person with the head and pincers of a crab.
    • View Profile
Re: DF Modloader mockup (and discussion on how to load mods in Steam DF)
« Reply #26 on: September 01, 2021, 04:40:52 pm »

One other thing just occurred to me, it might be a good idea to have a better way to remove large continuous (but related) chunks of tokens; like this:

Code: [Select]
[ATTACK:BITE:BODYPART:BY_CATEGORY:MOUTH]
[ATTACK_SKILL:BITE]
[ATTACK_VERB:bite:bites]
[ATTACK_CONTACT_PERC:100]
[ATTACK_PREPARE_AND_RECOVER:3:3]
[ATTACK_PRIORITY:MAIN]
[ATTACK_FLAG_CANLATCH]

The nature of REMOVE_TAG makes this kind of awkward; there's no way to remove this block without repeating the whole block but with REMOVE_TAG at the start of each one; now, that's kind of annoying (since you can't just copy paste it), but functional; the OTHER problem is that every other attack with [ATTACK_FLAG_CANLATCH] will get affected as well (and same goes for most of the other tokens here).

While creature variations in vanilla don't really bother dealing with this (see how ANIMAL_PERSON just erases all attack stuff altogether and just manually adds in new things), for mods which need to be able to make precision edits, this kind of thing is vital. The mantra I keep repeating to myself when thinking about this is "make EDIT as close as possible to manually going into the raw txt and making the changes you want manually and only those changes, and listing them such that later mod authors can make their own changes".

As both the arbitrary string searches and the full token argument searches are useful when it comes to CV_CONVERT_TAG (or really CVCT_TARGET), why not split the cases into CVCT_TARGET_STRING for the gila monsters and... probably CVCT_TARGET still for e.g. the BODY conversions, as I think that is the obvious way for it to work.

Yes, this is precisely what I was thinking.

In step 2, this would be processed line-by-line, and we would just get the normal TOAD raws.

Hmm, that seems to work out I guess (if you put mod B first it wouldn't remove OV_REMOVE_TAG right?), but as discussed below, there are other issues.

CONVERT_SPEC_TOKEN works similar to REMOVE_SPEC_TOKEN... and I know realize they are broken and do not solve the initial problem of mod C changing APPLY_OBJECT_VARIATION:ANIMAL_PERSON to APPLY_OBJECT_VARIATION:AP_C, blocking mod D from changing it into APPLT_OBJECT_VARIATION:AP_D.

Wait, was that the initial problem? I was under the impression the initial problem was about how APPLY_OBJECT_VARIATION and other such tokens need to be placed in a specific position and you need to be able to change it without letting the command run, so APPLY_OBJECT_VARIATION:ANIMAL_PERSON needs to be changed to APPLY_OBJECT_VARIATION:AP_C before the raws actually process APPLY_OBJECT_VARIATION:ANIMAL_PERSON because that will lead to a whole bunch of cleanup being needed.

That, and also needing to position some other tokens, like when messing with body detail plans and tissues (both altering and removing vanilla ones, and inserting in new ones); you need to make the edits before actually processing the object and the commands therein.

And likewise, if you remove APPLY_OBJECT_VARIATION:STANDARD_SWIMMING_GAITS using REMOVE_SPEC_TOKEN, then there's no getting it back.

I don't think this part is such a problem? Adding it back again with mod C after mod B removes it is relatively trivial, although if mod A in the meantime had changed the values inserted to STANDARD_SWIMMING_GAITS then you can't get those specific ones back (you'd have to pick mod A's version of it, or vanilla, or some other mod).

But I'm not sure it needs to be possible to target removals made by 1 specific other mod and then undo them, at least in a situation like this; the only situation where it seems appropriate to do something like this is with REMOVE_OBJECT, because otherwise changes can just be too widespread and hyperspecific.

Very wild and untested idea, what if REMOVE_SPEC_TOKEN and CONVERT_SPEC_TOKEN are not processed in step 1, but added to the start of the in-memory objects, in reverse order. And also they can target each other i.e. mod E:
Code: [Select]
[EDIT:CREATURE:BY_ID:TOAD]
[REMOVE_SPEC_TOKEN:APPLY_OBJECT_VARIATION:STANDARD_SWIMMING_GAITS]
and D:
Code: [Select]
[EDIT:CREATURE:BY_ID:TOAD]
[REMOVE_SPEC_TOKEN:REMOVE_SPEC_TOKEN:APPLY_OBJECT_VARIATION:STANDARD_SWIMMING_GAITS]
(loaded in that order) gives these TOAD raws at the start of step 2:
Code: [Select]
[CREATURE:TOAD]
[REMOVE_SPEC_TOKEN:REMOVE_SPEC_TOKEN:APPLY_OBJECT_VARIATION:STANDARD_SWIMMING_GAITS]
[REMOVE_SPEC_TOKEN:APPLY_OBJECT_VARIATION:STANDARD_SWIMMING_GAITS]
...
[APPLY_OBJECT_VARIATION:STANDARD_SWIMMING_GAITS:...]
...
Granted, this might not solve the aforementioned CONVERT_SPEC_TOKEN problem, and I also think these very long tokens look ridiculous.

I'm not sure, apart from what you say about them being a bit excessive, as I say above, I also don't think it's a great idea for mods to directly target each other, or to be implanting more kinds of commands inside objects, further muddying the "pure data" with commands. Putting commands inside the creature data is what got us in this mess to begin with after all.

This is why I originally suggested that the mod edits should directly make their changes rather than just tacking various OV commands onto the end and letting them run through. The initial problem was that you need to change [APPLY_OBJECT_VARIATION:ANIMAL_PERSON] to [APPLY_OBJECT_VARIATION:AP_C] before [APPLY_OBJECT_VARIATION:ANIMAL_PERSON] runs and makes a million little changes that need to be manually undone, so the EDIT needs to be applied to the raw text before the object is properly parsed.

Just like real manual editing; one mod author makes their changes to the file (maybe changing ANIMAL_PERSON to AP_C in one place), then passes it on, and the next mod authors looks at the file and makes their changes on top of that edited file, etc etc. The difference is that the mod author is instead programming in what changes to be made rather than making the changes themselves,

The tabs really make the first one, I was thinking of it like this:

That's fair; I actually did realize that problem, which is why I did the tabs to make it more legible.
« Last Edit: September 01, 2021, 04:44:08 pm by Mr_Crabman »
Logged

voliol

  • Bay Watcher
    • View Profile
    • Website
Re: DF Modloader mockup
« Reply #27 on: September 02, 2021, 10:23:08 am »

One other thing just occurred to me, it might be a good idea to have a better way to remove large continuous (but related) chunks of tokens; like this:

Code: [Select]
[ATTACK:BITE:BODYPART:BY_CATEGORY:MOUTH]
[ATTACK_SKILL:BITE]
[ATTACK_VERB:bite:bites]
[ATTACK_CONTACT_PERC:100]
[ATTACK_PREPARE_AND_RECOVER:3:3]
[ATTACK_PRIORITY:MAIN]
[ATTACK_FLAG_CANLATCH]

The nature of REMOVE_TAG makes this kind of awkward; there's no way to remove this block without repeating the whole block but with REMOVE_TAG at the start of each one; now, that's kind of annoying (since you can't just copy paste it), but functional; the OTHER problem is that every other attack with [ATTACK_FLAG_CANLATCH] will get affected as well (and same goes for most of the other tokens here).

While creature variations in vanilla don't really bother dealing with this (see how ANIMAL_PERSON just erases all attack stuff altogether and just manually adds in new things), for mods which need to be able to make precision edits, this kind of thing is vital. The mantra I keep repeating to myself when thinking about this is "make EDIT as close as possible to manually going into the raw txt and making the changes you want manually and only those changes, and listing them such that later mod authors can make their own changes".

I agree.

You could have some variety of REMOVE_TAG (and CONVERT_TAG) which only targets the "next" occurance of a token, relative to the insertion point decided by the GO_TO tokens. The annoyance of having to write REMOVE_TOKEN for each token wouldn't be affected, but it naturally would be more precise than what we have currently. e.g.
Code: [Select]
[GO_TO_TAG:ATTACK:BITE:BODYPART:BY_CATEGORY:MOUTH]
[REMOVE_NTAG:ATTACK:BITE:BODYPART:BY_CATEGORY:MOUTH]
[REMOVE_NTAG:ATTACK_SKILL:BITE]
[REMOVE_NTAG:ATTACK_VERB:bite:bites]
[REMOVE_NTAG:ATTACK_CONTACT_PERC:100]
[REMOVE_NTAG:ATTACK_PREPARE_AND_RECOVER:3:3]
[REMOVE_NTAG:ATTACK_PRIORITY:MAIN]
[REMOVE_NTAG:ATTACK_FLAG_CANLATCH]
If you want to remove only a bite attack. "N" for "next. It does have a weakness though; that it doesn't look for a block. If one of these tokens are not included in the BITE attack block, say ATTACK_FLAG_CANLATCH, then it will remove ATTACK_FLAG_CANLATCH from the next attack using it instead. Probably enough reason to make it work somewhat differently, or scrapping the idea for another solution.

Quote
CONVERT_SPEC_TOKEN works similar to REMOVE_SPEC_TOKEN... and I know realize they are broken and do not solve the initial problem of mod C changing APPLY_OBJECT_VARIATION:ANIMAL_PERSON to APPLY_OBJECT_VARIATION:AP_C, blocking mod D from changing it into APPLT_OBJECT_VARIATION:AP_D.

Wait, was that the initial problem? I was under the impression the initial problem was about how APPLY_OBJECT_VARIATION and other such tokens need to be placed in a specific position and you need to be able to change it without letting the command run, so APPLY_OBJECT_VARIATION:ANIMAL_PERSON needs to be changed to APPLY_OBJECT_VARIATION:AP_C before the raws actually process APPLY_OBJECT_VARIATION:ANIMAL_PERSON because that will lead to a whole bunch of cleanup being needed.

That, and also needing to position some other tokens, like when messing with body detail plans and tissues (both altering and removing vanilla ones, and inserting in new ones); you need to make the edits before actually processing the object and the commands therein.

I was misconstruing my memories of this reply:
[...]
And for even normal edits, there's a similar potential problem with the likes of COPY_TAGS_FROM and APPLY_CREATURE_VARIATION and so on, because even if you do all creature variations (including in mods) before creatures to solve the previous problem, still when the creature itself in vanilla is processed, the (modded) ANIMAL_PERSON variation is applied, and your later [EDIT] can't change it to [APPLY_CREATURE_VARIATION:NEW_ANIMAL_PERSON] because it's loaded later, and to get the desired result would need a lot of messy cleanup from the aftermath of [APPLY_CREATURE_VARIATION:ANIMAL_PERSON].
[...]
But upon reading it again (and some others below further explaining it) they turn out to be a reiteration of this last reply of yours (or rather vice-versa). I still think it is an issue that mods earlier in "the order" can have some precedence over the later ones, because the principle should be that later mods overwrite the earlier ones, but perhaps these kinds of conflicts can't be avoided, especially when you get into fringe cases. No matter what the system, it likely breaks somewhere, so the goal should be making sure those points of brittleness are nowhere important.

I'm not sure about the insertion of ov (or ot) tokens into the non-EDIT objects. I wouldn't go so far as saying they are muddled, but it might just be inconsequential. I thought I were sure, but this task is more and more dizzying for me. I think I need to take it calm for a few days in regards to this, thinking it over and perhaps modeling some in executable code instead of just text files.

Mr Crabman

  • Bay Watcher
  • A person with the head and pincers of a crab.
    • View Profile
Re: DF Modloader mockup
« Reply #28 on: September 03, 2021, 12:17:10 pm »

I still think it is an issue that mods earlier in "the order" can have some precedence over the later ones, because the principle should be that later mods overwrite the earlier ones, but perhaps these kinds of conflicts can't be avoided, especially when you get into fringe cases. No matter what the system, it likely breaks somewhere, so the goal should be making sure those points of brittleness are nowhere important.

Well, ultimately, a mod is going to have to add or remove or change tokens (either directly, or indirectly as commands that get executed later), and it seems less error-prone to just force later mods manually undo the changes that were made, rather than let them try to directly edit the previous mod's commands that would result in the changes.

That is, it seems better that if mod B deleted PET_VALUE, but mod C wants to add PET_VALUE, they just add it instead of assuming that mod A or vanilla had it and trying to use REMOVE_SPEC_TOKEN:OT_REMOVE_TAG to undo mod B's change. Just undo the removal by adding it back.

Mr Crabman

  • Bay Watcher
  • A person with the head and pincers of a crab.
    • View Profile
Re: DF Modloader mockup (and discussion on how to load mods in Steam DF)
« Reply #29 on: November 22, 2021, 12:33:46 pm »

Have you had a chance to test some ideas and/or put together some of a suggestion yet?

I ask because the in latest BlindIRL interview (and also kind of the last FotF reply), Tarn seems to imply he'll be starting work on modding/save file changes soonish.
Pages: 1 [2] 3