Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1] 2

Author Topic: Add 'mod' folder and its handling. Add detailed log output for mod loading  (Read 2212 times)

Evans

  • Bay Watcher
    • View Profile

Rather than just copy-paste into game raws, there should be:

Mod folder

Any subfolder there should be treated as separate mod. Files from there could be read as separate entities.
mod.txt for mod loading order.
'Override', for modified vanilla files and entries.
'Remove', for deleted vanilla files and entries.

This would make life easier for everybody.

Also, logging what mods load what and in what order, also eventual crashes.
Logged
getlost.lua # How to get rid of tavern guests
function getlost ()
   local unit = dfhack.gui.getSelectedUnit (true)
   unit.flags1.forest = true
end
getlost ()

Shonai_Dweller

  • Bay Watcher
    • View Profile
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #1 on: November 14, 2020, 04:47:43 pm »

Rather than just copy-paste into game raws, there should be:

Mod folder

Any subfolder there should be treated as separate mod. Files from there could be read as separate entities.
mod.txt for mod loading order.
'Override', for modified vanilla files and entries.
'Remove', for deleted vanilla files and entries.

This would make life easier for everybody.

Also, logging what mods load what and in what order, also eventual crashes.
Everything is changing for mods to make it compatible with Steam Workshop. Current system won't work at all and pretty much what you're suggesting will be implemented.
Logged

Evans

  • Bay Watcher
    • View Profile
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #2 on: November 14, 2020, 08:11:16 pm »

Rather than just copy-paste into game raws, there should be:

Mod folder

Any subfolder there should be treated as separate mod. Files from there could be read as separate entities.
mod.txt for mod loading order.
'Override', for modified vanilla files and entries.
'Remove', for deleted vanilla files and entries.

This would make life easier for everybody.

Also, logging what mods load what and in what order, also eventual crashes.
Everything is changing for mods to make it compatible with Steam Workshop. Current system won't work at all and pretty much what you're suggesting will be implemented.
Including logging, override and remove?
Logged
getlost.lua # How to get rid of tavern guests
function getlost ()
   local unit = dfhack.gui.getSelectedUnit (true)
   unit.flags1.forest = true
end
getlost ()

Shonai_Dweller

  • Bay Watcher
    • View Profile
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #3 on: November 14, 2020, 08:52:24 pm »

Rather than just copy-paste into game raws, there should be:

Mod folder

Any subfolder there should be treated as separate mod. Files from there could be read as separate entities.
mod.txt for mod loading order.
'Override', for modified vanilla files and entries.
'Remove', for deleted vanilla files and entries.

This would make life easier for everybody.

Also, logging what mods load what and in what order, also eventual crashes.
Everything is changing for mods to make it compatible with Steam Workshop. Current system won't work at all and pretty much what you're suggesting will be implemented.
Including logging, override and remove?
Seems like Workshop won't work without it (no idea of the exact requirements), so it sounds like it would. Hasn't started yet though, of course so mostly just musing in the various interviews on the subject.
Logged

Evans

  • Bay Watcher
    • View Profile
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #4 on: November 15, 2020, 07:10:42 am »

Including logging, override and remove?
Seems like Workshop won't work without it (no idea of the exact requirements), so it sounds like it would. Hasn't started yet though, of course so mostly just musing in the various interviews on the subject.
I'm not talking about workshop logging and functionality, but in game script handling :)
Logged
getlost.lua # How to get rid of tavern guests
function getlost ()
   local unit = dfhack.gui.getSelectedUnit (true)
   unit.flags1.forest = true
end
getlost ()

voliol

  • Bay Watcher
    • View Profile
    • Website
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #5 on: November 15, 2020, 09:50:55 am »

Darkhog made a similar thread a few months back (and so did Thansin in 2010 (!)), but it's still worth discussing as the work on it is getting closer and closer.

Reworking the mod structure in classic as well to match with whatever Workshop needs, would be the most sound option.
My guess is that all Workshop does is provide the mod files when you click the "subscribe" button, and which one should be removed when you click "unsubscribe", and then it's up to the game's handler to actually put the files somewhere sensible/install them. I doubt there is a standardized structure for mods on Steam. That said, both versions could have a folder with subfolders for each mod, and a logging file. Using Workshop as an interface, you could install mods to this folder, as well as remove them. If you don't have premium, you can either manually put mods in that folder, or do it through whatever in-game interface allows you to enable/disable/order the installed mods. Darkhog made the comparison to how Minecraft handles resource packs, and I think it's a pretty good one.

I wonder how such a system would work for applying mods to already generated worlds though. It's important that saves are allowed to use different sets of mods, especially as downloading whole versions of DF will no longer be an "easy" way to try out mods, as getting that modded version into Steam won't be easy, and the mods themselves can't contain Mephday/the new music. Removing a mod because you don't want it in your new world can't break older saves either.

Another issue is that Steam Worshop has you "subscribe" to the mods, instead of simply installing them. This means modders can update their mods and the subscriber automatically gets the updated version, without any further downloads. This is generally a good thing, but if you allow active saves to have their mods updated, there is a risk of them breaking due to this, and if you don't people will miss out on minor updates that would be compatible with their saves, which is a shame.

Putnam

  • Bay Watcher
  • DAT WIZARD
    • View Profile
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #6 on: November 16, 2020, 12:24:08 am »

Including logging, override and remove?
Seems like Workshop won't work without it (no idea of the exact requirements), so it sounds like it would. Hasn't started yet though, of course so mostly just musing in the various interviews on the subject.
I'm not talking about workshop logging and functionality, but in game script handling :)

The steam workshop cannot work with an overhaul of in-game mod handling. Mods cannot be combined as of now, but they must be combined for the steam workshop. That's what's being said here. The way mods are loaded needs to be changed in and of itself.

ArchimedesWojak

  • Bay Watcher
    • View Profile
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #7 on: November 16, 2020, 12:08:45 pm »

ptw
Logged
YET ANOTHER DATA-COLLECTION THREAD FROM MR. "NOT FEDERAL AUTHORITIES."
ArchimedesWojak is very militant against zoophilia due to his deeply held religious beliefs.

Evans

  • Bay Watcher
    • View Profile
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #8 on: November 17, 2020, 03:39:59 am »

Darkhog made a similar thread a few months back (and so did Thansin in 2010 (!)), but it's still worth discussing as the work on it is getting closer and closer.

Reworking the mod structure in classic as well to match with whatever Workshop needs, would be the most sound option.
My guess is that all Workshop does is provide the mod files when you click the "subscribe" button, and which one should be removed when you click "unsubscribe",
This is exactly what happens, but:

Quote
and then it's up to the game's handler to actually put the files somewhere sensible/install them. I doubt there is a standardized structure for mods on Steam.
There is a steam workshop folder for a given app, where mods are downloaded to by default, and:

Quote
That said, both versions could have a folder with subfolders for each mod, and a logging file.
Steam workshop has build in logging, which is separate from any game logging as it is handled by the client, and:

Quote
Using Workshop as an interface, you could install mods to this folder, as well as remove them. If you don't have premium, you can either manually put mods in that folder, or do it through whatever in-game interface allows you to enable/disable/order the installed mods. Darkhog made the comparison to how Minecraft handles resource packs, and I think it's a pretty good one.
Workshop is basically a centralized internal file download system that has very few rules (game id, mod id, basic location, having a game etc).

How the game handles modding, how the game handles scripts depends on how modding is implemented in a given game.
For example: in my mod EvansDF my 'CAT' entity can be different from VoliolDF 'CAT' entity due to the virtue of belonging to separate pack.
Or they could be the same entity conflicting. Or they could be the same entity loaded in the order depending on how mods are loaded.
Or they could use 'override' system, and your would override default cat, mine would be a separate entity.

Considering that modders creating mods rarely keep their files neatly organized and their entities properly named, while we will name ours EVANSDF_CAT and VOLIOLDF_CAT to make them stand out, I assure you the next hundred guys from steam will go with just a 'CAT'.

This is a rather serious issue to be considered, as there are tons of DF mods out there already. Setting up the stage for massive mod conflicts is bad idea.

Quote
I wonder how such a system would work for applying mods to already generated worlds though. It's important that saves are allowed to use different sets of mods, especially as downloading whole versions of DF will no longer be an "easy" way to try out mods, as getting that modded version into Steam won't be easy, and the mods themselves can't contain Mephday/the new music. Removing a mod because you don't want it in your new world can't break older saves either.
That is another issue here. Saves could use mod list to load rather than copy-paste of raws (that was dumb idea to begin with, even tho I understand why it made sense back then). Some sort of default item/creature/recipe for replacing missing definition could be present. It's not that hard really, to prevent game from crashing because some CAT definition is missing, but it could make game act funny later.

Quote
Another issue is that Steam Worshop has you "subscribe" to the mods, instead of simply installing them. This means modders can update their mods and the subscriber automatically gets the updated version, without any further downloads. This is generally a good thing, but if you allow active saves to have their mods updated, there is a risk of them breaking due to this, and if you don't people will miss out on minor updates that would be compatible with their saves, which is a shame.
Aye, and DF saves are not for 15 minute long games either. So copying raws makes sense here. Or not.
The thing is, the way game treats definitions NOW is as if they were hardcoded once world is created. Its bad.
It was bad to begin with (we can still hear a cry of thousand modder souls for being unable to add in one recipe into an existing world).
This will have to be changed and handled somehow.

I think leaving references to '1001 sun berry dishes' in the save game, while disabling the recipes when mod is uninstalled, instead of outright deleting these references and/or crashing (so that when mod is re downloaded later it just re-enables them) would be the right way to go.

I'm not talking about workshop logging and functionality, but in game script handling :)

The steam workshop cannot work with an overhaul of in-game mod handling. Mods cannot be combined as of now, but they must be combined for the steam workshop. That's what's being said here. The way mods are loaded needs to be changed in and of itself.
Aye, exactly.

If Putnam adds ten new cat types, they need to be loaded along the base game files. If I want to give these Putnam cats poison spitting attack later, I don't want to copy-paste his whole file (as he may want to add few new cats later on) I'd rather use 'append' option and simply insert that attack into their creature definition. That way Putnam may work on his cats as he sees fit, and my mod only adds extra feature to his work. Or If I want to make a mod pack but one of Putnam's cat does not fit into the setting, I just set my mod later in loading order and disable that particular cat with append [DISABLED] tag.

Sounds easy enough, but all this has to be implemented for long term mod support.
And yes, before someone asks, there are games that do all that, even more. See Rimworld with harmony dll extension.
« Last Edit: November 17, 2020, 12:55:03 pm by Evans »
Logged
getlost.lua # How to get rid of tavern guests
function getlost ()
   local unit = dfhack.gui.getSelectedUnit (true)
   unit.flags1.forest = true
end
getlost ()

Pillbo

  • Bay Watcher
    • View Profile
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #9 on: November 17, 2020, 11:46:42 am »

I'd imagine we'll need something like the Mod Manager mod someone made for Rimworld to handle mixing of mods. Lots of Rimworld mods conflict and break each other (or require another mod loaded first) so it's common so have in the description that conflict info: This mod needs to loaded before The X mod to prevent errors.

I know a lot of work will still need to be done on the developers side, but I think the community will need to figure a lot out on our end. I hope it winds up working as well as Rimworld's mods, I've had a lot of trouble in the past trying to mix together mods and I almost always go back to not using them.
Logged

voliol

  • Bay Watcher
    • View Profile
    • Website
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #10 on: November 17, 2020, 01:44:13 pm »

I'd imagine we'll need something like the Mod Manager mod someone made for Rimworld to handle mixing of mods. Lots of Rimworld mods conflict and break each other (or require another mod loaded first) so it's common so have in the description that conflict info: This mod needs to loaded before The X mod to prevent errors.
A mod of that kind wouldn't be possible, due to the limited scope of DF modding. Luckily, that also means there aren't too many ways for DF mods to break each other/the game. Basically the only ways they can do so is by either duplication errors, or by changing stuff in the same .txt file thereby overwriting each other.

Quote
I know a lot of work will still need to be done on the developers side, but I think the community will need to figure a lot out on our end. I hope it winds up working as well as Rimworld's mods, I've had a lot of trouble in the past trying to mix together mods and I almost always go back to not using them.
The community has been getting better at using prefixes/suffixes for the object names, the past few years. If you look back at old mods, even big ones, almost all of them have duplication friendly object names. Lots of new players/modders will be a challenge for sure, but not an insurmountable one.

Evans

  • Bay Watcher
    • View Profile
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #11 on: November 17, 2020, 04:15:29 pm »

I'd imagine we'll need something like the Mod Manager mod someone made for Rimworld to handle mixing of mods. Lots of Rimworld mods conflict and break each other (or require another mod loaded first) so it's common so have in the description that conflict info: This mod needs to loaded before The X mod to prevent errors.
A mod of that kind wouldn't be possible, due to the limited scope of DF modding. Luckily, that also means there aren't too many ways for DF mods to break each other/the game. Basically the only ways they can do so is by either duplication errors, or by changing stuff in the same .txt file thereby overwriting each other.
Which is why I am making this suggestion.

Quote
Quote
I know a lot of work will still need to be done on the developers side, but I think the community will need to figure a lot out on our end. I hope it winds up working as well as Rimworld's mods, I've had a lot of trouble in the past trying to mix together mods and I almost always go back to not using them.
The community has been getting better at using prefixes/suffixes for the object names, the past few years. If you look back at old mods, even big ones, almost all of them have duplication friendly object names. Lots of new players/modders will be a challenge for sure, but not an insurmountable one.
Its perhaps the right moment to realize, everybody, that Dwarf Fortress is no longer a niche project for nerds, full of bad jokes made of bugs and exploits.
DF is on it's way to sail the high seas and gain much, much wider audience.

Robust modding system, just like robust UI system, is something you basically do once, you do it right, and maybe tweak a bit later. It is not dependent on your artefacts and civilisation generation routines, not affected by your AI, by your soundtrack, by the number of plants and animals, by rhythm of your procedurally generated poetry you put in your game.
Once you have these two out of the way, you can put in pretty much everything you may want in you game afterwards.

But without these, and with people already having experience of other games (like mentioned RimWorld), players will be drawn in by DF infamy, hit the wall of hostile UI, bounce back, and leave negative reviews. Or swamp forum with questions why changes done to raws are not reflected in existing worlds and where they can find logs for mysterious crashes.
Logged
getlost.lua # How to get rid of tavern guests
function getlost ()
   local unit = dfhack.gui.getSelectedUnit (true)
   unit.flags1.forest = true
end
getlost ()

voliol

  • Bay Watcher
    • View Profile
    • Website
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #12 on: November 17, 2020, 06:10:02 pm »

Admittedly, I assumed some ”obligatory” changes in my answer to Pillbo, such as a way for the game to protest if you have two mods creating objects with the same identifier. Like a red icon next to said mods in the in-game mod selector saying ”hey, you have a duplication error here”. But I did not communicate these thoughts, which was stupid, really. Perhaps I thought you could read my mind, perhaps I thought the discussion had already made it clear that mod conflictions was the #1 thing to be dealt with. But I digress.


As a general/separate thought, it would be nice if CREATURE_CLASS and REACTION_CLASS could be expanded to include PLANT_CLASS, ENTITY_CLASS, MATERIAL_CLASS etc.
Arbritary labels are just really useful, and they could be used to target items for mod changes as well. Say you’ve modded in a better mushroom material for your own modded shrooms, and you want all the vanilla fungi (and those modded in by other people) to use it as well. Just target all plants with PLANT_CLASS:FUNGUS and redefine their PLANT_STRUCTURAL to be based off your mushroom material template. Admittedly that would break the colors of the mushroom-trees, so maybe not the brightest example, but I hope the gist, and possible flexibility of this, is still clear.

Pillbo

  • Bay Watcher
    • View Profile
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #13 on: November 18, 2020, 12:21:13 pm »

I was basically trying to say having a program that could intelligently splice together text files from various mods (and point out those errors or conflicts) is what people will want, not whatever specifics you're saying is impossible about that Mod Manager.  I don't see Toady creating that but leaving it to the modding community to hash it out. Even with it being standard that everything get uniquely named I'm not sure that's helpful for automated mod merging, or we're just likely to get worlds with dozens of barely differently named Drow or whatever when we try to merge of bunch of monster mods.

But what do I know, I'm not a modder.
« Last Edit: November 18, 2020, 12:23:50 pm by Pillbo »
Logged

Evans

  • Bay Watcher
    • View Profile
Re: Add 'mod' folder and its handling. Add detailed log output for mod loading
« Reply #14 on: November 18, 2020, 04:54:48 pm »

As a general/separate thought, it would be nice if CREATURE_CLASS and REACTION_CLASS could be expanded to include PLANT_CLASS, ENTITY_CLASS, MATERIAL_CLASS etc.
Arbritary labels are just really useful, and they could be used to target items for mod changes as well. Say you’ve modded in a better mushroom material for your own modded shrooms, and you want all the vanilla fungi (and those modded in by other people) to use it as well. Just target all plants with PLANT_CLASS:FUNGUS and redefine their PLANT_STRUCTURAL to be based off your mushroom material template. Admittedly that would break the colors of the mushroom-trees, so maybe not the brightest example, but I hope the gist, and possible flexibility of this, is still clear.
More tags would be great. For example, I would love to see [DYED] tag for clothing items, and related - and be able to set a separate storages for dyed silks for example.

There is class MAMMAL, but there is BIRD and so on and so on. All these labels could be used later for syndromes (change bones to steel for example, why not) or special effects (target birds only) etc etc.

Possibilities are endless.

I was basically trying to say having a program that could intelligently splice together text files from various mods (and point out those errors or conflicts) is what people will want, not whatever specifics you're saying is impossible about that Mod Manager.  I don't see Toady creating that but leaving it to the modding community to hash it out. Even with it being standard that everything get uniquely named I'm not sure that's helpful for automated mod merging, or we're just likely to get worlds with dozens of barely differently named Drow or whatever when we try to merge of bunch of monster mods.

But what do I know, I'm not a modder.
I doubt it can be done without any Toady's input. But something like you describe is possible - In Skyrim(?) mods had ids, and any item there was identified as MODID_ITEMID. Or maybe it was another game.

Any splicer could go the same way, if mod has some internal ID, it could just combine items without conflicts, maybe output warnings where there are items named in the same way.
Logged
getlost.lua # How to get rid of tavern guests
function getlost ()
   local unit = dfhack.gui.getSelectedUnit (true)
   unit.flags1.forest = true
end
getlost ()
Pages: [1] 2