Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[thistleknot]

I thought we had agreed that diff's were the way to go because of the potential of 3 way merges using common ancestors, which address's the concern to:

because the whole point of this thread is that we have a common input format that any tool or version of a tool can use, which won't change with improved logic

However... it's this "merging" that is preventing a "base" version.  What your asking for at this point is something that is NOT a diff comparison and is some sort of object tracking state.

The concept of "diff"s merely means you can reconstruct any mod using patches from a base version.

THEN FROM THEIR you can do advanced merging by rebuilding mods from patches (all from a base version), and doing some fancy 3way merging (using vanilla as base) from these rebuilt mods.

However, my point was... that we don't have anything.

However, since you have said you object to the diff approach, then I will not push it any further.

However, don't be upset if someone comes along and posts a diff patch solution in the meantime (whether I decided to try or not).

Either way.  It seems someone somewhere might get impatient and throwing some stuff together; otherwise unless we solve "the diff patch problem of the century using quite extensive object state tracking", who knows when.\

However, this is your show Peredexis.  I'm not going to act like I plan on taking the helm.  However, I may fancy some proof of concept diff patch batch solution.  I was on a roll until my damn laptop ac jack went out again for the 3rd time in 6 mo's... However, most all of the base ideas I put up in this thread.

[Dirst]

I think PE's objection was just to pointing users at a URL with a diff file that wouldn't function without the launcher.  The mod would be raw files (and scripts and graphics) that the launcher processes using its latest and greatest logic.  Remember, the base on one player's machine might be different than someone else's.  For example, I always do that tweak to put beards on female dwarves.  Others might have other permanent tweaks they want like making microcline a less obnoxious color.

Do we want to encourage players to make their own personal mod of local tweaks so that everything can truly call vanilla it's common ancestor?

[PeridexisErrant]

Yep. Using a standard diff to do the merge logic makes sense to me, but for interoperability mods should be stored and distributed whole instead of as patches.

[thistleknot]

hrmmm...

Well.  I can see how basing it on a base would break if you added beards to females and every mod afterwards didn't (basically the mods would say, no we took off beards) when your intended behaviour is to draw what the mods changed from what they used as their base.  Which we all "assume" is vanilla, or at least [we assume we] can have have a common ancestor derived; which is not always true, such as with total conversion mods.

However.  I guess it could be "rebased" on a bearded version as a base... but the mod load concept I had would break?

I was under the "mis-assumption" that since LNP pretty much shipped as a static build, that we could start with a static build of mods.  Things would most likely break if players tried to mod the vanilla base after we checked the mods worked with a common ancestor system.  However, in theory we shouldn't have to 'check' them if we plan on allowing importing of them.

If I loaded mods into an imported folder (via the whole mod set), derived diff's from based off vanilla, and tried to apply it to a non vanilla base... then I think there would be an issue at that point.   Because then all diff's would be contextual vs unified and then you'd have those issues because you don't have a common ancestor anymore.

Well either way.  I'm glad I learned some stuff about 3 way merges and kdiff3 at the very least.  My brain has been hardwired lately with github thinking, so that's why I'm so pro patches.  However, merge issues such as I've had with github would arise if applying to different bases.

[Dirst]

hrmmm...

Well.  I can see how basing it on a base would break if you added beards to females and every mod afterwards didn't (basically the mods would say, no we took off beards) when your intended behaviour is to draw what the mods changed from what they used as their base.  Which we all "assume" is vanilla, or at least [we assume we] can have have a common ancestor derived; which is not always true, such as with total conversion mods.

However.  I guess it could be "rebased" on a bearded version as a base... but the mod load concept I had would break?

I was under the "mis-assumption" that since LNP pretty much shipped as a static build, that we could start with a static build of mods.  Things would most likely break if players tried to mod the vanilla base after we checked the mods worked with a common ancestor system.  However, in theory we shouldn't have to 'check' them if we plan on allowing importing of them.

If I loaded mods into an imported folder (via the whole mod set), derived diff's from based off vanilla, and tried to apply it to a non vanilla base... then I think there would be an issue at that point.   Because then all diff's would be contextual vs unified and then you'd have those issues because you don't have a common ancestor anymore.

Well either way.  I'm glad I learned some stuff about 3 way merges and kdiff3 at the very least.  My brain has been hardwired lately with github thinking, so that's why I'm so pro patches.  However, merge issues such as I've had with github would arise if applying to different bases.

There are a couple different ways to handle this, but it only makes a difference in the outcome if we distribute patches.  If we distribute raws then all of these result in the same build.

1. Make a bunch of really simple mods that match what people tweak, and include them with the Starter Pack.  So there would be a Bearded Ladies mod and a Green Microcline mod and an English-only Names mod and so on.  This could cause... clutter.

2. Set up a "base" folder and a copy of that base called "My changes" that the user can tweak.  This one can be merged in like any other mod, but it should be checked for changes on every merge since it won't have the same kind of version discipline you'd get from a "formal" mod.  Obviously, no downloaded mod with have known-compatibility with this mod, so expect some false-positives that the user will need to figure out.

3. Include a tutorial for the adventurous to build their own mod.  Basically the same as 2 except the user makes the copy and applies a version number to it.

These aren't mutually exclusive, either.  If we want to combine 1 with another, the user should need to download the mini-mods from the repository (to encourage them to find everything else there).

Edit: And actually, in thistleknot's case the beards wouldn't be mentioned in the patches so everything should be fine and dandy.  Except now there is one more tag in creature_standard.txt which mis-aligns even flattened raws.  Contextual matching ought to fix it, but I wouldn't want to depend on it.

[Button]

The beards-as-base problem isn't really a problem, tbh. You'd just have to change your installation strategy. Instead of tweaking beards and then installing, the installer should consider any deviations from vanilla in the user's raw files a mod, and install that mod last and/or with highest priority.

It does point to the biggest problem, though, which is multifeatured mods. Because mod installation is currently such a hassle, mods start packaging more and more features together, when an automatic mod installer would be significantly simpler if each feature were packaged as its own individual mod. Then we wouldn't have to worry about whether any given change is significant to the mod or not: if it deviates from the raws, it's significant.

I suppose we could just assume every change is significant to the mod, fail on any conflict, and hope that modders change their packaging to be more, well, modular. That would be a lot more helpful than a manifest file.

[Merkator]

I do some research on parsing front. For now I can parse files and make nice group of tag. Caste for know not work correctly, sadly.
More interesting for me can be one template with every, single possible token in that can exist in some place.
For example most object files have some major token like ENTITY, ITEM_*, PLANT.
After that almost always exist `head` section of raw which have name of the token, and information about it.
Next go `body` which describe features that some token have. Like GRAZER of AMPHIBIOUS.
Lastly goes `caste` section and applied variation. And few tokens that can happen almost anywhere.
Having dictionary of every token and context wher some token go make so much easier parsing raws into workable form.
But as far as I know this is gaming community and we love to have Fun.
And writing schemes for in game text format isn't the fun that we want, sadly.

But if anyone have to much time. Can you post here the tokens that happen to exist in sections.
In this gist is the list of every token (only exception are language files).
Gist

[Dirst]

The beards-as-base problem isn't really a problem, tbh. You'd just have to change your installation strategy. Instead of tweaking beards and then installing, the installer should consider any deviations from vanilla in the user's raw files a mod, and install that mod last and/or with highest priority.

That was Method 2 that I mentioned above.  The only cost is that you need some disk space for a virgin copy of the raws for comparison.

It does point to the biggest problem, though, which is multifeatured mods. Because mod installation is currently such a hassle, mods start packaging more and more features together, when an automatic mod installer would be significantly simpler if each feature were packaged as its own individual mod. Then we wouldn't have to worry about whether any given change is significant to the mod or not: if it deviates from the raws, it's significant.

I suppose we could just assume every change is significant to the mod, fail on any conflict, and hope that modders change their packaging to be more, well, modular. That would be a lot more helpful than a manifest file.

I am all for modular mods, but the manifest is useful to declare dependencies and known these-two-mods-do-not-play-well-together cases.

The hypothetical Hard Farming mod could have a core that changes the plant raws, then a module that adds a bunch of new plants, and another module that changes a bunch of reactions.  A tidy modder would build the reactions using reaction classes so that they would function with or without the new plants, but then it depends on the raw changes (adding reaction classes) in the core bit of the mod.

The core mod probably doesn't even need a manifest file (unless it wants to declare incompatibility with specific other mods), but the two modules could use manifests to declare dependency on the core mod.  In this case, load order of the mods wouldn't matter, but there are cases when in would.

To the end user, they would appear as three mods: HardFarming v1.02, HardFarming New Plants v1.02 and HardFarming Iron Chef v1.02.  If the second or third is added to the list without the first, the launcher can display a message to the user even before attempting the merge.

[PeridexisErrant]

If a user has several standard small tweaks - beards or similar - they can make a mod that makes those changes, and apply it with the others.  The launcher is not going to automatically detect user mods, but they can just copy their whole install to the folder and it'll extract the mod/s. 

The whole system is based on having a full copy of the vanilla raws in the Mods folder, which is the common ancestor of all mods.  Trying to mod this copy would be a bad idea, and reversed by the first merge anyway.  The disk space this takes is more than saved by discarding identical files in mods. 

Having just finished an assignment on graph manipulation, I'm going to take another look at difflib soon.  It still makes no sense to me...  but maybe some focus can change that, after all my outstanding essays are done. 

[PeridexisErrant]

Great news!  I've tracked down the issue I was having with King Mir's merge code, and it's now working!  Specifically, the code currently on GitHub now works to merge any number of non-conflicting mods on a line-by-line basis. 

Caveats:  the error handling is somewhere between pathetic and totally missing, and using more than one mod that adds files is almost guaranteed to mess stuff up.  The diff works line by line, and there's no flattening yet.
Upsides:  any number of non-conflicting minor mods *will* work, but any that changes a line which has already been modified will have that file skipped (silently at the moment, I did mention that the error handling was bad).

Over the next week or so I'll try to improve the feedback passed to the user and the mod-level handling of merge failures, so that eg we don't say a mod was merged in OK when two of the files were skipped.  Accumulating and displaying / responding to problems should be a thing, since we get a per-file True/False for if the merge worked.  Once that's in, enough information can be displayed for the user to make this a potentially useful tool - at least as a CLI demonstration. 

If anyone has or can easily make a collection of minor mods - which make minimal changes for their purpose, and only modify files (no additions or removals) - it would be great to share them as a test bed. 

I'm very excited about this - the remaining logic is well within my reach, and then it's ready to be GUI-ified and beta tested.

[PeridexisErrant]

OK, I stayed up later than I should have and did some more work on the surrounding logic.  I'm happy to call it version 0.2 now, and I've rewritten the readme as well.  Anyone want to test it on some simple or not-so-simple mods?

https://github.com/PeridexisErrant/Py-Mod-Loader

[expwnent]

I haven't read the whole thread, but is there a reason you don't just use git as a versioning system?

[Button]

I haven't read the whole thread, but is there a reason you don't just use git as a versioning system?

 

[PeridexisErrant]

I haven't read the whole thread, but is there a reason you don't just use git as a versioning system?

The idea is to make mods part of the starter pack, andas easy to use and add for your self as graphics packs or utilities (and working in the same way to the user).  Git is great on the mod creation side, as is find and replace, and scripting, but it's not going to be a "drag, drop, click" solution which is what we're after.

I can't blame anyone for skipping the thread though, it kinda exploded...

[expwnent]

My main line of reasoning is that if you can learn to play DF you can learn to use git. If you just make one "base" repo of default DF and do branches accordingly and merge in the mods you want it should work pretty well.