I really like this idea, and it brilliantly handles the idea of removing a mod: you just rebuild from vanilla skipping the mod-pack you no longer want.Exactly. As well as the difficulty of removing a patch, it also avoids breaking things by removing a dependency for something else. This way, every combination the user tries is explicitly checked and rejected if it won't work.
The sequence you described would be a great starting point, but collision handling needs to happen sooner rather than later because virtually ALL custom buildings and reactions need to go into entity_default.txt. I think the responsibility for treading carefully in vanilla files should be placed on the modder, with some handholding in the launcher's documentation.Yeah, more intelligent handling of collisions is going to be required before too long; I absolutely agree that special cases should be kept to a minimum - or preferably avoided entirely. To avoid incompatibility there may eventually be a canonical way of doing it, but for now keep everything as simple as possible so it actually happens!
For example, it is more reasonable to tell the modder to translate his/her positioning hints into regular expressions than it is to ask the user (who may not know the first thing about raw tokens) if a partial match is valid. I do not recommend expecting the launcher to pile on special handling for umpteen different situations (inserting custom buildings, changing gaits, adjust stone tile, etc etc etc etc); pick one or two general formats.
Currently my biggest wish for mod-manager is some sort of state tracking. I.e. I don't like how it detects if mod is already merged in.
Second in wish list is this. Namely some smarter way of adding-removing everything and being raw aware. Imho there is no silver bullet: even multi-million programming industry did not crack this problem and there are so called "merge conflicts" that need to be sorted out by hand.
Yeah, more intelligent handling of collisions is going to be required before too long; I absolutely agree that special cases should be kept to a minimum - or preferably avoided entirely. To avoid incompatibility there may eventually be a canonical way of doing it, but for now keep everything as simple as possible so it actually happens!
That's why I don't think 'soon' is now - if the first version usually only allows one mod to substantially alter entity_default.txt, that's OK for the first version.
And an idea on how to handle mods that have been split up into modules:
Note that this is not a final idea, and for now each module can simply be treated as a standalone mod with an indicative name. This is much easier to handle, and probably more elegant overall. Despite the annoyance for modders, it might just be best to require all parts of split mods to be standalone and treat them as such. Conflicts or dependencies should be caught by the launcher anyway.
- Single folder for the whole mod
- subfolders for each part of it, each of which would be valid (if useless/conflicting) as a standalone mod
- some standard way of declaring what should be loaded by default, and in what order
- user can treat the collection as a single mod, or choose to see the more detailed settings
I think that this will require a manifest, which should not be modifiable (except obviously with a text editor). The user can adjust the settings, but this is only held until the launcher is closed. I'm leaning towards xml format; the second part of the manifest being the load order for submodules. Given that there is no top-level "raw" folder, a missing manifest will mean no modules are loaded and the collection is treated as an empty mod. The manifest can also include metadata such as name of mod, author, link to source, and description (so non-collection mods may also want one, though it's optional).
Hopefully, mods which have graphics options can put the main mod in a top-level "raw" folder - ASCII raws, graphics turned off - which will always be loaded, and then have modules which add in the graphics. This would actually handle a general case of options with shared dependencies. Thinking this through raises some issues about diffs; for consistency these mod--specific graphics should be diffed from vanilla but I can see that getting impractical or conflicted.
All of this more complex handling comes later though; for now a simple launcher that can do diffs and simply rejects conflicts will be a big step up. First the simple, then the practical, then the possible, and last of all take over the world!
Currently my biggest wish for mod-manager is some sort of state tracking. I.e. I don't like how it detects if mod is already merged in.
Second in wish list is this. Namely some smarter way of adding-removing everything and being raw aware. Imho there is no silver bullet: even multi-million programming industry did not crack this problem and there are so called "merge conflicts" that need to be sorted out by hand.
The rebuild-without-thing-to-remove approach handles this nicely, but only if the original set of mods and merge order is saved. The simplest way to do this would be to create a text file in the built raw folder; "included_mods.txt". First line includes a timestamp and ID for the launcher that created it, so people can detect if it was someone else's launcher. Second line is the base DF raws, eg "df_40_08". Subsequent lines give mods in order of application: "first_mod_applied", "why do people put spaces in folder names", "thismodlast".
I think that this is premature for this idea though, when we don't even have the basic concept implemented.
[FILE:raw/objects/creature_standard.txt] (opens the specified filename)
[UNDER_TAG:CREATURE:DWARF] (finds [CREATURE:DWARF])
[CHANGE:NAME:3:dwarf:dwarves:dwarven:dorf:dorfs:dorfen] (changes the subtag [NAME:dwarf:dwarves:dwarven] to [NAME:dorf:dorfs:dorfen] - the 3 is to specify the first three values are for searching)
[FILE:raw/objects/inorganic_stone_layer.txt]
[REMOVE_ALL:AQUIFER] (removes all AQUIFER tags)
<snip>
[CREATURE:#_MAN]This mod combs through all creature entries loaded at the point where it's applied, and looks for several conditions (it must have QUADRUPED in its body tag, it must have a NAME, it must not be a MEGABEAST, SEMIMEGABEAST, or POWER). For each creature that matches these conditions, it creates a new creature entry called <CREATURE>_MAN, switches its QUADRUPED body for a HUMANOID, strips it of MUNDANE, PET, and COMMON_DOMESTIC tags, increases its size by 1, gives it ability to learn, speak, open doors and equip equipment, and gives it new names and attacks to match. The mod was, appropriately, named after Dr.Moreau. :)
[COND:BODY@1&QUADRUPED][COND:MEGABEAST!][COND:SEMIMEGABEAST!][COND:POWER!][COND:NAME]
[NAME:#man:@1#men:@1#man]
[CAN_SPEAK][CAN_LEARN][CANOPENDOORS]
[SIZE:#+1]
[BODY:QUADRUPED!!HUMANOID:>>]
[MUNDANE:!]
[PET:!]
[EQUIPS]
[COMMON_DOMESTIC:!]
[NATURAL]
[PREFSTRING:mystery]
[CHILDNAME:NAME@1#man child:NAME@1#man children]
[ATTACK:MAIN:BYTYPE:GRASP:punch:punches:1:2:BLUDGEON][ATTACKFLAG_WITH]
[ATTACK:MAIN:BYTYPE:STANCE:kick:kicks:1:3:BLUDGEON][ATTACKFLAG_WITH]
I fully agree with the basic process, but I'm not sure about keeping the mods as full raw folders - at least not long-term. It is definitely the simplest approach to distribution, given the current state of the mod community, but it does bring about a few issues.
Distributing as a complete raw folder means that it's tied down to that single version of DF. If you have a mod for, let's say, 0.40.03, then the files need to be updated for 0.40.04 - even though the actual changes made by the mod may be the same. (if all the raws changed by the mod are identical between the two DF versions, you can still reuse it, of course, but assume that things change)
<snip>
Perhaps the quick approach is good enough for now, but we should at least be sure that the limitations are acceptable (until a better solution is made).
The key point for new players is an easily loaded collection, not the advanced merging.
@Dirst: Yep, sounds like we're on the same page.I think the steps would be:
How's this for a set of goals?
v0.1 - basic logic in place, folders findable, etc. Written in Python for PyLNP compatibility
- can derive a diff correctly and create a set of raws with one mod-patch applied
v0.2 - installs multiple non-conflicting mods correctly
- handles errors and conflicts gracefully
- write log file with mod merge order etc
- place holder handling for a manifest file (note existence)
v0.9 - GUI time! Integrated as a tab in the PyLNP - get it functional in new context, refactor to fit, etc.
.1 - implement manifest and use information from it in display (if present)
- 'simplify mod folders' option ala LNP, deleting extra files (not readme etc, but eg rest of DF install)
v1.0 - start finding or soliciting or formatting some mods
I like the principle of NO SPECIAL CASES. No special syntax, no different behaviour, just diffs derived from the full set of changed files in the raws. If that makes some advanced stuff impossible, simple is better. (I've been reading this list (https://en.wikipedia.org/wiki/List_of_software_development_philosophies))Are you saying to use diff to see if two mods modify the same file, or if diff3 returns an overlap? Seeing if two mods modify the same file doesn't need diff: you can assume they do if both include the same file. Testing if diff3 returns no overlap would need to be done for every combination of added mods, so you still have N factoral checks. But you're just dealing with a small number of text files, so maybe performance won't be an issue.
As to detecting merge conflicts, there are two ways to do it - both of which work to combine files. The fast and simple way is the standard merge conflict test; if your diff returns an error show the problem in the live preview.
The slow way, which will return no false negatives, is to add and subtract combinations of diffs to check that changes don't overwrite. For example, we take vanilla DF and apply mods by diffs A, B, and C in that order. We can then check for problems by confirming that VABC-A == VBC, VABC-B == VAC, skip the case VABC-C == VAB as trivially true, and also check VABC-AB == VC for completeness. Given that this is N-1 factorial checks for N mods, it could be too slow for a live preview but if we only call it when the fast merge check returns OK that should be acceptable. If one passes and the other fails, I'd probably alert the user and allow them to decide.
Modbase, rubble, Thistleknot's project are all awesome but that's not what new players need. They don't need amazing merge tools. They just need an easy way to try some mods, and the LNP provides the model I want for graphics and utilities already. If you can only use one mod at a time that's not a fatal flaw for these users!
For example, we take vanilla DF and apply mods by diffs A, B, and C in that order. We can then check for problems by confirming that VABC-A == VBC, VABC-B == VAC, skip the case VABC-C == VAB as trivially true, and also check VABC-AB == VC for completeness. Given that this is N-1 factorial checks for N mods
I like the principle of NO SPECIAL CASES. No special syntax, no different behaviour, just diffs derived from the full set of changed files in the raws. If that makes some advanced stuff impossible, simple is better. (I've been reading this list (https://en.wikipedia.org/wiki/List_of_software_development_philosophies))
[ITEM_WEAPON:ITEM_WEAPON_SSWORD_CRUEL]
[NAME:shortsword:shortswords]
[ADJECTIVE:Cruel]
[DAMAGE@:%ITEM_WEAPON_SWORD_SHORT+10:GORE]
[WEIGHT:35]
[SKILL:SWORD]
[CRIT_BOOST:1]
[TWO_HANDED:4]
[MINIMUM_SIZE:4]
[MATERIAL_SIZE:6]
[STICK_CHANCE:30]
I like the principle of NO SPECIAL CASES. No special syntax, no different behaviour, just diffs derived from the full set of changed files in the raws. If that makes some advanced stuff impossible, simple is better. (I've been reading this list (https://en.wikipedia.org/wiki/List_of_software_development_philosophies))
I think we can go one or two steps past vanilla diffs without getting bogged down.
@King MirSo effectively you're allowing modding scripts as long as they are run after your tool, with aquifers being an example of such a script.
I'm talking about overlapping diffs, not simply diffs in the same file; the latter is as you note unproblematic. Never underestimate the impact of a factorial performance hit - the problem is that it doesn't take many more than the test case to cause a big problem. It's (n-1)! too, because the last diff on can always be reversed.
Other things like the aquifer tags should be fine, as they're not done by merging raws but rather by editing them based on tags. Which *will* remain a separate function.
I think some kind of custom format is unavoidable; it just needs to be open.I like the principle of NO SPECIAL CASES. No special syntax, no different behaviour, just diffs derived from the full set of changed files in the raws. If that makes some advanced stuff impossible, simple is better. (I've been reading this list (https://en.wikipedia.org/wiki/List_of_software_development_philosophies))
I think we can go one or two steps past vanilla diffs without getting bogged down.
The minute you go beyond vanilla diffs, you're creating a custom patch format. This means you need to re-implement patching yourself; existing libraries cannot be reused.
We don't need a mod managing tool if all we have are mini-mods that can be unzipped on top of a vanilla install.I do. I want to be able to easily add and remove mini-mods without having to manually reapply each mod to vanilla on each remove or to work out how to remove a mini-mod while leaving others intact.
The minute you go beyond vanilla diffs, you're creating a custom patch format. This means you need to re-implement patching yourself; existing libraries cannot be reused.I think some kind of custom format is unavoidable; it just needs to be open.
We don't need a mod managing tool if all we have are mini-mods that can be unzipped on top of a vanilla install. We also don't get much bang for the buck if we don't store things as diffs of some type (a graphics pack can make tiny changes to lots of large files).
I did all that... with simple strings... I could reference certain values of certain tags of certain objects, like creating a sword with a damage value dependent on that of an existing sword.Quote from: Magic Weapons submod of Martial Arts+[ITEM_WEAPON:ITEM_WEAPON_SSWORD_CRUEL]
[NAME:shortsword:shortswords]
[ADJECTIVE:Cruel]
[DAMAGE@:%ITEM_WEAPON_SWORD_SHORT+10:GORE]
[WEIGHT:35]
[SKILL:SWORD]
[CRIT_BOOST:1]
[TWO_HANDED:4]
[MINIMUM_SIZE:4]
[MATERIAL_SIZE:6]
[STICK_CHANCE:30]
It's context-sensitive in this case (pulling the damage value from the same spot of the same tag of a different item of the same type), but was intended to work just as well for different-type objects and different tags.
The minute you go beyond vanilla diffs, you're creating a custom patch format. This means you need to re-implement patching yourself; existing libraries cannot be reused.I think some kind of custom format is unavoidable; it just needs to be open.
We don't need a mod managing tool if all we have are mini-mods that can be unzipped on top of a vanilla install. We also don't get much bang for the buck if we don't store things as diffs of some type (a graphics pack can make tiny changes to lots of large files).
I do. I want to be able to easily add and remove mini-mods without having to manually reapply each mod to vanilla on each remove or to work out how to remove a mini-mod while leaving others intact.
It's unavoidable in the long-term, definitely, but for the short-term, Peridexis is proposing to use a completely standard diff (with the pitfalls that brings along), to provide a more easily attainable starting point.
If you're making a custom format, then you'd prefer to make sure it is as complete (or extensible) as we would ever need, and that's when it takes more time - and you would likely end up with something like (but not necessarily identical to) ModBase.
One word: Rubble. (http://www.bay12forums.com/smf/index.php?topic=140853.0)
In response to all the ideas about flattening raw structures, maths in scripts, etc: different goals. The point of using standard diffs here is ease of use, and eye to future size reduction in the format, and *not* advanced merging. Making two unrelated changes to a file is enough!I see your point with maths in scripts, but flattening out raws is a simple way to allow more mods to be merged with very little cost. You'd still be using a standard diff, and standard input, but comments, the lack of newlines should not prevent mod merging.
In response to all the ideas about flattening raw structures, maths in scripts, etc: different goals. The point of using standard diffs here is ease of use, and eye to future size reduction in the format, and *not* advanced merging. Making two unrelated changes to a file is enough!I see your point with maths in scripts, but flattening out raws is a simple way to allow more mods to be merged with very little cost. You'd still be using a standard diff, and standard input, but comments, the lack of newlines should not prevent mod merging.
Yeah that's what I meant. Flatten the mods in the tool.In response to all the ideas about flattening raw structures, maths in scripts, etc: different goals. The point of using standard diffs here is ease of use, and eye to future size reduction in the format, and *not* advanced merging. Making two unrelated changes to a file is enough!I see your point with maths in scripts, but flattening out raws is a simple way to allow more mods to be merged with very little cost. You'd still be using a standard diff, and standard input, but comments, the lack of newlines should not prevent mod merging.
For clarity, I'm opposed to requiring this in the input format but think it's a good idea in the logic.
It should be easy enough to flatten all the raws in memory or create temporary copies processed in whatever way we like, which are then handled however. "merge(flatten(mod1), flatten(mod2))" rather than "merge(flat_mod1, flat_mod2)". Same effect, easy enough to code, more consistent, easier and more flexible for modders.
$files = @(get-childitem -include *.txt -recurse -path $path -filter $filter)
Write-Host "files loaded";
foreach ($file in $files) {
$outfile = "$file" + ".out"
Get-Content $file | Foreach-object {
$_ -replace '\[',"[`r`n" `
-replace '\]',"`r`n]"
} | Set-Content $outfile
}$files = @(get-childitem -include *.txt -recurse -path $path -filter $filter)
Write-Host "files loaded";
foreach ($file in $files) {
$outfile = "$file" + ".out"
Get-Content $file | Foreach-object {
$_ -replace '\[',"`r`n[" `
-replace '\]',"]`r`n"
} | Set-Content $outfile
}c_variation_default
[OBJECT:CREATURE_VARIATION]
[CREATURE_VARIATION:ANIMAL_PERSON]
[CV_REMOVE_TAG:NAME]
[CV_REMOVE_TAG:GENERAL_CHILD_NAME]
[CV_REMOVE_TAG:GENERAL_BABY_NAME]
[CV_REMOVE_TAG:CASTE_NAME]
[CV_REMOVE_TAG:CHILDNAME]
[CV_REMOVE_TAG:BABYNAME]
[CV_REMOVE_TAG:SMALL_REMAINS]
[CV_REMOVE_TAG:DESCRIPTION]
[CV_REMOVE_TAG:CREATURE_TILE]
[CV_REMOVE_TAG:COLOR]
[CV_REMOVE_TAG:MAXAGE]
[CV_REMOVE_TAG:SOUND]
sed -e "s/^[^[]*//" -e "s/][^\[]*$/]/" -e "s/][^[]*\[/]\n\[/g"It removes comments and splits tokens to one per line. To just split up tokens you can do this:sed -e "s/][^[]*\[/]\n\[/g"raw>object>entity_default.txt
?[ENTITY:MOUNTAIN]
+[NOPAIN]
-[PERMITTED_REACTION:SMELT_IRON] # just sample.
@[ENTITY:DWARVES]
? find line
+ mean insert line
- remove line
@ replace line
# comment
raw>object>item_weapon.txt
?[ITEM_WEAPON:Sword]
@[ITEM_WEAPON:Big Sword]
I was thinking about some more complex version of extended diff format.
For example:Code: [Select]raw>object>entity_default.txt
?[ENTITY:MOUNTAIN]
+[NOPAIN]
-[PERMITTED_REACTION:SMELT_IRON] # just sample.
@[ENTITY:DWARVES]
WhereCode: [Select]? find line
+ mean insert line
- remove line
@ replace line
# comment
This can be easily connected with each other.
For example if you want replace tag ITEM_WEAPON:Sword with ITEM_WEAPON:Big Sword
You need to just write it like thatCode: [Select]raw>object>item_weapon.txt
?[ITEM_WEAPON:Sword]
@[ITEM_WEAPON:Big Sword]
It should be much easier that regexp.
Here's a sed (*nix utility) command that will do it:Code: [Select]sed -e "s/^[^[]*//" -e "s/][^\[]*$/]/" -e "s/][^[]*\[/]\n\[/g"It removes comments and splits tokens to one per line. To just split up tokens you can do this:Code: [Select]sed -e "s/][^[]*\[/]\n\[/g"
You can also use those regular expressions with any regular expression library.
EDIT: But instead of doing this all in shell scripts, it would be better and more portable to use Python.
$files = @(get-childitem -include *.txt -recurse -path $path -filter $filter)
Write-Host "files loaded";
foreach ($file in $files) {
$out1Pass = "$file" + ".1pass"
$outFile = "$file" + "2"
Get-Content $file | Foreach-object {
$_ -replace "`t","" `
-replace '\]\[',"]`r`n["
} | Set-Content $outFile
}
Please make the process of integrating/formatting mods into this system very very easy.
It'd be a shame if there was yet another barrier to entry in the form of a standard which everyone had to learn and comply to.
import os
import fnmatch
import fileinput
import shutil
mod_folder = 'LNP/Mods/'
mod_folders_list = []
vanilla_folders_list = []
for mod in os.listdir(mod_folder):
if mod.startswith('df_'):
vanilla_folders_list.append(mod)
else:
mod_folders_list.append(mod)
def simplify_mod_and_df_folders():
for mod in os.listdir(mod_folder):
files_removed = 0
folders_removed = 0
for item in os.listdir(mod_folder + mod):
# delete anything top-level not containing string 'raw', 'readme', or 'config'
if 'raw' in item:
pass
elif 'readme' in item.lower():
pass
elif 'config' in item.lower():
pass
elif os.path.isfile(mod_folder + mod + '/' + item):
os.remove(mod_folder + mod + '/' + item)
files_removed += 1
else:
shutil.rmtree(mod_folder + mod + '/' + item)
folders_removed += 1
if files_removed + folders_removed == 0:
print(mod, 'folder is already simplified')
else:
print(mod, 'folder simplified! (removed', files_removed, 'files and', folders_removed, 'folders)')
simplify_mod_and_df_folders()
I don't think you want to modify the mods. Just ignore the parts that you don't care about. So you want to create a list of valid files files that are to be merged, but you don't want to remove everything else.
string vanilla_raw_location //constant
string generated_raw_location //constant
mod_folders_list = find_mod_folders()
mod_raw_paths = get_raw_paths(mod_folders_list)
copy_vanilla_raws(vanilla_raw_location, generated_raw_location)
for each mod_raw_path in mod_raw_paths
for each raw_relative_path in get_valid_raw_files(mod_raw_path)
from_file = mod_raw_path + raw_relative_path
to_file = generated_raw_location + raw_relative_path
vanilla_file = vanilla_raw_location + raw_relative_path
if file(to_file) does not exist
copy_file(from_file, to_file)
else if(file(vanilla_file) does not exist
cleanup_and_abort()
else if(diff3 (from_file,vanilla_file,to_file) does not merge smoothly
cleanup_and_abort()
else
merge(from_file,to_file)
You're just doing extra work that way. Trimming functions may be useful for somethings, but you don't need them here. You just need a list of valid files specified as a mod-relative path.I don't think you want to modify the mods. Just ignore the parts that you don't care about. So you want to create a list of valid files files that are to be merged, but you don't want to remove everything else.
The goal is to imitate the LNP graphics loading; it works with a full install being referenced but you can choose to simplify / delete inactive stuff if you want to to save space. Later the removal functions can be called on temporary copies for processing if the user doesn't want to delete this stuff, but for now I'm building them and don't mind testing directly.
Next up is to detect and remove vanilla files (likewise movable to temp files later), and then the first merge logic: overwrite a copy of the vanilla folder with the mod. Actual merging comes later ;D
You're just doing extra work that way. You just need a list of valid files specified as a mod-relative path.Quite possibly. However I'd rather have working code to refactor than spend years in design, so...
$files = @(get-childitem -include *.txt -recurse -path $path -filter $filter)
Write-Host "files loaded";
foreach ($file in $files) {
$out1Pass = "$file" + ".1pass"
$outFile = "$file" + "2"
Get-Content $file | Foreach-object {
$_ -replace '(?m)^\s*','' `
-replace '(\[.+?\][^\[\r\n]*)(?=\[)' , "`$1`r`n"
} | Set-Content $outFile
}
Understood. I don't wanna tell you how to code.You're just doing extra work that way. You just need a list of valid files specified as a mod-relative path.Quite possibly. However I'd rather have working code to refactor than spend years in design, so...
Just a quick question... Valdemar's mod manager (http://www.bay12forums.com/smf/index.php?topic=74828.0) looks like it does the comparison in a better way by pulling out all the entities from the raws and comparing them on an individual basis... Since the code is already available, and this way would make resolving conflicts clearer and easier, why don't you guys use this instead of a text comparison that is, in computational terms, relatively meaningless? I don't see the long term use of a diff patch.
import os
import shutil
import filecmp
# path from script to mods folder
mods_folder = 'LNP/Mods/'
mod_folders_list = []
for mod in os.listdir(mods_folder):
if mod.startswith('df_'):
# there must be only one folder beginning 'df_', the basis of comparison
vanilla_folder = mods_folder + mod
vanilla_raw_folder = vanilla_folder + '/raw/'
else:
mod_folders_list.append(mod)
def simplify_mod_and_df_folders():
for mod in os.listdir(mods_folder):
files_removed = 0
folders_removed = 0
for item in os.listdir(mods_folder + mod):
# delete anything top-level not containing string 'raw', 'readme', or 'config'
if item == 'raw':
pass
elif 'readme' in item.lower():
pass
elif 'config' in item.lower():
pass
elif os.path.isfile(mods_folder + mod + '/' + item):
os.remove(mods_folder + mod + '/' + item)
files_removed += 1
else:
shutil.rmtree(mods_folder + mod + '/' + item)
folders_removed += 1
if not files_removed + folders_removed == 0:
print(mod, 'folder simplified! (removed', files_removed, 'files and', folders_removed, 'folders)')
def remove_vanilla_files_from_mod_raws():
for mod in mod_folders_list:
mod_raw_folder = mods_folder + mod + '/raw/'
files_removed = 0
for file_tuple in os.walk(mod_raw_folder):
for item in file_tuple[2]:
item_path_str = os.path.join(file_tuple[0], item).replace('\\', '/').replace(mod_raw_folder, '')
if os.path.isfile(vanilla_raw_folder + item_path_str): # if the file exists in the vanilla raws
if filecmp.cmp(mod_raw_folder + item_path_str, vanilla_raw_folder + item_path_str): # and it's the same file
os.remove(mod_raw_folder + item_path_str)
files_removed += 1
if files_removed > 0:
print('\n' + mod + ':\n removed', files_removed, 'files identical to vanilla raws')
def make_raws_with_mods():
print('What mods do you want to load?')
for mod in mod_folders_list:
print(' ', mod_folders_list.index(mod), mod)
mods_to_load = input('Enter the indicies in load order seperated by spaces.\n ')
# gives ordered list with indicies as strings
mods_to_load = mods_to_load.split(' ')
print('Just kidding, this doesn\'t do anything yet!')
simple_folders_q = input('Do you want to simplify mod folders? (y/n)\n ')
if simple_folders_q == 'y':
simplify_mod_and_df_folders()
print(' Done!\n')
else:
print('Not simplifying mod folders\n')
remove_files_q = input('Do you want to remove files from mods which are identical to vanilla raws? (y/n)\n ')
print(remove_files_q)
if remove_files_q == 'y':
remove_vanilla_files_from_mod_raws()
print(' Done!\n')
else:
print('Not removing vanilla files\n')
make_raws_with_mods()
$files = @(get-childitem -include *.txt -recurse -path $path -filter $filter)
Write-Host "files loaded";
foreach ($file in $files) {
$out1Pass = "$file" + ".1pass"
$outFile = "$file" + "2"
Get-Content $file | Foreach-object {
$_ -replace "`t","" `
-replace '\]\[',"]`r`n["
} | Set-Content $outFile
}
The best'd be if there were total conversions to choose from, and also mini mods/addon packs that expand the base game, like streamlined leather and display case for eg.
Ok, I now have code that will turn a folder full od modded DF installs into a much smaller folder full of mods conforming with the format at the top of the thread. That is, 'LNP/Mods/$mod' contains only files with 'readme' or 'config' in the name, and a raw folder containing those files that are not identical to vanilla.Looks good. One small fix:
For Accelerated Modest Mod (40.08), that shrinks it from 10MB to 2MB. Not bad for such naive code!Code: [Select]import os
import shutil
import filecmp
# path from script to mods folder
mods_folder = 'LNP/Mods/'
mod_folders_list = []
for mod in os.listdir(mods_folder):
if mod.startswith('df_'):
# there must be only one folder beginning 'df_', the basis of comparison
vanilla_folder = mods_folder + mod
vanilla_raw_folder = vanilla_folder + '/raw/'
else:
mod_folders_list.append(mod)
def simplify_mod_and_df_folders():
for mod in os.listdir(mods_folder):
files_removed = 0
folders_removed = 0
for item in os.listdir(mods_folder + mod):
# delete anything top-level not containing string 'raw', 'readme', or 'config'
if item == 'raw':
pass
elif 'readme' in item.lower():
pass
elif 'config' in item.lower():
pass
elif os.path.isfile(mods_folder + mod + '/' + item):
os.remove(mods_folder + mod + '/' + item)
files_removed += 1
else:
shutil.rmtree(mods_folder + mod + '/' + item)
folders_removed += 1
if not files_removed + folders_removed == 0:
print(mod, 'folder simplified! (removed', files_removed, 'files and', folders_removed, 'folders)')
def remove_vanilla_files_from_mod_raws():
for mod in mod_folders_list:
mod_raw_folder = mods_folder + mod + '/raw/'
files_removed = 0
for file_tuple in os.walk(mod_raw_folder):
for item in file_tuple[2]:
item_path_str = os.path.join(file_tuple[0], item).replace('\\', '/').replace(mod_raw_folder, '')
if os.path.isfile(vanilla_raw_folder + item_path_str): # if the file exists in the vanilla raws
if filecmp.cmp(mod_raw_folder + item_path_str, vanilla_raw_folder + item_path_str): # and it's the same file
os.remove(mod_raw_folder + item_path_str)
files_removed += 1
if files_removed > 0:
print('\n' + mod + ':\n removed', files_removed, 'files identical to vanilla raws')
def make_raws_with_mods():
print('What mods do you want to load?')
for mod in mod_folders_list:
print(' ', mod_folders_list.index(mod), mod)
mods_to_load = input('Enter the indicies in load order seperated by spaces.\n ')
# gives ordered list with indicies as strings
mods_to_load = mods_to_load.split(' ')
print('Just kidding, this doesn\'t do anything yet!')
simple_folders_q = input('Do you want to simplify mod folders? (y/n)\n ')
if simple_folders_q == 'y':
simplify_mod_and_df_folders()
print(' Done!\n')
else:
print('Not simplifying mod folders\n')
remove_files_q = input('Do you want to remove files from mods which are identical to vanilla raws? (y/n)\n ')
print(remove_files_q)
if remove_files_q == 'y':
remove_vanilla_files_from_mod_raws()
print(' Done!\n')
else:
print('Not removing vanilla files\n')
make_raws_with_mods()
how do I go about swapping your -e lines with the -replace function?-e is not a function. It's just telling sed that each of those is a separate regular expression commands to apply. The "s" however, is.
I don't see how the -e is specifying what to match and replace with.
As a workaround, I was going to installcygwinsed for gnuwin32
$_ -replace "^[^[]*",""
$_ -replace "][^\[]*$","]"
$_ -replace "][^[]*\[","]\n\["
item_gloves
[OBJECT:ITEM]
###test###[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]
[NAME:gauntlet:gauntlets]
[ARMORLEVEL:2]
[UPSTEP:1]
[SHAPED]
[LAYER:ARMOR]
[COVERAGE:100]
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]item_gloves
[OBJECT:ITEM]
###test###
[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]###test###
[NAME:gauntlet:gauntlets]
###test###[ARMORLEVEL:2]
[UPSTEP:1]
###test###[SHAPED]
[LAYER:ARMOR]###test######test###
[COVERAGE:100]
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]
import re
import sys
print re.sub("][^[]*\[","]\n[",sys.stdin.read()) C:\Games\Dwarf Fortress\github comparisons\BasedOnVanillaRaws\BasedOnVanillaRaws
\test_SafetoDeleteMe>py
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:38:22) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> import sys
>>>
>>> print re.sub("][^[]*\[","]\n[",sys.stdin.read())
File "<stdin>", line 1
print re.sub("][^[]*\[","]\n[",sys.stdin.read())
^
SyntaxError: invalid syntax
>>>"^[^[]*"Here's an explanation of what's going on with "^[^[]*"
I searched for that string, closest I found is
"s/^[^[]*//"
in
sed -e "s/^[^[]*//" -e "s/][^\[]*$/]/" -e "s/][^[]*\[/]\n\[/g"
so
this?
sed -e "s/\s//" -e "s/][^\[]*$/]/" -e "s/][^[]*\[/]\n\[/g"
yep
Maybe your version of Python is being pedantic about () for print statements. Adding ( after print, and ) at the end should fix that.Code: [Select]C:\Games\Dwarf Fortress\github comparisons\BasedOnVanillaRaws\BasedOnVanillaRaws
\test_SafetoDeleteMe>py
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:38:22) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> import sys
>>>
>>> print re.sub("][^[]*\[","]\n[",sys.stdin.read())
File "<stdin>", line 1
print re.sub("][^[]*\[","]\n[",sys.stdin.read())
^
SyntaxError: invalid syntax
>>>
yeah, I'm just trying to nail down the "correct" parse method so I can do some diff comparisons and attempt some merges, but... comments are a big deal. They add signature data to the file by being a contextual signature, and they are harmless/uniform if put on their own line like tokens are
preserving comments also preserves commented out tokens, as well as file_name information.
the problem I've noticed with most past solutions, is... everything is checking if two brackets are next to each other, but fail to separate non bracket comments apart from token lines.
This is important because if a comment is adjacent to a token. a diff will read that as more than what it should, because diff reads the whole line. For tokens, we only want a token read, not some comment[token] or [token]comment. It should be
comment
[token]
or
[token]
comment
import re
import sys
print(re.sub("]\s*([^[]*?)\s*[","]\n\\1\n\[",sys.stdin.read()))would this match any character followed by ]?That matches any whitespace or [, so no.
And I DON'T want to remove comments (ATM)
/[\s\[]/
You need to feed it input with <raw_file.txt, and write the output to file with >outputfile.txt. so the command might look like "python pyscript.py <creature_birds.txt >creature_birds.test.txt". You can also paste the script directly into the terminal with python -c 'script_body', again specifying input and output files with < and >.
but it just sits there blank looking at me as if I'm supposed to feed it data
I tried typing in *.txt...
still blank
C:\Games\Dwarf Fortress\github comparisons\BasedOnVanillaRaws\BasedOnVanillaRaws
\test_SafetoDeleteMe>py test.py < item_gloves.txt
Traceback (most recent call last):
File "test.py", line 3, in <module>
print(re.sub("]\s*([^[]*?)\s*[","]\n\\1\n[",sys.stdin.read()))
File "C:\Python34\lib\re.py", line 175, in sub
return _compile(pattern, flags).sub(repl, string, count)
File "C:\Python34\lib\re.py", line 288, in _compile
p = sre_compile.compile(pattern, flags)
File "C:\Python34\lib\sre_compile.py", line 465, in compile
p = sre_parse.parse(p, flags)
File "C:\Python34\lib\sre_parse.py", line 746, in parse
p = _parse_sub(source, pattern, 0)
File "C:\Python34\lib\sre_parse.py", line 358, in _parse_sub
itemsappend(_parse(source, state))
File "C:\Python34\lib\sre_parse.py", line 484, in _parse
raise error("unexpected end of regular expression")
sre_constants.error: unexpected end of regular expression
C:\Games\Dwarf Fortress\github comparisons\BasedOnVanillaRaws\BasedOnVanillaRaws
\test_SafetoDeleteMe>C:\Games\Dwarf Fortress\github comparisons\BasedOnVanillaRaws\BasedOnVanillaRaws
\test_SafetoDeleteMe>python test.py <item_gloves.txt >test.txt
Traceback (most recent call last):
File "test.py", line 3, in <module>
print(re.sub("]\s*([^[]*?)\s*[","]\n\\1\n[",sys.stdin.read()))
File "C:\Python34\lib\re.py", line 175, in sub
return _compile(pattern, flags).sub(repl, string, count)
File "C:\Python34\lib\re.py", line 288, in _compile
p = sre_compile.compile(pattern, flags)
File "C:\Python34\lib\sre_compile.py", line 465, in compile
p = sre_parse.parse(p, flags)
File "C:\Python34\lib\sre_parse.py", line 746, in parse
p = _parse_sub(source, pattern, 0)
File "C:\Python34\lib\sre_parse.py", line 358, in _parse_sub
itemsappend(_parse(source, state))
File "C:\Python34\lib\sre_parse.py", line 484, in _parse
raise error("unexpected end of regular expression")
sre_constants.error: unexpected end of regular expression
item_gloves
[OBJECT:ITEM]
###test###[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]###test###
[NAME:gauntlet:gauntlets]###test###
[ARMORLEVEL:2][UPSTEP:1]###test###
[SHAPED]
[LAYER:ARMOR]###test######test###
[COVERAGE:100]
###TEST
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]
item_gloves
[OBJECT:ITEM]
###test###
[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]
###test###
[NAME:gauntlet:gauntlets]
###test###
[ARMORLEVEL:2]
[UPSTEP:1]
###test###
[SHAPED]
[LAYER:ARMOR]
###test######test###
[COVERAGE:100]
###TEST
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]
sed s/replace_me/replace_with/g <from_file.txt >to_file.txtAnd this is the python script (which can be run from file)import re
import sys
for line in sys.stdin.readlines():
print(re.sub("replace_me","replace_with",line))
(If run from the terminal be sure to use single quotes around the code or escape the double quotes.)The issue is not with python, I just had a bug in my regular expression.
sed just gets you succinctness at the cost of portability. This is is how to swap with sed:Code: [Select]sed s/replace_me/replace_with/g <from_file.txt >to_file.txtAnd this is the python script (which can be run from file)Code: [Select]import re(If run from the terminal be sure to use single quotes around the code or escape the double quotes.)
import sys
for line in sys.stdin.readlines():
print(re.sub("replace_me","replace_with",line))
Of course replace_me and replace_with as appropriate.
Regular expressions are worth learning for their own sake, so don't think of it as wasted time.
I don't know. i've spent too much time on this already. I could have just finished up my own manual parsing of tokens in c by now.
I have no idea how regular expressions work. There were some fancy suggestions to use readahead, and I think negative readahead (that were mentioned on stackexchange)
item_gloves
[OBJECT:ITEM]
###test###
[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]
###test###
[NAME:gauntlet:gauntlets]
###test###
[ARMORLEVEL:2]
[UPSTEP:1]
###test###
[SHAPED]
[LAYER:ARMOR]
###test######test###
[COVERAGE:100]
###TEST
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]
sed -e "s/\t//g" -e "s/(?m)^\s*//g" -e "s/\]\(.\)/]\r\1/g;s/ *\[/[/g;s/\(.\)\[/\1\r[/g" %%~na.txt > %%~na.outecho off
for /f %%a in ('dir /b *.txt') do sed -e "s/\t//g" -e "s/(?m)^\s*//g" -e "s/\]\(.\)/]\r\1/g;s/ *\[/[/g;s/\(.\)\[/\1\r[/g" %%~na.txt > %%~na.out
erase *.txt
ren *.out *.txt
REM remove all blanklines
REM -e "s/^ *//; s/ *$//; /^$/d; s/\r//; /^\s*$/d"
echo on
item_gloves
[OBJECT:ITEM]
###test###[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]###test###
[NAME:gauntlet:gauntlets]###test###
[ARMORLEVEL:2][UPSTEP:1]###test###
[SHAPED]
[LAYER:ARMOR]###test######test###
[COVERAGE:100]
###TEST
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]
item_gloves
[OBJECT:ITEM]
[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]
###test###
[NAME:gauntlet:gauntlets]
###test###
[ARMORLEVEL:2]
[UPSTEP:1]
###test###
[SHAPED]
[LAYER:ARMOR]
###test######test###
[COVERAGE:100]
###TEST
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]{RampageMod:13}[LARGE_ROAMING]
[BIOME:ANY_TROPICAL_FOREST]
{Zootastic:2}[BIOME:SHRUBLAND_TROPICAL]
{Zootastic:3}{RampageMod:14}[POPULATION_NUMBER:15:30]
{Zootastic:4}{RampageMod:15}[CLUSTER_NUMBER:3:7]
{RampageMod:16}[BENIGN]
[MEANDERER]
[NATURAL]
{Zootastic:5}[PREFSTRING:strength]
I do not think combinediff is exactly what I thought it was I think instead it just merges two sequential merges into one merge.Well, an a la cart menu that resolves serious conflicts is outside the scope of this project. It should be sufficient to use a method that handles having the insertion point move unexpectedly (which is what I tried to describe above). In that example, both mods tried to adjust the elephant's POPULATION_NUMBER and CLUSTER_NUMBER. The tool would note the multiple changes for the user, with last-in-winning if the merge is allowed to proceed (in this case, Zootastic values would end up in the generated raws). Note that this is pulling back from my earlier insistence on regular expressions. We get at least 80% of the usefulness here without turning modders into regex-warriors.
So no silver bullet on that.
So... no "ala cart merge of mods, if those mods have conflicts" and being able to add on any combination of things you want.
Instead, I would recommend the mod author or tool author, take a poll and ask the community what base of mods should be offered to the community, because most likely, it will be a set # of options & combinations.
I thought of something while I didn't have access to my machine. There are two reasons why N-way merged files break.
Case 1: Mod A adds or deletes lines before mod B tries to change something. Mod B's changes end up in the wrong place, which may or may not cause issues.
Case 2: Mod A and mod B try to change the same line.
....
This should ensure that changes land where the modder thinks they will based on a vanilla diff. Won't ensure that the changes are compatible, but at least they won't be off in some other object.
Does that make sense to anyone other than me?
.The two pass method Dirst suggests is essentially the same thing but done in a roundabout way.
I'd go the route of Valdemar's stuff. First flatten everything, then read/index all the objects/entities into structures, then do the diff patch generation on an object by object basis. It's basically the same thing but with with the added bonus of the program being able to say explicitly what objects were deleted, what's been added and so on. You have two levels of differences - the objects present difference, and the difference between equatable objects themselves. The process to resolve conflicts becomes much more mechanical then, rather than holding hands up and saying, "well, it doesn't work and we don't know why".
<snip>
The diff patch stuff is a good start and totally necessary, but IMO it'd be pointless to do this without going the route of interrogating the raws in a raw-language manner. PE said he was adverse to an "advanced mod loader", which I understand... But I feel this way is the minimum, not an advanced feature.
Merge logic should not present the user with choices beyond the list of mods and load order - if the logic cannot produce a known correct result, the merge should be refused. This protects users from broken raws; put another way it's the difference between a mod loader which does some merging and a mod merge tool. Details on what happened could be written to a log file, but probably shouldn't be shown to all users - remember that the target audience is people who are still new to DF and would otherwise avoid mods.
could clash with that removal
No, it could still be a problem. Imagine vanilla has raws for A, B, and C. Mod1 modifies A and removes C. Mod2 doesn't modify A or C - if it did we would refuse the merge - but instead modifies B, which has C as a dependency. The raws are now broken, unless we have some *really* impressive parsing to catch this kind of thing.
I don't know how common this might be, but for minor mods it doesn't seem too likely. I think the best way to deal with this in the short to medium term is just to live with it and reduce the chances by way of major mods, which are more likely to do this, generally being incompatible.
The format also assumes that any absent file is not deleted but rather identical to vanilla, which might help in this case but should probably be run past some modders to see if it would break things. Maybe it just needs to be the full raw folder instead of files changed from vanilla...
def make_raws_with_mods():
print('What mods do you want to load?')
for mod in mod_folders_list:
print(' ', mod_folders_list.index(mod), mod)
mods_to_load = input('Enter the indicies of the mods to load, in order, seperated by spaces:\n ')
mods_to_load = mods_to_load.split(' ')
mod_load_order = []
for index in mods_to_load:
mod_load_order.append(mod_folders_list[int(index)])
mixed_raws_folder = mods_folder + 'temp/raw/'
# remove an old folder if exists. Name reserved for this reason!
if os.path.exists(mixed_raws_folder):
if os.path.isfile(mixed_raws_folder):
os.remove(mixed_raws_folder)
else:
shutil.rmtree(mixed_raws_folder)
# create folder of vanilla raws to operate on
shutil.copytree(vanilla_raw_folder, mixed_raws_folder)
print('\nFolder for merging created - "'+mixed_raws_folder+'" - with vanilla raws.')
# start merging mods!
merge_next_mod('', mod_load_order, -1)
# get back after looping through
print('\nMod merging complete! The merged mods can be found in the mod folder as "temp".')
def merge_next_mod(next_mod, mod_load_order, mods_merged):
if not next_mod == '':
mod_merge_logic(next_mod, mods_merged)
if not mod_load_order == []:
next_mod = mod_load_order.pop(0)
mods_merged += 1
merge_next_mod(next_mod, mod_load_order, mods_merged)
def mod_merge_logic(mod, mods_merged):
print('\n(placeholder merge logic for', mod + '; no action taken;', mods_merged, 'mods merged already)')
# see eg vanilla file removal function to do per-file comparisonsThe format also assumes that any absent file is not deleted but rather identical to vanilla, which might help in this case but should probably be run past some modders to see if it would break things. Maybe it just needs to be the full raw folder instead of files changed from vanilla...
That shouldn't be such a big problem as the modder/an overseer could just put a blank file in instead..?
You could do this as a setting, but for minor mods, I think they're likely to just be the modified file in the proper raw folder and nothing else. So the default should be no file implies no changes.The format also assumes that any absent file is not deleted but rather identical to vanilla, which might help in this case but should probably be run past some modders to see if it would break things. Maybe it just needs to be the full raw folder instead of files changed from vanilla...
That shouldn't be such a big problem as the modder/an overseer could just put a blank file in instead..?
That is the obvious workaround, and if we do it in the code modders don't have to comply with the standard at all. Just reverse the current comparison tool to compare "for each file in vanilla, if no such file in mod, create blank file by that name". Identical files are still removed which saves a lot of space for micro mods, and the diff from vanilla to an empty file is tiny so no worries there either. Included above.
No, it could still be a problem. Imagine vanilla has raws for A, B, and C. Mod1 modifies A and removes C. Mod2 doesn't modify A or C - if it did we would refuse the merge - but instead modifies B, which has C as a dependency. The raws are now broken, unless we have some *really* impressive parsing to catch this kind of thing.
I don't know how common this might be, but for minor mods it doesn't seem too likely. I think the best way to deal with this in the short to medium term is just to live with it and reduce the chances by way of major mods, which are more likely to do this, generally being incompatible.
The format also assumes that any absent file is not deleted but rather identical to vanilla, which might help in this case but should probably be run past some modders to see if it would break things. Maybe it just needs to be the full raw folder instead of files changed from vanilla...
... Which is exactly why I'd go the route of Valdemar's stuff. First flatten everything, then read/index all the objects/entities into structures, then do the diff patch generation on an object by object basis. It's basically the same thing but with with the added bonus of the program being able to say explicitly what objects were deleted, what's been added and so on. You have two levels of differences - the objects present difference, and the difference between equatable objects themselves. The process to resolve conflicts becomes much more mechanical then, rather than holding hands up and saying, "well, it doesn't work and we don't know why".
In case 1 you'd know that the added lines of B would at least be in the correct object, and if the program was smart it could drop those lines in right after the correct preceding line.
In case 2 you'd know in what object the conflict arose and could prompt the user with relevant information.
This way would make this basic stuff easier, and open the door for more advanced stuff like making connections throughout the raws to entirely remove problem objects. The two pass method Dirst suggests is essentially the same thing but done in a roundabout way.
The diff patch stuff is a good start and totally necessary, but IMO it'd be pointless to do this without going the route of interrogating the raws in a raw-language manner. PE said he was adverse to an "advanced mod loader", which I understand... But I feel this way is the minimum, not an advanced feature.
Thistleknot is working on a more robust merge tool (http://www.bay12forums.com/smf/index.php?topic=142188), and there's also Rubble (http://www.bay12forums.com/smf/index.php?topic=140853.0).... Which is exactly why I'd go the route of Valdemar's stuff. First flatten everything, then read/index all the objects/entities into structures, then do the diff patch generation on an object by object basis. It's basically the same thing but with with the added bonus of the program being able to say explicitly what objects were deleted, what's been added and so on. You have two levels of differences - the objects present difference, and the difference between equatable objects themselves. The process to resolve conflicts becomes much more mechanical then, rather than holding hands up and saying, "well, it doesn't work and we don't know why".
In case 1 you'd know that the added lines of B would at least be in the correct object, and if the program was smart it could drop those lines in right after the correct preceding line.
In case 2 you'd know in what object the conflict arose and could prompt the user with relevant information.
This way would make this basic stuff easier, and open the door for more advanced stuff like making connections throughout the raws to entirely remove problem objects. The two pass method Dirst suggests is essentially the same thing but done in a roundabout way.
The diff patch stuff is a good start and totally necessary, but IMO it'd be pointless to do this without going the route of interrogating the raws in a raw-language manner. PE said he was adverse to an "advanced mod loader", which I understand... But I feel this way is the minimum, not an advanced feature.
I'm with Hermes.
IMO a tool that can only load a single minor mod is trivial, because most minor mods are set up to just overwrite the vanilla folder. If you want a mod loader that can load one mod at a time, it doesn't need all this logic - just a vanilla raw directory, a bunch of minor mod raw directories, and the ability to select which minor mod you want to load on top of the vanilla raw directory in the application raw folder. To load one mod, there's no need to diff. So I assume you want something that can load more than one set of raws... but merging multiple mods through diff would be a nightmare. The dupes alone... Not to mention the really problematic part of merging, creature variations.
For example, the vampires-drink-booze-fix I have uses creature variations to replace humanoids' blood with BLOOD2 or ICHOR2, which is default blood with a syndrome. Diff wouldn't find any problems loading the boozefix alongside another mod that did something to blood, but trying to load a creature with modifications to BLOOD after BLOOD has already been replaced with BLOOD2 (and so no longer exists in the creature definition)? There's no way to find those kinds of conflicts without loading raws.
PeridexisErrant is aiming for a simple way to sample mods, and for a pre-1.0 it's perfectly fine for it to be one mod at a time. I believe that a useable tool should be able to handle some simple merging, like putting two mod's worth of PERMITTED_REACTIONs into the same entity_default.txt file, or modifying one creature when a creature before it in the same file had a line added/removed. In this regime, two mods are in conflict only in the narrow sense that they attempt to modify the same vanilla tag. That could be an insta-fail or a last-in-wins depending on settings, especially since load order is user-definable.
if the logic cannot produce a known correct result, the merge should be refused. This protects users from broken raws; put another way it's the difference between a mod loader which does some merging and a mod merge tool.
A manifest file (which is planned) can list dependencies, and it could also list known semantic conflicts like the one you mentioned. Such a manifest entry could specify whether the mod is completely incompatible, or there is a required load order.
I am strongly opposed to anything which would require a more complicated input than folders of changed raws, as outlined in OP. This is because I feel having a simple, usable, standard format is very important for adoption from both users and modders. If it becomes common to have a readme.txt and configuration.xml - to hold info such as author, name, homepage, base DF version, etc in the latter case - it would be good to use that information but their absence must be handled gracefully.
sed -r "s/\t//g;s/([^]].+)\[/\1\n[/g;s/\]([^[].+)$/]\n\1/g" testfile.txtecho off
for /f %%a in ('dir /b *.txt') do sed -r "s/\t//g;s/([^]].+)\[/\1\n[/g;s/\]([^[].+)$/]\n\1/g" %%~na.txt > %%~na.out
REM erase *.txt
REM ren *.out *.txt
REM remove all blanklines
for /f %%a in ('dir /b *.out') do sed -e "s/^ *//; s/ *$//; /^$/d; s/\r//; /^\s*$/d" %%~na.out > %%~na.out2
echo on
item_gloves
[OBJECT:ITEM]
###test###
[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]###test###
[NAME:gauntlet:gauntlets]
###test###[ARMORLEVEL:2]
[UPSTEP:1]
###test###[SHAPED]
[LAYER:ARMOR]###test######test###
[COVERAGE:100]
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]
item_gloves
[OBJECT:ITEM]
###test###
[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]
###test###
[NAME:gauntlet:gauntlets]
###test###
[ARMORLEVEL:2]
[UPSTEP:1]
###test###
[SHAPED]
[LAYER:ARMOR]
###test######test###
[COVERAGE:100]
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]
QuoteA manifest file (which is planned) can list dependencies, and it could also list known semantic conflicts like the one you mentioned. Such a manifest entry could specify whether the mod is completely incompatible, or there is a required load order.
It's planned? Haven't seen it on this thread. As far as manifest files, PE said just the other day that:I am strongly opposed to anything which would require a more complicated input than folders of changed raws, as outlined in OP. This is because I feel having a simple, usable, standard format is very important for adoption from both users and modders. If it becomes common to have a readme.txt and configuration.xml - to hold info such as author, name, homepage, base DF version, etc in the latter case - it would be good to use that information but their absence must be handled gracefully.
Which would seem to imply no manifest file required. If planning is going on/that requirement has been reversed elsewhere, I'll shut up and let you work, since I don't want to butt in when I don't have all the information.
echo off
REM s/\][^]]*/&\n/g doesn't work right
REM to remove [[ and ]] s/\][^]]*/&\n/g; s/\[[^[]*/&\n/g
REM to split up ][
REM for /f %%a in ('dir /b *.txt') do sed -e "s/\]\[/\]\n\[/g" %%~na.txt > %%~na.out
REM put tokens on their own line
for /f %%a in ('dir /b *.txt') do sed -e "s/\[[^][]*\]/\n&\n/g" %%~na.txt > %%~na.out
REM remove tabs
for /f %%a in ('dir /b *.out') do sed -r "s/\t//g" %%~na.out > %%~na.out2
REM for /f %%a in ('dir /b *.out') do sed -r "s/\t//g;s/([^]].+)\[/\1\n[/g;s/\]([^[].+)$/]\n\1/g" %%~na.out > %%~na.out2
REM remove all blanklines
for /f %%a in ('dir /b *.out2') do sed -e "s/^ *//; s/ *$//; /^$/d; s/\r//; /^\s*$/d" %%~na.out2 > %%~na.out3
REM cleanup
REM erase *.txt
REM ren *.out3 *.txt
REM erase *.out
echo on
sed -e "s/\[[^][]*\]/\n&\n/g"
The problem with relying on a manifest is that it massively cuts down on your potential input. At the moment, that's every mod out there - if it can work installed on vanilla DF (without graphics or dfhack), that install can be fed into the tool / converted to our format / stupidly merged. If we require a manifest, that shrinks the pool to those mods made with this tool in mind or manually updated or compatibility by someone, which is to say no mods at all. So requiring a manifest is fine, so long as we can derive a sensible one from the mod alone, and the defaults aren't too restrictive. Good ways to use it would be to show extra information to the user (author, update link, etc), or helpful but non-critical info for the program - eg known non-conflicting mods (useful for mods that were split up, program then ignores detected conflicts between them as false), base raws version, etc. Again though, the program should work well enough without a manifest file.
The problem with relying on a manifest is that it massively cuts down on your potential input. At the moment, that's every mod out there - if it can work installed on vanilla DF (without graphics or dfhack), that install can be fed into the tool / converted to our format / stupidly merged.
the double type was an accident
as to:
"remove any line consisting entirely of square brackets would be nice"
that was done intentionally. To keep the contextual information in place (such as commented out options like mw has). The mess of brackets was due to me purposely putting a bunch in.
This format should work. It merely extracts [tokens] and put's them on their own line. Everything else is left to whatever line it was on.
There is a much simpler version of the script that just grabs the [tokens], but you'd have to reinject the filename at the topCode: [Select]sed -e "s/\[[^][]*\]/\n&\n/g"
Modders still have to pack responsibly or you must include a manifest. ... either way there is an onus on the modder to conform to the raw folder structure, which they currently do not do at all.
Creating a catch-all formatter that would magically pull out all the relevant files is impossible. The Wanderer's Friend mod dumps everything within the raw structure, but the user has to select which features they want to install by selectively copying files across. Only way you can do this properly is to have a sentient/English speaking AI that doesn't mind spending its life reading mod readme.txt files.
Point is, somewhere in the chain a human has to check it over and understand if the mod is compatible with the loader. This line...QuoteThe problem with relying on a manifest is that it massively cuts down on your potential input. At the moment, that's every mod out there - if it can work installed on vanilla DF (without graphics or dfhack), that install can be fed into the tool / converted to our format / stupidly merged.... is false. Some mods, by chance, will work. Many/most of them won't or will be incorrectly installed and produce unwanted/unintended behaviour.
If I was going to future proof this, I'd bite the bullet and put a manifest in from the start. Make another program or a webform or something that can generate the manifest easily, or just some guidelines. Set up some mods yourself this way and then encourage others to use it. Make sure the manifest has version control, then later you can make it work with DFHack scripts and other things.
@ECHO OFF
SETLOCAL EnableDelayedExpansion
FOR /f "tokens=*" %%a IN ('DIR /b /a-d "*.txt"') DO (
SET Var=%%a
ECHO !Var:~0,-4!>>TempFile.txt
ECHO.>>TempFile.txt
grep -oE "\[[^][]+\]" %%a >>"TempFile.txt"
sed -e "s/\[[^][]*\]/\n&\n/g" "TempFile.txt" >> "TempFile2.txt"
sed -e "s/^ *//; s/ *$//; /^$/d; s/\r//; /^\s*$/d" "TempFile2.txt" > "TempFile3.txt"
DEL "%%a"
REN "TempFile3.txt" "%%a"
DEL "TempFile3.txt"
DEL "TempFile2.txt"
DEL "TempFile1.txt"
)
PAUSE
test
[OBJECT:ITEM]
[ITEM_GLOVES:ITEM_GLOVES_GAUNTLETS]
[NAME:gauntlet:gauntlets]
[ARMORLEVEL:2]
[UPSTEP:1]
[SHAPED]
[LAYER:ARMOR]
[COVERAGE:100]
[LAYER_SIZE:20]
[LAYER_PERMIT:15]
[MATERIAL_SIZE:2]
[SCALED]
[BARRED]
[METAL]
[LEATHER]
[HARD]
- transform all mod folders into our format, permanently (once ever per mod)Regarding steps one and two, I still don't get why you want to transform mods into our format. Mods should work with the tool out of the box. You can do some caching as an optimization, so that you don't have to do the same work per mod, each time you merge mods, but this is a performance optimization, and it does not make sense to try to make the process faster at the cost of ease of use. If you do want caching like that, it would be better to use a system that looks at modification time instead of manually asking the user to pre-process the Mod.
- create (temporary) flattened raws for vanilla and all mods (once per run)
- create a diff between flat vanilla and each flat mod (once per run)
- select mods to load and load order, and each time this changes:
* analyse diffs in order; if changed areas overlap reject merge and return first overlapping mod
- attempt merges in order; if any fail reject the merge and return first non-merging mod
- if merge was rejected inform the user which mod caused failure, otherwise offer new raws
Regarding a manifest file, I agree with PeridexisErrant that we don't want to have any special requirements on the format of a Mod. A mod, capable of being installed by copying over vanilla should be all that is needed for a Mod to work.
Do you guys have any specific mods you're using as a test bed? I've got some (crazy) ideas, but not sure what to best test with.No, but creating a testbed would be useful. Thistleknot suggested testing merging DF0.40.x raws and accelerated mod raws over DF0.34.11, but that's an ambitious goal. Another obvious goal is any particular mini-Mod on top of any graphics pad.
OK, here's where I leave this for while: input format, folder structure, and a code skeleton that works... for everything except the actual merge bit. You may want to expand the information passed through the functions to the merge logic, I kept it minimal so it doesn't seem to even hold folder variables.
Download link. (http://dffd.wimbli.com/file.php?id=9428)Spoiler: code (click to show/hide)
I started working on a script to
A)Identify conflicting mod files based on a diff
B)Merge non-conflicting files.
The intent is to be two function that work on multiple versions of the same file.
A simple implementation of this would be to implement A as assuming any two non identical files are conflicting and B as moving the mod file to the generated mod. My script would be a drop in replacement for that.
testbed modsI just visited DFFD for a couple of mods, I think Rise of the Mushroom Kingdom as a major mod, Accelerated Modest Mod in the middle, and Eevee Fortress which just adds them as a playable civ. As noted above some mods require installation over vanilla before they're usable, but that's not much of a challenge.
manifestReading everything relevant to merge logic from the content is the plan, but an optional manifest is still nice for stuff like displaying the author and an update link, maybe a one-sentence summary of the mod, that kind of thing. Some fields can be autofilled from the raws / folder names / etc, but others need a human to write them.
I should have caught this earlier. This is a horribly wasteful way to loop, and may cause a stack overflow. Just write a for loop that loops over each mod. No need for recursion.Code: (just the mod merging functions) [Select]def merge_next_mod(next_mod, mod_load_order, mods_merged):
if not next_mod == '':
mod_merge_logic(next_mod, mods_merged)
if not mod_load_order == []:
next_mod = mod_load_order.pop(0)
mods_merged += 1
merge_next_mod(next_mod, mod_load_order, mods_merged)
Yeah that's one of the functions I'm basically working on. The other is to see if merging is safe in the first place.OK, here's where I leave this for while: input format, folder structure, and a code skeleton that works... for everything except the actual merge bit. You may want to expand the information passed through the functions to the merge logic, I kept it minimal so it doesn't seem to even hold folder variables.
Download link. (http://dffd.wimbli.com/file.php?id=9428)Spoiler: code (click to show/hide)I started working on a script to
A)Identify conflicting mod files based on a diff
B)Merge non-conflicting files.
The intent is to be two function that work on multiple versions of the same file.
A simple implementation of this would be to implement A as assuming any two non identical files are conflicting and B as moving the mod file to the generated mod. My script would be a drop in replacement for that.
If you start with the code above, you can just fill out the function "def mod_merge_logic(mod, mods_merged):" - which was the idea of posting it :) However I think that passing arguments to these functions may actually be counterproductive, as you lose all the other variables that have been set up (like, eg, paths and number of mods already merged).
I plan on (sometime soonish I hope) fixing that, adding useful comments before that function explaining the variables you can use, and adding a flatten_raws function somewhere.
As long as it's optional.QuotemanifestReading everything relevant to merge logic from the content is the plan, but an optional manifest is still nice for stuff like displaying the author and an update link, maybe a one-sentence summary of the mod, that kind of thing. Some fields can be autofilled from the raws / folder names / etc, but others need a human to write them.
I think there are some files that may "break" if flattened the way we do.Yeah, it seems some files can be flattened, others cannot. You'd have to just keep track of which files cannot, based on their name and directory.
this is taken from vanilla.
seek out [CONTEXT:HIST_FIG:TRANS_NAME] at [CONTEXT:ABSTRACT_BUILDING:TRANS_NAME] over in [CONTEXT:SITE:TRANS_NAME]
if dwarf fortress reads this as oneline... then my parse breaks it
that's in data\speech
I was able to merge the plant and advciv one's fairly easily. However, the direforge and plantfix modified the same area of permitted reactions of some types of drinks.
By the way, what do you guys think about mods that are essentially bundles of small tweaks? Would it be helpful to users to separate them a tweak at a time, or do you think that would overwhelm with too much choice & all the tweaks should be bundled together?I think this would depend on the mod. Tweaks that have the same objective probably should be bundled. But unrelated tweaks, may be better off separated.
Without a mod loading tool bundling seems to be the clear winner, but with a whole list to select from it might be better to include more modularity?
What are y'all's thoughts on applying graphics packs to the resulting raw monstrosity? (And I use the term affectionately). Are mods going to have to be included in graphics'd versions in order to be used with a tileset; will graphics packs be treated like just another mod, merged in at the end; or will there be special handling for graphics packs?Ideally graphics will be treated like any other mod, meaning mods should be applied to vanilla, and the Mod Starter Pack would merge the mod with the graphics. It remains to be seen how practical that is.
Due to practical concerns, I'd lean the other way - treat graphics as a special case and assume that we're using ASCII with [graphics:no]; note that tilesets are still compatible, just not graphics.QuoteWhat are y'all's thoughts on applying graphics packs to the resulting raw monstrosity? (And I use the term affectionately). Are mods going to have to be included in graphics'd versions in order to be used with a tileset; will graphics packs be treated like just another mod, merged in at the end; or will there be special handling for graphics packs?Ideally graphics will be treated like any other mod, meaning mods should be applied to vanilla, and the Mod Starter Pack would merge the mod with the graphics. It remains to be seen how practical that is.
Graphics packs can change raws quite a bit. A quick look at Phoebus's shows that creature tiles, language files, inorganic stones, and plant files are all changed by the pack.Due to practical concerns, I'd lean the other way - treat graphics as a special case and assume that we're using ASCII with [graphics:no]; note that tilesets are still compatible, just not graphics.QuoteWhat are y'all's thoughts on applying graphics packs to the resulting raw monstrosity? (And I use the term affectionately). Are mods going to have to be included in graphics'd versions in order to be used with a tileset; will graphics packs be treated like just another mod, merged in at the end; or will there be special handling for graphics packs?Ideally graphics will be treated like any other mod, meaning mods should be applied to vanilla, and the Mod Starter Pack would merge the mod with the graphics. It remains to be seen how practical that is.
Some graphics packs are based on standard raws and others aren't. This is historically to free up the tiles for accented letters for other things, but with the Text will be Text plugin that could be reversed - which would mean that the raw folder could be left entirely alone by graphics packs. They also have to mess around with the /data folder a lot, and if we can avoid needing to touch that we probably should - separation of concerns to avoid causing yet more conflicts. I'm aware that this limits the range of compatible mods somewhat, but I don't see encouraging a return to the baseline standard as a terrible position to take; and we can always extend later.
Graphics packs can change raws quite a bit. A quick look at Phoebus's shows that creature tiles, language files, inorganic stones, and plant files are all changed by the pack.
Graphics packs may need to be special cases, but ignoring them completely would not allow mods to work with graphics, which is a big problem.
Gfx should be easy. Just do a diff between ASCII n a gfx mod. And apply it to a 3rd party mod. :-*
:'( :'( :-*
Basic graphics processing can probably be added within the existing framework, though I assume we'll need a non-diff compare logic for image files (messy but not hard). I'd do that at the same general stage we apply the first round of upgrades, like avoiding flattening / destroying the book title files.
Don't graphics packs normally only *add* image files, not change them? Unless they change existing file, you really just need to copy over those extra PNGs (and if we want to store them in a single patch file, we could always do something like using Base64 to store the entire file in a plain-text format).
There's certainly a use-case for providing individual tiles to be replaced, but simple image copying should get us pretty far, I'd think, with or without TwbT.
Phoebus-Direforge.
I don't know why, but the phoebus fonts have screwy ascii fonts EVERYTIME I try to merge phoebus with another mod. The names of dwarf's come up with weird symbols.
Here's some of the merge conflicts I get with plant and direforge
http://imgur.com/Rjvo7RL
import difflib
def can_merge_seq (mod_text, vanilla_text, into_text):
van_mod_match = difflib.SequenceMatcher(None, vanilla_text, mod_text)
van_gen_match = difflib.SequenceMatcher(None, vanilla_text, into_text)
van_mod_seqs = van_mod_match.get_matching_blocks()
van_gen_seqs = van_gen_match.get_matching_blocks()
cur_v = 0
while cur_v < len(vanilla_text) :
(i1,j1,n1) = van_mod_seqs[0]
(i2,j2,n2) = van_gen_seqs[0]
if i1 > cur_v and i2 > cur_v:
return False
if i1 + n1 - cur_v < i2 + n2 - cur_v:
cur_v = i1+n1
van_mod_seqs.pop(0)
else:
cur_v = i2 + n2
van_gen_seqs.pop(0)
return True
print can_merge_seq ('anything at all','vanilla','vanilla')
print can_merge_seq ('oooh','vanilla','vanilla')
print can_merge_seq ('vanil','vanilla','nilla')
print can_merge_seq ('van','vanilla','la')
print can_merge_seq ('van','vanilla','la')
print can_merge_seq ('vani','vanilla','lla')
print can_merge_seq ('vonilla','vanilla','banana')
print can_merge_seq ('vannilla','vanilla','banana')
print can_merge_seq ('vonilla','vanilla','venilla')
Changes to d_init.txt and init.txt may need to be treated specially. Maybe just let graphics modify them, then post-process the files with user settings in a script.Ideally the graphics packs will only change the fields they actually need to change, making this a non-issue.
"Oh, those should be able to merge well enough together one after the other. I mean, from a logic perspective. Obviously merge tools tend to be... a little simplistic."
As to what Button was saying about the merge conflict:
http://www.bay12forums.com/smf/index.php?topic=142295.msg5586461#msg5586461Quote"Oh, those should be able to merge well enough together one after the other. I mean, from a logic perspective. Obviously merge tools tend to be... a little simplistic."
The fact that we could have just "added those lines on top of each other" means that could be an option to present the user.
I think the reason that happened is because one mod added 1 line, and the other mod added say 5 lines to the same spot. Both mods/patches read the original location of that FILE AS BLANK. Since we are only ever merging two mods at a time, since VANILLA is always our base. If both lines expected a blank spot (can be read from patch file), and they found something there (in the case of merge conflicts). It's safe to assume that we CAN ADD BOTH PATCHES, ONE ON TOP OF THE OTHER.
Actually, you know... now that I think about it. Our base files don't have blank spots. So what's happening is... their is a token mismatch. At that point we can introduce logic on how to deal with token mismatches. It's probably one mod realizing another mod wrote to that spot.
As to what Button was saying about the merge conflict:The problem is, sometimes additions in the same spot don't conflict, sometimes they do. Two mods adding a creature in the same spot are probably ok. Two mods adding a the same tag to the same creature might cause problems. A Simple solution is to not allow adding to the same spot.
http://www.bay12forums.com/smf/index.php?topic=142295.msg5586461#msg5586461Quote"Oh, those should be able to merge well enough together one after the other. I mean, from a logic perspective. Obviously merge tools tend to be... a little simplistic."
The fact that we could have just "added those lines on top of each other" means that could be an option to present the user.
I think the reason that happened is because one mod added 1 line, and the other mod added say 5 lines to the same spot. Both mods/patches read the original location of that FILE AS BLANK. Since we are only ever merging two mods at a time, since VANILLA is always our base. If both lines expected a blank spot (can be read from patch file), and they found something there (in the case of merge conflicts). It's safe to assume that we CAN ADD BOTH PATCHES, ONE ON TOP OF THE OTHER.
Actually, you know... now that I think about it. Our base files don't have blank spots. So what's happening is... their is a token mismatch. At that point we can introduce logic on how to deal with token mismatches. It's probably one mod realizing another mod wrote to that spot.
We've been talking about this in the abstract, but is there a test case of two mods that we'd like to be able to merge with one doing complex reordering, and the other adding or removing stuff? At the very least we need to make sure that we properly detect that the mods can't be naively merged.
This still doesn't do a good job with re-ordering things. To get that, you'd probably have to nuke the vanilla file completely and replace it using a different filename. The launcher would detect that one mod's changes got deleted by another, but absent some advanced raw-aware logic it just isn't possible to merge the two. That advanced logic comes... later.
We've been talking about this in the abstract, but is there a test case of two mods that we'd like to be able to merge with one doing complex reordering, and the other adding or removing stuff? At the very least we need to make sure that we properly detect that the mods can't be naively merged.
This still doesn't do a good job with re-ordering things. To get that, you'd probably have to nuke the vanilla file completely and replace it using a different filename. The launcher would detect that one mod's changes got deleted by another, but absent some advanced raw-aware logic it just isn't possible to merge the two. That advanced logic comes... later.
All right, I've got a little pack of minor mods with potentially interesting interactions uploaded now. http://dffd.wimbli.com/file.php?id=9443 .Thanks, looking at it now. Already clear to me that two mods that add the same file cannot be trivially merged.
These features coexist happily in my raws, but may be a little challenging to merge together automatically, so I hope you find them useful :).
We've been talking about this in the abstract, but is there a test case of two mods that we'd like to be able to merge with one doing complex reordering, and the other adding or removing stuff? At the very least we need to make sure that we properly detect that the mods can't be naively merged.
This still doesn't do a good job with re-ordering things. To get that, you'd probably have to nuke the vanilla file completely and replace it using a different filename. The launcher would detect that one mod's changes got deleted by another, but absent some advanced raw-aware logic it just isn't possible to merge the two. That advanced logic comes... later.
"Two mods adding a the same tag to the same creature might cause problems"If two mods add the same content in the same spot, we may be able to detect that and allow it. In the general case there's the possibility that they are changing the same token in incompatible ways.
Couldn't this type of conflict be caught by reading the two patch files and being like, "hey, BOTH OF THESE are adding the same token in the same contextual match".
Of course it might be a little simplistic.
Two mods might be incorporating modest mod fixes but reordered the tokens! Aghast!
That would still result in duplicates.
To address that situation, advanced mod merging would have to be implemented and detect object individual token +/- changes
You'd have to figure out top level tokens for that. That's not in PeridexisErrant's version 1 objectives. It also means you have to track two kinds of changes separately: those that add/remove/reorder top level tokens, and those that modify existing entities.We've been talking about this in the abstract, but is there a test case of two mods that we'd like to be able to merge with one doing complex reordering, and the other adding or removing stuff? At the very least we need to make sure that we properly detect that the mods can't be naively merged.
This still doesn't do a good job with re-ordering things. To get that, you'd probably have to nuke the vanilla file completely and replace it using a different filename. The launcher would detect that one mod's changes got deleted by another, but absent some advanced raw-aware logic it just isn't possible to merge the two. That advanced logic comes... later.
One solution to this problem (and is what I asked RawExplorer mod author a while back if he could incorporate).
Was object tag-id alphabetizing.
However, comments kind of create havoc with that. However, if WE DID ALL THIS ON OUR END. We could alphabetize our flattened raws based on some parse token-id trick.
The great thing about it is, the FIRST <token> is our object:type, so immediately, we can determine what we should be alphabetizing on.
All right, I've got a little pack of minor mods with potentially interesting interactions uploaded now. http://dffd.wimbli.com/file.php?id=9443 .So as an update on this here's what I found:
These features coexist happily in my raws, but may be a little challenging to merge together automatically, so I hope you find them useful :).
So as an update on this here's what I found:
2)the two larger mods modify c_variation_default.txt with the same changes, which should be ok.
Woops. Good catch. You're right. Changes are in different locations, so there is no problem.So as an update on this here's what I found:
2)the two larger mods modify c_variation_default.txt with the same changes, which should be ok.
Rreally? They should have compatible but different changes, in different locations.
EDIT: Didn't understand the problem.
The short answer is no, because we want to be conservative about allowing conflicting mods. If you don't care about failing on true conflicts, then you may have some luck making a smarter merge tool that sometimes does the wrong thing.
I'm working on a 3 way merge script atm, (testing it at this point) but I don't plan to make it too fancy, because I don't want to merge mods that really do conflict.
N-way merge algorithm here:EDIT: Didn't understand the problem.
The short answer is no, because we want to be conservative about allowing conflicting mods. If you don't care about failing on true conflicts, then you may have some luck making a smarter merge tool that sometimes does the wrong thing.
I'm working on a 3 way merge script atm, (testing it at this point) but I don't plan to make it too fancy, because I don't want to merge mods that really do conflict.
I'm sorry, i've been keeping up like 80% of the posts since about page 5 or 6, I parsed through page 5 and didn't see where you reference n way merges. The thought is familiar, I'd love to see your solution.
If two mods add content to the same spot, like to the same creature, there is a high chance that the merges conflict, especially if they also remove a tag in that spot.
If two mods add content to the same spot, like to the same creature, there is a high chance that the merges conflict, especially if they also remove a tag in that spot.
The question is, though, what about two mods that add content to the same spot because that's the natural place to add things? Like, at the end of a creature variation definition for instance. It's usual to add new tags at the end of the variation, not in the middle somewhere.
Then they might add conflicting content. I'd err on the side of safety and not allow the modification. Unless they are adding the same content, in which case you don't want to add it twice.If two mods add content to the same spot, like to the same creature, there is a high chance that the merges conflict, especially if they also remove a tag in that spot.
The question is, though, what about two mods that add content to the same spot because that's the natural place to add things? Like, at the end of a creature variation definition for instance. It's usual to add new tags at the end of the variation, not in the middle somewhere.
How automated do you want to have it? I have merged dozens of mods by hand, its not that much work...The idea is to have it completely automated and able to merge and remove mods.
Think about this simple case: 1 mod adds a creature specific token at the end of a creature definition. another adds a creature. If you add them in the wrong order, you could get the creature specific token to the wrong creature if you don't order the merge right.
One mod adds [PET] at the end of a vanilla creature. Another mod adds a new creature at the same spot. Both are merely additive. But if merged in the wrong way, the result would add [PET] to the new creature instead of the vanilla creature.Think about this simple case: 1 mod adds a creature specific token at the end of a creature definition. another adds a creature. If you add them in the wrong order, you could get the creature specific token to the wrong creature if you don't order the merge right.
I think some simple way to either derive a precombined patch
or to keep track of line changes as you add/remove lines would b in order to address this concern
Basically, you would check against vanilla, and determine if the changes are merely additive or subtractive.
Then if they are additive, it's all green light. Merge with rest of patches (barring any duplicate line additions?)
Yeah, it get's complicated apparently...
I was just assuming always checking a patch against vanilla would the base "Additive" part of the mod. And if you could somehow map the BEFORE and AFTER line states of working mod vs vanilla, you could somehow accommodate future line injections based on tracking insertion points or something.
but apparently, it gets complicated, and one might as well do some object tracking at that point.
What happens to mutually exclusive mods?That's a rather benign error. The result is still a valid set of raws. So it's not a big problem.
One removes kimberlite, the other adds new uses for diamonds (glass cutting for example). No kimberlite = No diamonds. While the Raws would fit and the program would merge them, you would have an unuseable mod in the end.
import difflib
def do_merge_seq (mod_text, vanilla_text, gen_text):
if vanilla_text == gen_text: #this should happen often
return mod_text
van_mod_match = difflib.SequenceMatcher(None, vanilla_text, mod_text)
van_gen_match = difflib.SequenceMatcher(None, vanilla_text, gen_text)
van_mod_ops = van_mod_match.get_opcodes()
van_gen_ops = van_gen_match.get_opcodes()
output_file_temp = []
cur_v = 0
while cur_v < len(vanilla_text) :
(mod_tag, mod_i1, mod_i2, mod_j1, mod_j2) = van_mod_ops[0]
(gen_tag, gen_i1, gen_i2, gen_j1, gen_j2) = van_gen_ops[0]
#print van_mod_ops[0]
#print van_gen_ops[0]
#print cur_v
if mod_tag == 'equal' and gen_tag == 'equal' :
if mod_i2 < gen_i2:
output_file_temp += vanilla_text[cur_v:mod_i2]
cur_v = mod_i2
van_mod_ops.pop(0)
else:
output_file_temp += vanilla_text[cur_v:gen_i2]
cur_v = gen_i2
van_gen_ops.pop(0)
if mod_i2 == gen_i2 :
van_mod_ops.pop(0)
else:
if mod_tag != 'equal' :
output_file_temp += mod_text[mod_j1:mod_j2]
cur_v = mod_i2
van_mod_ops.pop(0)
if mod_i2 == gen_i2 :
van_gen_ops.pop(0)
elif gen_tag!='equal':
output_file_temp += gen_text[gen_j1:gen_j2]
cur_v = gen_i2
van_gen_ops.pop(0)
if mod_i2 == gen_i2 :
van_mod_ops.pop(0)
#if neither gen_tag nor mod_tag is 'equal', this mod can't be merged
#print (output_file_temp)
if van_mod_ops:
(mod_tag, mod_i1, mod_i2, mod_j1, mod_j2) = van_mod_ops[0]
output_file_temp += mod_text[mod_j1:mod_j2]
if van_gen_ops:
(gen_tag, gen_i1, gen_i2, gen_j1, gen_j2) = van_gen_ops[0]
output_file_temp += gen_text[gen_j1:gen_j2]
return output_file_temp
vanilla_file = "vanilla"
output_file = "vanilla"
print ("----")
print output_file
output_file = do_merge_seq ('anything at all', vanilla_file, output_file)
print output_file
print ("----")
output_file = vanilla_file
print output_file
output_file = do_merge_seq ('nilla', vanilla_file, output_file)
print output_file
output_file = do_merge_seq ('vanill', vanilla_file, output_file)
print ''.join(output_file)
print ("----")
output_file = vanilla_file
print output_file
output_file = do_merge_seq ('vonilla', vanilla_file, output_file)
print output_file
output_file = do_merge_seq ('banana', vanilla_file, output_file)
print ''.join(output_file)
print ("----")
output_file = vanilla_file
print output_file
output_file = do_merge_seq ('banana', vanilla_file, output_file)
print output_file
output_file = do_merge_seq ('vonilla', vanilla_file, output_file)
print ''.join(output_file)
print ("----")
output_file = vanilla_file
print output_file
output_file = do_merge_seq ('banana', vanilla_file, output_file)
print output_file
output_file = do_merge_seq ('vanilla', vanilla_file, output_file)
print ''.join(output_file)
I don't understand why you guys are beating around the bush with this. The way PE wants the version 1 to work you should be able to detect all these conflicts but you have to refuse the merge each time. There is no way a blind diff patch can merge any mod that takes something away, guaranteed error free.
Appealing to PE again... you're in a much better position than I ever have been to impose a standard for mods, because the starter pack you curate is so popular. I don't see what the rush is, if you spend some time trying to set up a single, decent system that is flexible enough to handle the diverse range of DF mods, that would be great.
However, I do agree that there should be a non-manifest way of merging, so I'd go the route of making a two tier mod-integration system where mods without a manifest are given lower priority and are more readily excluded if they remove objects from the raws. As everyone seems to be slowly realizing, there are so many potential conflicts you have to go all the way to actually make the thing work properly.
That said I feel like I'm trolling a bit, so I'll bow out here for now, I guess you guys are going in a different direction than I'd anticipated. Really hope it works out, good luck! :)
I was thinking I might write a last-pass errorchecker which could do some basic raw comprehension and catch errors which the diff might miss - finding dupes across files, reactions with no possible reagents, things like that. Could be expanded into a more fully-functional syntactic parser later. It'll give me an excuse to pick up Python again - I've been stuck doing Java at work.
import re
import sys
if len(sys.argv) < 2:
filepath = str(input("FILEPATH> "))
else:
filepath = sys.argv[1]
def reader(filepath):
tokens = []
name = ""
with open("test.txt", 'r') as file:
for line in file:
if name == "":
name = line
for i in re.findall("\[[A-Za-z0-9:]*?\]", line):
tokens.append(i)
return (name, tokens)
def main():
name, tokens = reader(filepath)
for i in tokens:
print(i)
if __name__ == "__main__":
main()
{
"name":<name of the mod>,
"version":<version of the mod>,
"compatible-major":{
"name-of-major-mod":"version-of-major-mod"
}
}
Something like this. Where * is wildcard.
{
"name":"Mycrazymod",
"version":"0.0.1",
"compatible-major": {
"MasterworkDF":"5.*"
}
}
C:\Games\Dwarf Fortress\mod testbed\df_40_09_win [3-way]> git branch 40_09_DireForge-Flattened f18f255
C:\Games\Dwarf Fortress\mod testbed\df_40_09_win [3-way]> git merge 3459e01
Auto-merging raw/objects/plant_standard.txt
CONFLICT (content): Merge conflict in raw/objects/plant_standard.txt
Auto-merging raw/objects/entity_default.txt
CONFLICT (content): Merge conflict in raw/objects/entity_default.txt
Automatic merge failed; fix conflicts and then commit the result.
C:\Games\Dwarf Fortress\mod testbed\df_40_09_win [3-way +48 ~23 -0 !2 | +0 ~0 -0 !2]>
It's a git branching issue. I DO NOT KNOW if it can be done offline. What this really is, is a way for me to merge a ton of mods now.C:\Games\Dwarf Fortress\mod testbed\df_40_09_win [3-way +2 ~0 -0 !]> git merge 514bde0 13a563f
Trying simple merge with 514bde0
Trying simple merge with 13a563f
Merge made by the 'octopus' strategy.
raw/objects/creature_darkdwarf.txt | 363 +++++++++++++
raw/objects/creature_firedwarf.txt | 363 +++++++++++++
raw/objects/creature_frostdwarf.txt | 362 +++++++++++++
raw/objects/creature_stonedwarf.txt | 362 +++++++++++++
raw/objects/creature_stormdwarf.txt | 362 +++++++++++++
raw/objects/creature_wilddwarf.txt | 362 +++++++++++++
raw/objects/descriptor_shape_standard.txt | 25 +
raw/objects/entity_darkdwarf.txt | 821 +++++++++++++++++++++++++++++
raw/objects/entity_default.txt | 9 +
raw/objects/entity_firedwarf.txt | 815 +++++++++++++++++++++++++++++
raw/objects/entity_frostdwarf.txt | 817 +++++++++++++++++++++++++++++
raw/objects/entity_stonedwarf.txt | 822 ++++++++++++++++++++++++++++++
raw/objects/entity_stormdwarf.txt | 815 +++++++++++++++++++++++++++++
raw/objects/entity_wilddwarf.txt | 816 +++++++++++++++++++++++++++++
raw/objects/plant_crops.txt | 218 +++++---
raw/objects/plant_garden.txt | 169 +++++-
raw/objects/plant_standard.txt | 128 ++++-
raw/objects/reaction_plantfix.txt | 92 ++++
18 files changed, 7618 insertions(+), 103 deletions(-)
create mode 100644 raw/objects/creature_darkdwarf.txt
create mode 100644 raw/objects/creature_firedwarf.txt
create mode 100644 raw/objects/creature_frostdwarf.txt
create mode 100644 raw/objects/creature_stonedwarf.txt
create mode 100644 raw/objects/creature_stormdwarf.txt
create mode 100644 raw/objects/creature_wilddwarf.txt
create mode 100644 raw/objects/entity_darkdwarf.txt
create mode 100644 raw/objects/entity_firedwarf.txt
create mode 100644 raw/objects/entity_frostdwarf.txt
create mode 100644 raw/objects/entity_stonedwarf.txt
create mode 100644 raw/objects/entity_stormdwarf.txt
create mode 100644 raw/objects/entity_wilddwarf.txt
create mode 100644 raw/objects/reaction_plantfix.txt
C:\Games\Dwarf Fortress\mod testbed\df_40_09_win [3-way]>
git checkout master
git pull origin feature1 feature2
git checkout develop
git pull . master (or maybe git rebase ./master)
The first command changes your current branch to master.
The second command pulls in changes from the remote feature1 and feature2 branches. This is an "octopus" merge because it merges more than 2 branches. You could also do two normal merges if you prefer.
The third command switches you back to your develop branch.
The fourth command pulls the changes from local master to develop.
Hope that helps.
EDIT: Note that git pull will automatically do a fetch so you don't need to do it manually. It's pretty much equivalent to git fetch followed by git merge.
git merge B CThis short script should extract every single token in file. Maybe someone find it useful.We already hit on the token extraction two ver... Both sed regex commands; , one that derives tokens using regex like ur ex then injects filename on top, and another that puts tokens on their own lines effectively keeping comments. Both ver remove all whitespace. :)
Silly question among all this talk of different technical problems and solutions: Has anyone asked around, written some PMs to mod authors about writing and packaging their mods in a standardized system?
Because I havent seen a single modder here, besides Putnam and me, and both our usual projects are a bit large to be included as 'minor mods' that can be easily packaged.
A button? :)
No, sorry, I dont connect your name with any mods I recall... and there are no threads in the mod release board by your name. So as unfortunate as it is, I have to declare you a button. ;)
Highly detailed explanation
AlphabetizingA multi-line match for regular expressions should be able to scoop up everything between tags and associate it with the next valid tag. The special cases are (1) the header up to the [OBJECT:FOO] token, (2) header comments after [OBJECT:FOO] but before the first [FOO:BAR] and (3) comments at the end of the file. (1) and (3) should be easy. We just have to pick a standard for (2), is it associated with the header or the first object?
I've been pondering this idea on how to alphabetize raws to address the issue of TC mods or mods that completely reorder the contents of the files.
It may be possible with regular expressions (sed, or python, or grep), but unfortunately, the solution I'm thinking of involves breaking comments that are listed right before an object would be broken apart from the next object... so, I'd rather just remove comments than try to parse them using some batch script.
Otherwise, a 3-way merge is probably going to require the source directories of each version and then in a shell script:
mkdir VersionABC
for i in (cd VersionA && ls -R); do
if [ -f VersionA/$i ]; then
diff3 --merge VersionB/$i VersionA/$i VersionC/$i > VersionABC/$i
else
mkdir -p VersionABC/$i
fi
done
both our usual projects are a bit large to be included as 'minor mods' that can be easily packaged.
Silly question among all this talk of different technical problems and solutions: Has anyone asked around, written some PMs to mod authors about writing and packaging their mods in a standardized system?
Because I havent seen a single modder here, besides Putnam and me, and both our usual projects are a bit large to be included as 'minor mods' that can be easily packaged.
def per_file_mod_merge_logic(vanilla_raw_folder, mod_raw_folder, mixed_raw_folder, file):
if os.path.isfile(mixed_raw_folder + file):
# preprocess files here
pass
# merge logic goes here
# see https://docs.python.org/2/library/difflib.html
else:
pass
# nothing for now, but later just copy file over:
#shutil.copy(mod_raw_folder + file, mixed_raw_folder)
Standart diff format look like this
@@ -4,6 +4,6 @@
- something
+ something else
...
We can use something like that
@@ =[CREATURE:WOLVERINE]>[CASTE:MALE], 6 =[CREATURE:WOLVERINE]>[CASTE:MALE],6
-something
+something else
...
I've substantially refactored the code. We now have one monster procedure that simplifies folders (deletes all files that aren't needed per our mod format), some placeholder input stuff to get an ordered list of mods to load, and then for each mod pass each filename to a merge function for one-file-at-a-time merging.Yep, I was planning to do that already.
So basically that function needs a lot more work, but the rest is at least good enough to work as a placeholder.Code: [Select]def per_file_mod_merge_logic(vanilla_raw_folder, mod_raw_folder, mixed_raw_folder, file):
if os.path.isfile(mixed_raw_folder + file):
# preprocess files here
pass
# merge logic goes here
# see https://docs.python.org/2/library/difflib.html
else:
pass
# nothing for now, but later just copy file over:
#shutil.copy(mod_raw_folder + file, mixed_raw_folder)
https://github.com/PeridexisErrant/Py-Mod-Loader
I had a look at the functions King Mir wrote earlier in the thread, but just got errors. Do you mind having another look, and maybe trying to make them usable in the function above? Once we get that working, we have a working prototype!Spoiler: King Mir's Code (click to show/hide)
I'm also not quite sure what the final position on preprocessing of the files was - we have so many test snippets around I can't tell which would be good to use. Thistleknot - could you post a description of how you think files should be processed before doing the diffs?
[/quote]My position is avoid preprocessing that changes the mod itself, but do do preprocessing during integration.
As for what's implemented, I think thistleknot figured out how to flatten raws, but it's in sh shell scripts/stray shell commands. But apparently is was helpful in updating his mods to 0.40 so we have a useful proof of concept. And you can lift the regular expressions from it.
dir /b %2\raw\objects
mkdir VersionABC
mkdir VersionABC\raw
mkdir VersionABC\raw\objects
for /f %%f in ('dir /b %2\raw\objects') do diff3 --merge %1\raw\objects\%%f %2\raw\objects\%%f %3\raw\objects\%%f > VersionABC\raw\objects\%%f
diff3Batch.bat accmod 34_11 civforge
mkdir VersionABC
for i in (cd VersionA && ls -R); do
if [ -f VersionA/$i ]; then
diff3 --merge VersionB/$i VersionA/$i VersionC/$i > VersionABC/$i
else
mkdir -p VersionABC/$i
fi
done
C:\temp>DIFF3 -e -m civforge\raw\objectS\entity_default.txt 34_11\raw\objects\en
tity_default.txt accmod\raw\objects\entity_default.txt >temp.txt
dir /b %2\raw\objects
mkdir VersionABC
mkdir VersionABC\raw
mkdir VersionABC\raw\objects
for /f %%f in ('dir /b %2\raw\objects') do diff3 -e --merge %1\raw\objects\%%f %2\raw\objects\%%f %3\raw\objects\%%f > VersionABC\raw\objects\%%f
diff3batch modb moda modc<use a nonstandard diff format>If you want to write one, that would be awesome. It should take two files as input, derive the diff, and be able to apply that to another file.
<diff3 is awesome>Amen. Unfortunately I want this to work on random Windows computers too :'(
For a class called 6.033 (Computer Systems Engineering) at MIT, I was required to design a collaborative, distributed text editor. It’s supposed to work like Git—if two users make concurrent changes to the document, the editor should try to automatically merge the two changes (or report a merge conflict).So far as I can tell, this solves basically all of our problems if we can get permission to use it.
However, the merge algorithm had to be more intelligent than Git’s line-by-line diff merger—if one user moved a paragraph of text (for example) to a different location in the document, and another user concurrently edited the wording of that paragraph, then the merger should be able to detect that the paragraph was both edited and moved, automatically.
I wanted the merge algorithm to be even more general. It shouldn’t have any notion of “paragraph” or “sentence.” Rather, if any piece of text is moved and edited concurrently, this should be resolved automatically. Furthermore, it should work even if two (or more) users edit the text and another user user moves it. Moreover, I wanted it to even work recursively—for example, if one user moves a chapter in a book, while another user moves a paragraph within that chapter, and yet another user moves a sentence within that paragraph, and still another user changes the wording of that sentence, the merge algorithm should be able to handle all of that without any user intervention, and without any notion of “chapter,” “paragraph,” “sentence,” etc. It should just “do the right thing,” whether you’re working on code, a novel, a scientific paper, or any other text-based document.
While I wasn’t required to actually implement the algorithm, I ended up doing it in Python anyway (https://gist.github.com/boyers/4713315). I later discovered that I had independently invented the operational transformation.
[CREATURE:GIANT_LEOPARD_GECKO]
[COPY_TAGS_FROM:GECKO_LEOPARD]
[APPLY_CREATURE_VARIATION:GIANT]
[CV_REMOVE_TAG:CHANGE_BODY_SIZE_PERC]
[APPLY_CURRENT_CREATURE_VARIATION]
[GO_TO_END]
[SELECT_CASTE:ALL]
[CHANGE_BODY_SIZE_PERC:400700]
[GO_TO_START]
[NAME:giant leopard gecko:giant leopard geckos:giant leopard gecko]
[CASTE_NAME:giant leopard gecko:giant leopard geckos:giant leopard gecko]
[DESCRIPTION:A large monster in the shape of a gecko.]
[POPULATION_NUMBER:10:20]
[CLUSTER_NUMBER:1:1]
[CREATURE_TILE:'G']
[COLOR:6:0:1]
[PETVALUE:500]
[MOUNT_EXOTIC]
[GO_TO_END]
[PREFSTRING:amazing sticky feet]
[PREFSTRING:coloration]
[APPLY_CREATURE_VARIATION:STANDARD_QUADRUPED_GAITS:900:657:438:219:1900:2900] 40 kph
[APPLY_CREATURE_VARIATION:STANDARD_SWIMMING_GAITS:2990:2257:1525:731:4300:6100] 12 kph
[APPLY_CREATURE_VARIATION:STANDARD_CRAWLING_GAITS:2990:2257:1525:731:4300:6100] 12 kph
[APPLY_CREATURE_VARIATION:STANDARD_CLIMBING_GAITS:2990:2257:1525:731:4300:6100] 12 kph[CREATURE:GIANT_LEOPARD_GECKO]
[COPY_TAGS_FROM:GECKO_LEOPARD]
[APPLY_CREATURE_VARIATION:GIANT]
[CV_REMOVE_TAG:CHANGE_BODY_SIZE_PERC]
[APPLY_CURRENT_CREATURE_VARIATION]
[GO_TO_END]
[SELECT_CASTE:ALL]
[CHANGE_BODY_SIZE_PERC:400700]
[GO_TO_START]
[NAME:giant leopard gecko:giant leopard geckos:giant leopard gecko]
[CASTE_NAME:giant leopard gecko:giant leopard geckos:giant leopard gecko]
[DESCRIPTION:A large monster in the shape of a gecko.]
[POPULATION_NUMBER:10:20]
[CLUSTER_NUMBER:1:1]
[CREATURE_TILE:'G']
[COLOR:6:0:1]
[PETVALUE:500]
[MOUNT_EXOTIC]
[GO_TO_END]
[PREFSTRING:amazing sticky feet]
[PREFSTRING:coloration]
[APPLY_CREATURE_VARIATION:STANDARD_QUADRUPED_GAITS:900:657:438:219:1900:2900] 40 kph
[APPLY_CREATURE_VARIATION:STANDARD_SWIMMING_GAITS:2990:2257:1525:731:4300:6100] 12 kph
[APPLY_CREATURE_VARIATION:STANDARD_CRAWLING_GAITS:2990:2257:1525:731:4300:6100] 12 kph
[APPLY_CREATURE_VARIATION:STANDARD_CLIMBING_GAITS:2990:2257:1525:731:4300:6100] 12 kph
[PET][CREATURE:GIANT_LEOPARD_GECKO]
[COPY_TAGS_FROM:GECKO_LEOPARD]
[APPLY_CREATURE_VARIATION:GIANT]
[CV_REMOVE_TAG:CHANGE_BODY_SIZE_PERC]
[APPLY_CURRENT_CREATURE_VARIATION]
[GO_TO_END]
[SELECT_CASTE:ALL]
[CHANGE_BODY_SIZE_PERC:400700]
[GO_TO_START]
[NAME:giant leopard gecko:giant leopard geckos:giant leopard gecko]
[CASTE_NAME:giant leopard gecko:giant leopard geckos:giant leopard gecko]
[DESCRIPTION:A large monster in the shape of a gecko.]
[POPULATION_NUMBER:10:20]
[CLUSTER_NUMBER:1:1]
[CREATURE_TILE:'G']
[COLOR:6:0:1]
[PET_EXOTIC]
[PETVALUE:500]
[MOUNT_EXOTIC]
[GO_TO_END]
[PREFSTRING:amazing sticky feet]
[PREFSTRING:coloration]
[APPLY_CREATURE_VARIATION:STANDARD_QUADRUPED_GAITS:900:657:438:219:1900:2900] 40 kph
[APPLY_CREATURE_VARIATION:STANDARD_SWIMMING_GAITS:2990:2257:1525:731:4300:6100] 12 kph
[APPLY_CREATURE_VARIATION:STANDARD_CRAWLING_GAITS:2990:2257:1525:731:4300:6100] 12 kph
[APPLY_CREATURE_VARIATION:STANDARD_CLIMBING_GAITS:2990:2257:1525:731:4300:6100] 12 kph
[CREATURE:DESERT TORTOISE]
[DESCRIPTION:A tiny shelled reptile that lives in the desert.]
[NAME:desert tortoise:desert tortoises:desert tortoise]
[CASTE_NAME:desert tortoise:desert tortoises:desert tortoise]
[CHILD:1][GENERAL_CHILD_NAME:desert tortoise hatchling:desert tortoise hatchlings]
[CREATURE_TILE:'t'][COLOR:6:0:0]
[PETVALUE:50]
[BENIGN][NATURAL][PET_EXOTIC]
[BIOME:ANY_DESERT]
[LARGE_ROAMING]
[POPULATION_NUMBER:10:30]
[CLUSTER_NUMBER:1:1]
[PREFSTRING:shells]
[PREFSTRING:longevity]
[CANNOT_JUMP]
Diff3 works on windows. I'm on windows. Just include the executable.The first one is the ancestor. If you do it, be sure to try the other two in both orders.
When I get home I'll do your test your merge test case but tell me which is the common ancestor?
Diff3 works on windows. I'm on with does. Just include the executableIf it can be included in the compiled version of the PyLNP, that would be great. See below though, I'm not sure we need it.
Yeah I saw that project. I don't know how good it is. It might be too smart, or it might be too dumb. Thing is, just because two mods can by combined in some predictable way, doesn't mean that it's safe to do so. I didn't really look at what that code does in such cases.Yes, they can be merged by either clever use of a two-way diff, diff3 or Boyer's tool. (though I haven't tested this specifically)
I'll reiterate this test case:<snip>
Can these be merged? does the order or merging effect the result? I posit that it must not be allowed to be merged. Does diff3 merging allow it? Does stephan boyer's code?
>>> (input load order)
$mod0 <GREEN> (because first mod is always OK)
$mod1 <ORANGE> (because validity unknown)
$mod2 cannot be merged, try a different load order. <RED> (merge failed)
>>> (input load order)we could get a similar effect to diff3 by simply applying a diff from vanilla to the mod to the already-merged-mods wrapped in something to catch merge conflicts
I'd probably try for a simple script where you feed it a raw folder and it just returns whether the raws are valid or not; extensible to also check functionality later, and call that on the mixed folder after each mod is added. It can also be used to check that the mods themselves are valid, which would be nice!
UpdateOk, I think that's a problem. putting the pet token on the wrong creature is an incorrect merge.
I see the issue your test case raises.
It replaces the pet token with the new creature vs injecting the newcreature either before/after pet token.
UpdateThere are two problems with that:
I think a lot of the concern with diff3 and it's "block matching" can be alleviated if we parse the raws before we diff3 them. I recommend (if possible) to alphabetize each .txt file's objects, and then parse them to remove whitespace and put tokens on their own individual lines. That way when mod folders are processed in the same manner, the diff between base and the mod will be on an individual token level. I think diff3 is smart enough to catch inserted blocks
I tried the py file you guys are talking about and this latest, I just have no luck with pythonI think the problem is I'm using Python 2.7 (because that's what's installed), and you probably have Python 3.x. But I'll fix those errors for you.Spoiler (click to show/hide)
I'm kinda bummed that your [pet] newcreature situation kind of breaks merging. I would think if the diff program was smart enough to figure out what token it was next to, either before or after token's of the diff. That it could still be added. The fact that the other mod didn't add/remove that specific token bothers me. The diff app should have seen that two mods were bringing different tokens to the same line. So both should have been accomodated by checking the before/after tokens on adjacent lines...I can't think of a way diff3 could be smarter at merging.
Oh well. 1st world problems right.
import os
import difflib
context_lines = 2
if os.path.isfile(mixed_raw_folder+file+'.patch'):
os.remove(mixed_raw_folder+file+'.patch')
for line in difflib.unified_diff(open(vanilla_raw_folder + file).readlines(),
open(mod_raw_folder + file).readlines(), n=context_lines):
with open(mixed_raw_folder+file+'.patch', 'a') as item:
item.write(line)
<Python 3.x compatible version>Well, it no longer freaks out about the print statement :) Unfortunately it also outputs the contents of the vanilla file :(
You can get difflib.Differ.unified_diff() to print out a unified diff, but merging isn't provided in the library.Code: [Select]import os
import difflib
context_lines = 2
if os.path.isfile(mixed_raw_folder+file+'.patch'):
os.remove(mixed_raw_folder+file+'.patch')
for line in difflib.unified_diff(open(vanilla_raw_folder + file).readlines(),
open(mod_raw_folder + file).readlines(), n=context_lines):
with open(mixed_raw_folder+file+'.patch', 'a') as item:
item.write(line)
Creating a unified patch file with a few lines of context (two lines matches within but not between objects) fixes the [pet] issue, but I can't work out how to apply a unified patch with python. Argh.
It should return 1 though. If it returns 1, then the output is garbage; it detected a conflict and gave up. It prints out vanilla because it got to the end of vanilla before finding a conflict. To see it returned 1, "echo $?" imediately after running it. You can run it like this:<Python 3.x compatible version>Well, it no longer freaks out about the print statement :) Unfortunately it also outputs the contents of the vanilla file :(
My testing, though I don't follow the various opcodes, shows that the output_file_temp returned by do_merge_seq() is the same as the contents of the vanilla file.
vanilla_dict={"Creature":{"Giant_leopard_gekko":{all the key/value pairs from vanilla here}}}Mod_1={"Creature":{"Giant_leopard_gekko":{all the key/value pairs from vanilla here,"PET":''}}}Mod_2={"Creature":{"Giant_leopard_gekko":{all the key/value pairs from vanilla here}},
"Desert_tortoise":{all the key/value pairs from new animal here}}So for every key, one could check if the value (i.e. a new dict) is the same or not and if it is not the same, one can do this recursively. A simple 2 dict comparison can be found here (https://stackoverflow.com/questions/1165352/fast-comparison-between-two-python-dictionary)Separating it into two levels like that does solve that particular problem. But you can't just use a dict, because you need to preserve the order of some tags.I haven't read the entire thread, just the last few pages... quite a discussion going on here :)Spoiler (click to show/hide)
What about writing a custom json/xml/whatever style parser that puts these things into (multi-level) dict structures and then comparing the dict structures? This should look like that:Code: (vanilla) [Select]vanilla_dict={"Creature":{"Giant_leopard_gekko":{all the key/value pairs from vanilla here}}}Code: (Pet) [Select]Mod_1={"Creature":{"Giant_leopard_gekko":{all the key/value pairs from vanilla here,"PET":''}}}Code: (Add animal) [Select]Mod_2={"Creature":{"Giant_leopard_gekko":{all the key/value pairs from vanilla here}},So for every key, one could check if the value (i.e. a new dict) is the same or not and if it is not the same, one can do this recursively. A simple 2 dict comparison can be found here (https://stackoverflow.com/questions/1165352/fast-comparison-between-two-python-dictionary)
"Desert_tortoise":{all the key/value pairs from new animal here}}
We thus have some options:and as a final step, one should parse the mixed mod dict into a file.
- Stuff that is unchanged is copied to the mixed mod dict
- Stuff that is simply added (as the [PET] tag or the new creature) will be added to the mixed mod dict
- Stuff that is changed --> check if the same key/value pair is changed in both mods --> yes: problem; no: copy the change to the mixed mod dict
- Stuff that is removed in the mod --> remove from mixed mod dict
you need to preserve the order of some tags.Is there some kind of rule for that? After just briefly scanning some of the files, I haven't seen a "clear" pattern, besides indentation and even that does not seem to be consistent in all cases.
beyond my current skills.Time to learn sth new :)
I'm not a modder, so I don't know the details, but some tags clearly suggest that order matters for them, like [GO_TO_END]. Other tags, like [PET] can be put anywhere after the creature token.you need to preserve the order of some tags.Is there some kind of rule for that? After just briefly scanning some of the files, I haven't seen a "clear" pattern, besides indentation and even that does not seem to be consistent in all cases.
Go for it, and I'll keep writing documentation and design ideas for stuff I can't code yet :PDesign is good. There's a lot of fairly strait forward stuff that needs to be done to manage everything. You need to be able to specify the list of mods. You need to be able to delete the output when merging fails. You probably want to figure out which two mods conflict when merging, which requires extra analysis. And of course the GUI -- designing and stubbing out the GUI can help plan what features you want even if they aren't immediately implemented.
I'm not a modder, so I don't know the details, but some tags clearly suggest that order matters for them, like [GO_TO_END]. Other tags, like [PET] can be put anywhere after the creature token.you need to preserve the order of some tags.Is there some kind of rule for that? After just briefly scanning some of the files, I haven't seen a "clear" pattern, besides indentation and even that does not seem to be consistent in all cases.
There are four kinds of order dependence in the raws, with an example for each at the end.I'm not a modder, so I don't know the details, but some tags clearly suggest that order matters for them, like [GO_TO_END]. Other tags, like [PET] can be put anywhere after the creature token.you need to preserve the order of some tags.Is there some kind of rule for that? After just briefly scanning some of the files, I haven't seen a "clear" pattern, besides indentation and even that does not seem to be consistent in all cases.
[GO_TO_END] is pretty much only there because castes are not declared at the start. Castes and not-creature tokens imbedded in creatures (I.E tissues and materials) are the only thing where positioning matterse.
echo off
REM put tokens on their own line | REM remove tabs | remove all blanklines
for /f %%a in ('dir /b *.txt') do sed -e "s/\[[^][]*\]/\n&\n/g" %%~na.txt | sed -r "s/\t//g" | sed -e "s/^ *//; s/ *$//; /^$/d; s/\r//; /^\s*$/d" > %%~na.out |type %%~na.out
REM cleanup
ren *.out3 *.txt
erase *.out
echo on
What about writing a custom json/xml/whatever style parser that puts these things into (multi-level) dict structures and then comparing the dict structures? This should look like that:
I think we end up with full featured parser. Anyone here with some knowledge about Haskell and Parsec. ;)Good point.
What about writing a custom json/xml/whatever style parser that puts these things into (multi-level) dict structures and then comparing the dict structures? This should look like that:...
What about using attributes for tags like that?
I was messing around with formats for defining legal raw objects of various types. Mainly what I found is that XML isn't great for it, because it doesn't deal gracefully for tags which are allowed in any order. Might be best to define a custom format if we want to go into it that far.
Button: sound great. I thought myself about something like that.
It may be really much better way.
Storing object is not much problem.
But you remember about tag order and the whole [CASTE] thing.
What kind of data structure you use? List with tuple for each token or ordereddict with tuples or ordereddicts as values?
>>> from pyparsing import nestedExpr
>>> txt = "{ { a } { b } { { { c } } } }"
>>>
>>> nestedExpr('{','}').parseString(txt).asList()
[[['a'], ['b'], [[['c']]]]]
>>>
What about writing a custom json/xml/whatever style parser that puts these things into (multi-level) dict structures and then comparing the dict structures? This should look like that:...
Yeah, if it could do caste tokens
and maybe even alphabetize?
You can't store objects in a map/dict when order matters. Use a list or array. That means for creature tokens, because of variations like animal people. Alphabetizing is a problem for the same reason.This generalizes for most raw files... there is a top-level object (like CREATURE or REACTION) that is usually, but not always, self-contained. The biggest exception is that base creatures must exist "earlier in the raws" than any variants of that creature. Raw files of a certain type are parsed in alphabetical order based on the internal name at the first line of the file, and tags within a file are parsed in order of appearance.
I was looking at lexar's...So for a bit of background:
but... I was thinking we need a parser no?
http://pyparsing.wikispaces.com/
found pyparsing
http://stackoverflow.com/questions/1651487/python-parsing-bracketed-blocksCode: [Select]>>> from pyparsing import nestedExpr
>>> txt = "{ { a } { b } { { { c } } } }"
>>>
>>> nestedExpr('{','}').parseString(txt).asList()
[[['a'], ['b'], [[['c']]]]]
>>>
lexer research:
If not
lexar wise I've found:
Pygments
http://pygments.org/docs/lexers/
Ply
http://www.dabeaz.com/ply/
and a big list
https://wiki.python.org/moin/LanguageParsing
You can't store objects in a map/dict when order matters. Use a list or array. That means for creature tokens, because of variations like animal people. Alphabetizing is a problem for the same reason.This generalizes for most raw files... there is a top-level object (like CREATURE or REACTION) that is usually, but not always, self-contained. The biggest exception is that base creatures must exist "earlier in the raws" than any variants of that creature. Raw files of a certain type are parsed in alphabetical order based on the internal name at the first line of the file, and tags within a file are parsed in order of appearance.
You can't store objects in a map/dict when order matters. Use a list or array. That means for creature tokens, because of variations like animal people. Alphabetizing is a problem for the same reason.This generalizes for most raw files... there is a top-level object (like CREATURE or REACTION) that is usually, but not always, self-contained. The biggest exception is that base creatures must exist "earlier in the raws" than any variants of that creature. Raw files of a certain type are parsed in alphabetical order based on the internal name at the first line of the file, and tags within a file are parsed in order of appearance.
Don't worry, King Mir, the dict is just for object lookup. Order is preserved within the object.
I didn't realize that base creatures had to be earlier in the raws than their variations; I'll make a note of that for write-out logic.
Do we really want to auto-alphabetize everything generally? Some things like gems are kinda convenient if sorted by worth. On the other hand, for stockpiles are clearer when things are in alphabetical order.Everything? No.
Agreeably, second level lags who's order doesn't effect the game at all should be put in a dict/map to make it easy to check for changes to the same token.
l = l.strip('[').strip(']').split(':')
token = (l[0], l[1:])
For parsing there are literally tons of tools and libs.You solve this by writing a proper lexer, that converts a string into a list of tokens. Then write a state machine for each kind of token. So in your example the list of tokens starts like this: ['[',KEYWORD="TILE",':','1',':' ....
Even re may be all we need. Only problem I found was stuff like
[TILE:1:3:':':4]
It is just sample. In C I would just parse char by char. In python to.
If not this little uggly things whole parser could be almost in one line likeCode: [Select]l = l.strip('[').strip(']').split(':')
token = (l[0], l[1:])
Damn, i would love to know how you solve this...
Do we really want to auto-alphabetize everything generally? Some things like gems are kinda convenient if sorted by worth. On the other hand, for stockpiles are clearer when things are in alphabetical order.Everything? No.
Agreeably, second level lags who's order doesn't effect the game at all should be put in a dict/map to make it easy to check for changes to the same token.
For the dict/map idea, is there a reason to stop at two levels? For most structures, it'd be sufficient to soak up all subtokens and just keep them with the parent. Castes, however, are a major pain in the ass. Or, they would be if Dwarves had asses (http://www.bay12forums.com/smf/index.php?topic=142739).
The only pseudo-simple solution I can think of is to treat caste declarations and caste selections as milestones within the CREATURE object. A similar tag on the other side of such a milestone is considered a different tag rather than a duplicate one. This will keep the caste-level tags from overwriting each other without requiring an exhaustive list of caste-level tags.
[CREATURE:FANTASY_DRAGON_ANCIENT]
[NAME:ancient dragon:ancient dragons:ancient dragon]
[CREATURE_TILE:255][COLOR:7:0:1]
===== BIOME AND NUMBERS
[BIOME:ANY_LAND]
[APPLY_CREATURE_VARIATION:POP_NUMBERS:1:3:1:1:5]
===== MEGABEAST INFO
[MEGABEAST]
[DIFFICULTY:11]
[ATTACK_TRIGGER:120:50000:1000000]
[SPHERE:FIRE]
[SPHERE:EARTH]
[SPHERE:SKY]
[SPHERE:MOUNTAINS]
[SPHERE:LIGHTNING]
[SPHERE:STORMS]
[SPHERE:RULERSHIP]
===== IMMUNITIES
[APPLY_CREATURE_VARIATION:IMMUNITIES_MEGABEAST]
===== TRAITS
[APPLY_CREATURE_VARIATION:TRAITS_MEGABEAST]
[APPLY_CREATURE_VARIATION:TRAITS_COLOSSAL]
[PETVALUE:15000]
[PET_EXOTIC]
[INTELLIGENT]
[TRAINABLE]
[MOUNT]
[FLIER]
[LARGE_PREDATOR]
[BONECARN]
[ALL_ACTIVE]
[SWIMS_INNATE]
[CHILD:0]
[PREFSTRING:unfathomable power]
===== SIZE, AGE, AND SPEED
[BODY_SIZE:0:0:2000000]
[BODY_SIZE:1000:0:2000000000]
[BODY_SIZE:10000:0:2100000000]
[APPLY_CREATURE_VARIATION:STANDARD_FLYING_GAITS:900:528:352:176:1900:2900] 50 kph
[APPLY_CREATURE_VARIATION:STANDARD_QUADRUPED_GAITS:900:711:521:293:1900:2900] 30 kph
[APPLY_CREATURE_VARIATION:STANDARD_SWIMMING_GAITS:2206:1692:1178:585:3400:4900] 15 kph
[APPLY_CREATURE_VARIATION:STANDARD_CLIMBING_GAITS:2206:1692:1178:585:3400:4900] 15 kph
[APPLY_CREATURE_VARIATION:STANDARD_CRAWLING_GAITS:2206:1692:1178:585:3400:4900] 15 kph
===== SENSES
[VIEWRANGE:#] -- default 20
[LOW_LIGHT_VISION:#] -- highest 10000, default 0
[VISION_ARC:#:#]
[ODOR_LEVEL:#] -- default 50
[SMELL_TRIGGER:#] -- no smell 10000, default 50
===== CREATURE CLASSES
[CREATURE_CLASS:ELEMENTAL_WARD]
[CREATURE_CLASS:MAGIC_RESIST]
[CREATURE_CLASS:MEGABEAST]
[CREATURE_CLASS:DRAGON]
[CREATURE_CLASS:COLOSSAL]
===== LAIR AND HABIT
[LAIR:SHRINE:100]
[LAIR_HUNTER]
[HABIT_NUM:TEST_ALL]
[HABIT:GIANT_NEST:100]
===== BODY
[BODY:DRAGON:ATTACHMENT_HEAD_4HORNS:ATTACHMENT_LIMBS_SPIKESLEG:ATTACHMENT_MISC_SPIKESTAIL]
[BODYGLOSS:CLAW_FOOT]
[BODY_DETAIL_PLAN:MATERIALS_LEVEL_9]
[REMOVE_MATERIAL:HAIR]
[USE_MATERIAL_TEMPLATE:SCALE:SCALE_TEMPLATE_LEVEL_9]
[USE_MATERIAL_TEMPLATE:TALON:NAIL_TEMPLATE_LEVEL_5]
[USE_MATERIAL_TEMPLATE:HORN:HORN_TEMPLATE_LEVEL_5]
[USE_MATERIAL_TEMPLATE:SPIKE:SPIKE_TEMPLATE_LEVEL_5]
[BODY_DETAIL_PLAN:TISSUES_LEVEL_9]
[REMOVE_TISSUE:HAIR]
[USE_TISSUE_TEMPLATE:SCALE:SCALE_TEMPLATE_LEVEL_9]
[USE_TISSUE_TEMPLATE:TALON:CLAW_TEMPLATE_LEVEL_5]
[USE_TISSUE_TEMPLATE:HORN:HORN_TEMPLATE_LEVEL_5]
[USE_TISSUE_TEMPLATE:SPIKE:SPIKE_TEMPLATE_LEVEL_5]
[BODY_DETAIL_PLAN:VERTEBRATE_TISSUE_LAYERS:SCALE:FAT:MUSCLE:BONE:CARTILAGE]
[TISSUE_LAYER:BY_CATEGORY:TOE:TALON:FRONT]
[SELECT_TISSUE_LAYER:HEART:BY_CATEGORY:HEART]
[PLUS_TISSUE_LAYER:SCALE:BY_CATEGORY:THROAT]
[TL_MAJOR_ARTERIES]
[BODY_DETAIL_PLAN:STANDARD_HEAD_POSITIONS]
[BODY_DETAIL_PLAN:HUMANOID_RIBCAGE_POSITIONS]
[USE_MATERIAL_TEMPLATE:SINEW:SINEW_TEMPLATE]
[TENDONS:LOCAL_CREATURE_MAT:SINEW:200]
[LIGAMENTS:LOCAL_CREATURE_MAT:SINEW:200]
[USE_MATERIAL_TEMPLATE:BLOOD:BLOOD_TEMPLATE]
[BLOOD:LOCAL_CREATURE_MAT:BLOOD:LIQUID]
[USE_MATERIAL_TEMPLATE:PUS:PUS_TEMPLATE]
[PUS:LOCAL_CREATURE_MAT:PUS:LIQUID]
[HAS_NERVES]
[GETS_WOUND_INFECTIONS]
[GETS_INFECTIONS_FROM_ROT]
[SELECT_TISSUE:SCALE]
[RELATIVE_THICKNESS:8]
[BODY_APPEARANCE_MODIFIER:LENGTH:90:95:98:100:102:105:110]
[BODY_APPEARANCE_MODIFIER:HEIGHT:90:95:98:100:102:105:110]
[BODY_APPEARANCE_MODIFIER:BROADNESS:90:95:98:100:102:105:110]
===== ATTRIBUTES AND SKILLS
[PHYS_ATT_RANGE:STRENGTH:4000:4100:4200:4500:4800:4900:5000]
[PHYS_ATT_RANGE:AGILITY:3000:3100:3200:3500:3800:3900:4000]
[PHYS_ATT_RANGE:TOUGHNESS:3500:3600:3700:4000:4300:4400:4500]
[PHYS_ATT_RANGE:ENDURANCE:3500:3600:3700:4000:4300:4400:4500]
[PHYS_ATT_RANGE:DISEASE_RESISTANCE:4000:4100:4200:4500:4800:4900:5000]
[PHYS_ATT_RANGE:RECUPERATION:3000:3100:3200:3500:3800:3900:4000]
[MENT_ATT_RANGE:WILLPOWER:5000:5000:5000:5000:5000:5000:5000]
[MENT_ATT_RANGE:SPATIAL_SENSE:3500:3600:3700:4000:4300:4400:4500]
[MENT_ATT_RANGE:KINESTHETIC_SENSE:5000:5000:5000:5000:5000:5000:5000]
[MENT_ATT_RANGE:FOCUS:5000:5000:5000:5000:5000:5000:5000]
[NATURAL_SKILL:BITE:12]
[NATURAL_SKILL:GRASP_STRIKE:12]
[NATURAL_SKILL:STANCE_STRIKE:12]
[NATURAL_SKILL:MELEE_COMBAT:12]
[NATURAL_SKILL:SITUATIONAL_AWARENESS:12]
[NATURAL_SKILL:DISCIPLINE:12]
===== ATTACKS
[APPLY_CREATURE_VARIATION:ATTACK_BITE]
[APPLY_CREATURE_VARIATION:ATTACK_TAIL_SWIPE]
[APPLY_CREATURE_VARIATION:ATTACK_CLAW]
[APPLY_CREATURE_VARIATION:ATTACK_HORN]
[APPLY_CREATURE_VARIATION:ATTACK_SPIKE]
[APPLY_CREATURE_VARIATION:SPECIAL_ATTACK_WING_BUFFET_COLOSSAL]
[APPLY_CREATURE_VARIATION:SPECIAL_ATTACK_STOMP_COLOSSAL]
[APPLY_CREATURE_VARIATION:SPECIAL_ATTACK_TAIL_SWEEP_COLOSSAL]
[APPLY_CREATURE_VARIATION:SPECIAL_ATTACK_CLEAVE_CLAW_COLOSSAL]
[APPLY_CREATURE_VARTIAION:SPECIAL_ATTACK_IMPALE_HORN_COLOSSAL]
===== CASTES
[CASTE:FEMALE_RED]
[FEMALE]
[LAYS_EGGS]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGGSHELL:SOLID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_WHITE:LIQUID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_YOLK:LIQUID]
[EGG_SIZE:400000]
[CLUTCH_SIZE:1:2]
[CASTE:MALE_RED]
[MALE]
[CASTE:FEMALE_BLUE]
[FEMALE]
[LAYS_EGGS]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGGSHELL:SOLID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_WHITE:LIQUID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_YOLK:LIQUID]
[EGG_SIZE:400000]
[CLUTCH_SIZE:1:2]
[CASTE:MALE_BLUE]
[MALE]
[CASTE:FEMALE_WHITE]
[FEMALE]
[LAYS_EGGS]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGGSHELL:SOLID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_WHITE:LIQUID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_YOLK:LIQUID]
[EGG_SIZE:400000]
[CLUTCH_SIZE:1:2]
[CASTE:MALE_WHITE]
[MALE]
[CASTE:FEMALE_BROWN]
[FEMALE]
[LAYS_EGGS]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGGSHELL:SOLID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_WHITE:LIQUID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_YOLK:LIQUID]
[EGG_SIZE:400000]
[CLUTCH_SIZE:1:2]
[CASTE:MALE_BROWN]
[MALE]
===== CASTE DETAILS
[SELECT_CASTE:FEMALE_RED]
[SELECT_ADDITIONAL_CASTE:MALE_RED]
[DESCRIPTION:Very few ancient dragons exist today. These dragons are the original members of the elder race that has gone all but extinct. Those that do still exist are the ones that were in the middle of the breach when it closed. It is said that half of the mind of these once noble creatures was left in the other realm and now they are little more than intelligent beasts who rule the skies. Red - The red dragons are masters of fire.]
[CASTE_NAME:ancient red dragon:ancient red dragons:ancient red dragon]
[FIREIMMUNE_SUPER]
===== CASTE CLASSES
[CREATURE_CLASS:FIRE4]
===== CASTE INTERACTIONS
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_FIRE_FIRE_JET]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_FIRE_FIREBALL]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_FIRE_FLAME_BLAST]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_FIRE_INCINERATE]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_FIRE_DRAGON_FIRE]
===== CASTE TISSUE LAYER GROUPS
[SET_TL_GROUP:BY_CATEGORY:ALL:SCALE]
[TL_COLOR_MODIFIER:RED:1:CRIMSON:1:CARMINE:1]
[TLCM_NOUN:scales:PLURAL]
[SET_TL_GROUP:BY_CATEGORY:EYE:EYE]
[TL_COLOR_MODIFIER:BLACK:1]
[TLCM_NOUN:eyes:PLURAL]
===== CASTE DETAILS
[SELECT_CASTE:FEMALE_BLUE]
[SELECT_ADDITIONAL_CASTE:MALE_BLUE]
[DESCRIPTION:Very few ancient dragons exist today. These dragons are the original members of the elder race that has gone all but extinct. Those that do still exist are the ones that were in the middle of the breach when it closed. It is said that half of the mind of these once noble creatures was left in the other realm and now they are little more than intelligent beasts who rule the skies. Blue - The blue dragons are masters of ice.]
[CASTE_NAME:ancient blue dragon:ancient blue dragons:ancient blue dragon]
===== CASTE CLASSES
[CREATURE_CLASS:ICE4]
===== CASTE INTERACTIONS
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_ICE_ICICLE]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_ICE_COLD_SNAP]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_ICE_RAY_OF_FROST]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_ICE_FREEZE]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_ICE_FROST_NOVA]
===== CASTE TISSUE LAYER GROUPS
[SET_TL_GROUP:BY_CATEGORY:ALL:SCALE]
[TL_COLOR_MODIFIER:AZURE:1:AQUA:1:COBALT:1:CERULEAN:1:MIDNIGHT_BLUE:1]
[TLCM_NOUN:scales:PLURAL]
[SET_TL_GROUP:BY_CATEGORY:EYE:EYE]
[TL_COLOR_MODIFIER:BLACK:1]
[TLCM_NOUN:eyes:PLURAL]
===== CASTE DETAILS
[SELECT_CASTE:FEMALE_WHITE]
[SELECT_ADDITIONAL_CASTE:MALE_WHITE]
[DESCRIPTION:Very few ancient dragons exist today. These dragons are the original members of the elder race that has gone all but extinct. Those that do still exist are the ones that were in the middle of the breach when it closed. It is said that half of the mind of these once noble creatures was left in the other realm and now they are little more than intelligent beasts who rule the skies. White - The white dragons are masters of air]
[CASTE_NAME:ancient white dragon:ancient white dragons:ancient white dragon]
===== CASTE CLASSES
[CREATURE_CLASS:AIR4]
===== CASTE INTERACTIONS
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_AIR_LIGHTNING]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_AIR_CHAIN_LIGHTNING]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_AIR_SHOCK]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_AIR_GUST]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_AIR_CYCLONE]
===== CASTE TISSUE LAYER GROUPS
[SET_TL_GROUP:BY_CATEGORY:ALL:SCALE]
[TL_COLOR_MODIFIER:WHITE:1:IVORY:1]
[TLCM_NOUN:scales:PLURAL]
[SET_TL_GROUP:BY_CATEGORY:EYE:EYE]
[TL_COLOR_MODIFIER:BLACK:1]
[TLCM_NOUN:eyes:PLURAL]
===== CASTE DETAILS
[SELECT_CASTE:FEMALE_BROWN]
[SELECT_ADDITIONAL_CASTE:MALE_BROWN]
[DESCRIPTION:Very few ancient dragons exist today. These dragons are the original members of the elder race that has gone all but extinct. Those that do still exist are the ones that were in the middle of the breach when it closed. It is said that half of the mind of these once noble creatures was left in the other realm and now they are little more than intelligent beasts who rule the skies. Brown - The brown dragons are masters of earth.]
[CASTE_NAME:ancient brown dragon:ancient brown dragons:ancient brown dragon]
===== CASTE CLASSES
[CREATURE_CLASS:EARTH4]
===== CASTE INTERACTIONS
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_EARTH_BOULDER]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_EARTH_STALAGMITE]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_EARTH_SAND_STORM]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_EARTH_PETRIFY]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_EARTH_EARTHQUAKE]
===== CASTE TISSUE LAYER GROUPS
[SET_TL_GROUP:BY_CATEGORY:ALL:SCALE]
[TL_COLOR_MODIFIER:BROWN:1:DARK_BROWN:1:LIGHT_BROWN:1]
[TLCM_NOUN:scales:PLURAL]
[SET_TL_GROUP:BY_CATEGORY:EYE:EYE]
[TL_COLOR_MODIFIER:BLACK:1]
[TLCM_NOUN:eyes:PLURAL]
===== MISC
[SELECT_CASTE:ALL]
[SELECT_MATERIAL:ALL]
[MULTIPLY_VALUE:25]
[COLDDAM_POINT:NONE]
[HEATDAM_POINT:NONE]
[IGNITE_POINT:NONE]
[IF_EXISTS_SET_MELTING_POINT:55000]
[IF_EXISTS_SET_BOILING_POINT:57000]
[SPEC_HEAT:30000]
[SELECT_MATERIAL:BLOOD]
[PLUS_MATERIAL:PUS]
[MELTING_POINT:10000]
[EXTRA_BUTCHER_OBJECT:BY_TOKEN:HEART]
[EBO_ITEM:TOOL:ITEM_TOOL_CREATURE_HEART_DRAGON_ANCIENT:NONE:NONE]
[EXTRA_BUTCHER_OBJECT:BY_TOKEN:BRAIN]
[EBO_ITEM:SMALLGEM:NONE:INORGANIC:SOUL_GEM_DRAGON_ANCIENT]
That would be my suggestion.
Honestly if I were working on this I would start with a program that just divided the raws into dictionaries that would be easy to check against eachother. For example:Viola, now you have a dictionary that you can just compare to another dictionary and if it has extra entries or less entries you can adjust your output accordingly. An advantage to this is you can also specify exactly how you want the output to look, so you could, for instance, output into a broken up structure that is more easy to read like
- Create object type dictionary (e.g. CREATURE)
- Create sub dictionary for each creature (e.g. TOAD)
- Read raws from top to bottom, placing each entry into the dictionary for specific creature (e.g. MAXAGE:70:100)
- If you hit a certain token (like USE_MATERIAL_TEMPLATE, SELECT_TISSUE_LAYER, or, the big one, CASTE) create a sub-sub dictionary for that tag
- Continue adding to the sub-sub dictionary until you hit a token that exits out of it (e.g. another CASTE or USE_MATERIAL_TEMPLATE etc...)
Code: [Select][CREATURE:FANTASY_DRAGON_ANCIENT]That would be my suggestion.
[NAME:ancient dragon:ancient dragons:ancient dragon]
[CREATURE_TILE:255][COLOR:7:0:1]
===== BIOME AND NUMBERS
[BIOME:ANY_LAND]
[APPLY_CREATURE_VARIATION:POP_NUMBERS:1:3:1:1:5]
===== MEGABEAST INFO
[MEGABEAST]
[DIFFICULTY:11]
[ATTACK_TRIGGER:120:50000:1000000]
[SPHERE:FIRE]
[SPHERE:EARTH]
[SPHERE:SKY]
[SPHERE:MOUNTAINS]
[SPHERE:LIGHTNING]
[SPHERE:STORMS]
[SPHERE:RULERSHIP]
===== IMMUNITIES
[APPLY_CREATURE_VARIATION:IMMUNITIES_MEGABEAST]
===== TRAITS
[APPLY_CREATURE_VARIATION:TRAITS_MEGABEAST]
[APPLY_CREATURE_VARIATION:TRAITS_COLOSSAL]
[PETVALUE:15000]
[PET_EXOTIC]
[INTELLIGENT]
[TRAINABLE]
[MOUNT]
[FLIER]
[LARGE_PREDATOR]
[BONECARN]
[ALL_ACTIVE]
[SWIMS_INNATE]
[CHILD:0]
[PREFSTRING:unfathomable power]
===== SIZE, AGE, AND SPEED
[BODY_SIZE:0:0:2000000]
[BODY_SIZE:1000:0:2000000000]
[BODY_SIZE:10000:0:2100000000]
[APPLY_CREATURE_VARIATION:STANDARD_FLYING_GAITS:900:528:352:176:1900:2900] 50 kph
[APPLY_CREATURE_VARIATION:STANDARD_QUADRUPED_GAITS:900:711:521:293:1900:2900] 30 kph
[APPLY_CREATURE_VARIATION:STANDARD_SWIMMING_GAITS:2206:1692:1178:585:3400:4900] 15 kph
[APPLY_CREATURE_VARIATION:STANDARD_CLIMBING_GAITS:2206:1692:1178:585:3400:4900] 15 kph
[APPLY_CREATURE_VARIATION:STANDARD_CRAWLING_GAITS:2206:1692:1178:585:3400:4900] 15 kph
===== SENSES
[VIEWRANGE:#] -- default 20
[LOW_LIGHT_VISION:#] -- highest 10000, default 0
[VISION_ARC:#:#]
[ODOR_LEVEL:#] -- default 50
[SMELL_TRIGGER:#] -- no smell 10000, default 50
===== CREATURE CLASSES
[CREATURE_CLASS:ELEMENTAL_WARD]
[CREATURE_CLASS:MAGIC_RESIST]
[CREATURE_CLASS:MEGABEAST]
[CREATURE_CLASS:DRAGON]
[CREATURE_CLASS:COLOSSAL]
===== LAIR AND HABIT
[LAIR:SHRINE:100]
[LAIR_HUNTER]
[HABIT_NUM:TEST_ALL]
[HABIT:GIANT_NEST:100]
===== BODY
[BODY:DRAGON:ATTACHMENT_HEAD_4HORNS:ATTACHMENT_LIMBS_SPIKESLEG:ATTACHMENT_MISC_SPIKESTAIL]
[BODYGLOSS:CLAW_FOOT]
[BODY_DETAIL_PLAN:MATERIALS_LEVEL_9]
[REMOVE_MATERIAL:HAIR]
[USE_MATERIAL_TEMPLATE:SCALE:SCALE_TEMPLATE_LEVEL_9]
[USE_MATERIAL_TEMPLATE:TALON:NAIL_TEMPLATE_LEVEL_5]
[USE_MATERIAL_TEMPLATE:HORN:HORN_TEMPLATE_LEVEL_5]
[USE_MATERIAL_TEMPLATE:SPIKE:SPIKE_TEMPLATE_LEVEL_5]
[BODY_DETAIL_PLAN:TISSUES_LEVEL_9]
[REMOVE_TISSUE:HAIR]
[USE_TISSUE_TEMPLATE:SCALE:SCALE_TEMPLATE_LEVEL_9]
[USE_TISSUE_TEMPLATE:TALON:CLAW_TEMPLATE_LEVEL_5]
[USE_TISSUE_TEMPLATE:HORN:HORN_TEMPLATE_LEVEL_5]
[USE_TISSUE_TEMPLATE:SPIKE:SPIKE_TEMPLATE_LEVEL_5]
[BODY_DETAIL_PLAN:VERTEBRATE_TISSUE_LAYERS:SCALE:FAT:MUSCLE:BONE:CARTILAGE]
[TISSUE_LAYER:BY_CATEGORY:TOE:TALON:FRONT]
[SELECT_TISSUE_LAYER:HEART:BY_CATEGORY:HEART]
[PLUS_TISSUE_LAYER:SCALE:BY_CATEGORY:THROAT]
[TL_MAJOR_ARTERIES]
[BODY_DETAIL_PLAN:STANDARD_HEAD_POSITIONS]
[BODY_DETAIL_PLAN:HUMANOID_RIBCAGE_POSITIONS]
[USE_MATERIAL_TEMPLATE:SINEW:SINEW_TEMPLATE]
[TENDONS:LOCAL_CREATURE_MAT:SINEW:200]
[LIGAMENTS:LOCAL_CREATURE_MAT:SINEW:200]
[USE_MATERIAL_TEMPLATE:BLOOD:BLOOD_TEMPLATE]
[BLOOD:LOCAL_CREATURE_MAT:BLOOD:LIQUID]
[USE_MATERIAL_TEMPLATE:PUS:PUS_TEMPLATE]
[PUS:LOCAL_CREATURE_MAT:PUS:LIQUID]
[HAS_NERVES]
[GETS_WOUND_INFECTIONS]
[GETS_INFECTIONS_FROM_ROT]
[SELECT_TISSUE:SCALE]
[RELATIVE_THICKNESS:8]
[BODY_APPEARANCE_MODIFIER:LENGTH:90:95:98:100:102:105:110]
[BODY_APPEARANCE_MODIFIER:HEIGHT:90:95:98:100:102:105:110]
[BODY_APPEARANCE_MODIFIER:BROADNESS:90:95:98:100:102:105:110]
===== ATTRIBUTES AND SKILLS
[PHYS_ATT_RANGE:STRENGTH:4000:4100:4200:4500:4800:4900:5000]
[PHYS_ATT_RANGE:AGILITY:3000:3100:3200:3500:3800:3900:4000]
[PHYS_ATT_RANGE:TOUGHNESS:3500:3600:3700:4000:4300:4400:4500]
[PHYS_ATT_RANGE:ENDURANCE:3500:3600:3700:4000:4300:4400:4500]
[PHYS_ATT_RANGE:DISEASE_RESISTANCE:4000:4100:4200:4500:4800:4900:5000]
[PHYS_ATT_RANGE:RECUPERATION:3000:3100:3200:3500:3800:3900:4000]
[MENT_ATT_RANGE:WILLPOWER:5000:5000:5000:5000:5000:5000:5000]
[MENT_ATT_RANGE:SPATIAL_SENSE:3500:3600:3700:4000:4300:4400:4500]
[MENT_ATT_RANGE:KINESTHETIC_SENSE:5000:5000:5000:5000:5000:5000:5000]
[MENT_ATT_RANGE:FOCUS:5000:5000:5000:5000:5000:5000:5000]
[NATURAL_SKILL:BITE:12]
[NATURAL_SKILL:GRASP_STRIKE:12]
[NATURAL_SKILL:STANCE_STRIKE:12]
[NATURAL_SKILL:MELEE_COMBAT:12]
[NATURAL_SKILL:SITUATIONAL_AWARENESS:12]
[NATURAL_SKILL:DISCIPLINE:12]
===== ATTACKS
[APPLY_CREATURE_VARIATION:ATTACK_BITE]
[APPLY_CREATURE_VARIATION:ATTACK_TAIL_SWIPE]
[APPLY_CREATURE_VARIATION:ATTACK_CLAW]
[APPLY_CREATURE_VARIATION:ATTACK_HORN]
[APPLY_CREATURE_VARIATION:ATTACK_SPIKE]
[APPLY_CREATURE_VARIATION:SPECIAL_ATTACK_WING_BUFFET_COLOSSAL]
[APPLY_CREATURE_VARIATION:SPECIAL_ATTACK_STOMP_COLOSSAL]
[APPLY_CREATURE_VARIATION:SPECIAL_ATTACK_TAIL_SWEEP_COLOSSAL]
[APPLY_CREATURE_VARIATION:SPECIAL_ATTACK_CLEAVE_CLAW_COLOSSAL]
[APPLY_CREATURE_VARTIAION:SPECIAL_ATTACK_IMPALE_HORN_COLOSSAL]
===== CASTES
[CASTE:FEMALE_RED]
[FEMALE]
[LAYS_EGGS]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGGSHELL:SOLID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_WHITE:LIQUID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_YOLK:LIQUID]
[EGG_SIZE:400000]
[CLUTCH_SIZE:1:2]
[CASTE:MALE_RED]
[MALE]
[CASTE:FEMALE_BLUE]
[FEMALE]
[LAYS_EGGS]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGGSHELL:SOLID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_WHITE:LIQUID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_YOLK:LIQUID]
[EGG_SIZE:400000]
[CLUTCH_SIZE:1:2]
[CASTE:MALE_BLUE]
[MALE]
[CASTE:FEMALE_WHITE]
[FEMALE]
[LAYS_EGGS]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGGSHELL:SOLID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_WHITE:LIQUID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_YOLK:LIQUID]
[EGG_SIZE:400000]
[CLUTCH_SIZE:1:2]
[CASTE:MALE_WHITE]
[MALE]
[CASTE:FEMALE_BROWN]
[FEMALE]
[LAYS_EGGS]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGGSHELL:SOLID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_WHITE:LIQUID]
[EGG_MATERIAL:LOCAL_CREATURE_MAT:EGG_YOLK:LIQUID]
[EGG_SIZE:400000]
[CLUTCH_SIZE:1:2]
[CASTE:MALE_BROWN]
[MALE]
===== CASTE DETAILS
[SELECT_CASTE:FEMALE_RED]
[SELECT_ADDITIONAL_CASTE:MALE_RED]
[DESCRIPTION:Very few ancient dragons exist today. These dragons are the original members of the elder race that has gone all but extinct. Those that do still exist are the ones that were in the middle of the breach when it closed. It is said that half of the mind of these once noble creatures was left in the other realm and now they are little more than intelligent beasts who rule the skies. Red - The red dragons are masters of fire.]
[CASTE_NAME:ancient red dragon:ancient red dragons:ancient red dragon]
[FIREIMMUNE_SUPER]
===== CASTE CLASSES
[CREATURE_CLASS:FIRE4]
===== CASTE INTERACTIONS
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_FIRE_FIRE_JET]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_FIRE_FIREBALL]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_FIRE_FLAME_BLAST]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_FIRE_INCINERATE]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_FIRE_DRAGON_FIRE]
===== CASTE TISSUE LAYER GROUPS
[SET_TL_GROUP:BY_CATEGORY:ALL:SCALE]
[TL_COLOR_MODIFIER:RED:1:CRIMSON:1:CARMINE:1]
[TLCM_NOUN:scales:PLURAL]
[SET_TL_GROUP:BY_CATEGORY:EYE:EYE]
[TL_COLOR_MODIFIER:BLACK:1]
[TLCM_NOUN:eyes:PLURAL]
===== CASTE DETAILS
[SELECT_CASTE:FEMALE_BLUE]
[SELECT_ADDITIONAL_CASTE:MALE_BLUE]
[DESCRIPTION:Very few ancient dragons exist today. These dragons are the original members of the elder race that has gone all but extinct. Those that do still exist are the ones that were in the middle of the breach when it closed. It is said that half of the mind of these once noble creatures was left in the other realm and now they are little more than intelligent beasts who rule the skies. Blue - The blue dragons are masters of ice.]
[CASTE_NAME:ancient blue dragon:ancient blue dragons:ancient blue dragon]
===== CASTE CLASSES
[CREATURE_CLASS:ICE4]
===== CASTE INTERACTIONS
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_ICE_ICICLE]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_ICE_COLD_SNAP]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_ICE_RAY_OF_FROST]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_ICE_FREEZE]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_ICE_FROST_NOVA]
===== CASTE TISSUE LAYER GROUPS
[SET_TL_GROUP:BY_CATEGORY:ALL:SCALE]
[TL_COLOR_MODIFIER:AZURE:1:AQUA:1:COBALT:1:CERULEAN:1:MIDNIGHT_BLUE:1]
[TLCM_NOUN:scales:PLURAL]
[SET_TL_GROUP:BY_CATEGORY:EYE:EYE]
[TL_COLOR_MODIFIER:BLACK:1]
[TLCM_NOUN:eyes:PLURAL]
===== CASTE DETAILS
[SELECT_CASTE:FEMALE_WHITE]
[SELECT_ADDITIONAL_CASTE:MALE_WHITE]
[DESCRIPTION:Very few ancient dragons exist today. These dragons are the original members of the elder race that has gone all but extinct. Those that do still exist are the ones that were in the middle of the breach when it closed. It is said that half of the mind of these once noble creatures was left in the other realm and now they are little more than intelligent beasts who rule the skies. White - The white dragons are masters of air]
[CASTE_NAME:ancient white dragon:ancient white dragons:ancient white dragon]
===== CASTE CLASSES
[CREATURE_CLASS:AIR4]
===== CASTE INTERACTIONS
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_AIR_LIGHTNING]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_AIR_CHAIN_LIGHTNING]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_AIR_SHOCK]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_AIR_GUST]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_AIR_CYCLONE]
===== CASTE TISSUE LAYER GROUPS
[SET_TL_GROUP:BY_CATEGORY:ALL:SCALE]
[TL_COLOR_MODIFIER:WHITE:1:IVORY:1]
[TLCM_NOUN:scales:PLURAL]
[SET_TL_GROUP:BY_CATEGORY:EYE:EYE]
[TL_COLOR_MODIFIER:BLACK:1]
[TLCM_NOUN:eyes:PLURAL]
===== CASTE DETAILS
[SELECT_CASTE:FEMALE_BROWN]
[SELECT_ADDITIONAL_CASTE:MALE_BROWN]
[DESCRIPTION:Very few ancient dragons exist today. These dragons are the original members of the elder race that has gone all but extinct. Those that do still exist are the ones that were in the middle of the breach when it closed. It is said that half of the mind of these once noble creatures was left in the other realm and now they are little more than intelligent beasts who rule the skies. Brown - The brown dragons are masters of earth.]
[CASTE_NAME:ancient brown dragon:ancient brown dragons:ancient brown dragon]
===== CASTE CLASSES
[CREATURE_CLASS:EARTH4]
===== CASTE INTERACTIONS
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_EARTH_BOULDER]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_EARTH_STALAGMITE]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_EARTH_SAND_STORM]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_EARTH_PETRIFY]
[APPLY_CREATURE_VARIATION:SPELL_ELEMENTAL_EARTH_EARTHQUAKE]
===== CASTE TISSUE LAYER GROUPS
[SET_TL_GROUP:BY_CATEGORY:ALL:SCALE]
[TL_COLOR_MODIFIER:BROWN:1:DARK_BROWN:1:LIGHT_BROWN:1]
[TLCM_NOUN:scales:PLURAL]
[SET_TL_GROUP:BY_CATEGORY:EYE:EYE]
[TL_COLOR_MODIFIER:BLACK:1]
[TLCM_NOUN:eyes:PLURAL]
===== MISC
[SELECT_CASTE:ALL]
[SELECT_MATERIAL:ALL]
[MULTIPLY_VALUE:25]
[COLDDAM_POINT:NONE]
[HEATDAM_POINT:NONE]
[IGNITE_POINT:NONE]
[IF_EXISTS_SET_MELTING_POINT:55000]
[IF_EXISTS_SET_BOILING_POINT:57000]
[SPEC_HEAT:30000]
[SELECT_MATERIAL:BLOOD]
[PLUS_MATERIAL:PUS]
[MELTING_POINT:10000]
[EXTRA_BUTCHER_OBJECT:BY_TOKEN:HEART]
[EBO_ITEM:TOOL:ITEM_TOOL_CREATURE_HEART_DRAGON_ANCIENT:NONE:NONE]
[EXTRA_BUTCHER_OBJECT:BY_TOKEN:BRAIN]
[EBO_ITEM:SMALLGEM:NONE:INORGANIC:SOUL_GEM_DRAGON_ANCIENT]
I have nothing at all against the idea of scope creep, more advanced parsing would be great.Honestly if I were working on this I would start with a program that just divided the raws into dictionaries that would be easy to check against each other. For example:I think the technical name for that approach is "scope creep." This sounds like a decent avenue to pursue, but not for a first version. False-positive rejects are fine since the intended audience is newbs who will probably only load mods pre-loaded with the Starter Pack. That handful can have their compatibility worked out in manifest files if absolutely necessary.Viola, now you have a dictionary that you can just compare to another dictionary and if it has extra entries or less entries you can adjust your output accordingly. An advantage to this is you can also specify exactly how you want the output to look, so you could, for instance, output into a broken up structure that is more easy to read like
- Create object type dictionary (e.g. CREATURE)
- Create sub dictionary for each creature (e.g. TOAD)
- Read raws from top to bottom, placing each entry into the dictionary for specific creature (e.g. MAXAGE:70:100)
- If you hit a certain token (like USE_MATERIAL_TEMPLATE, SELECT_TISSUE_LAYER, or, the big one, CASTE) create a sub-sub dictionary for that tag
- Continue adding to the sub-sub dictionary until you hit a token that exits out of it (e.g. another CASTE or USE_MATERIAL_TEMPLATE etc...)
Once the idea of merge-in mods takes off, more intelligent parsing will be far more valuable. The important thing for now is to make sure we don't make design decisions that tie our hands down the road.
I should do some sketches soon to show what I have in mind. If anyone wnats to get started let me know and I'll prioritise them.3. GUI-ify as tab in a fork of the PyLNP.3) Definitely up there. If there's somebody reading this thread looking to contribute, this would be a good way to help.
BSTART = '['
BEND = ']'
STR = ASCII+
TYPE = STR
TILE = "'" STR "'"
ARG = SEP STR|TILE
TOKEN = BSTART TYPE ARG* BEND
So one of the goals of this is that mods that conflict are marked as conflicting, not silently resolved to favor one. Favoring one may be suitable for some merging applications, like making your own mod bundle, but for someone trying out a batch of mods, having the order the mods are loaded in not matter is a good thing.I think the merger should raise a warning if there is any overlap, but allow the user to proceed with some simple tiebreaker rules (either a priority list, or a load order with last-in-wins).
So I haven't read all the comments since Friday because it is so late it's early but I just remembered I had yet to upload this so here's the mod loader guys http://dffd.wimbli.com/file.php?id=9509
If you just run it off the bat it'll give errors on the plant files, the raw files need to be flattened out because it gets confused by comments being on the same line as a tag. It's just the load-in not the write-out yet but the write-out should be fairly obvious once you take a look at it.
So I haven't read all the comments since Friday because it is so late it's early but I just remembered I had yet to upload this so here's the mod loader guys http://dffd.wimbli.com/file.php?id=9509
If you just run it off the bat it'll give errors on the plant files, the raw files need to be flattened out because it gets confused by comments being on the same line as a tag. It's just the load-in not the write-out yet but the write-out should be fairly obvious once you take a look at it.
Nice! Unfortunately it doesn't seem to like the plant_crops, plant_garden, or plant_new_trees files - I got eleven thousand lines of complaining about them! :o Should be easy enough to fix, I imagine, and I look forward to seeing a full merge based on this (or any other) concept.
from pyparsing import Word, ZeroOrMore, Suppress, Optional, Regex
from pyparsing import alphas, alphanums
from pyparsing import ParseException
lbracket = Suppress('[')
rbracket = Suppress(']')
tile = Regex("\'.\'")
arg = Suppress(':') + (Word(alphanums) ^ tile)
self.token = lbracket + Word(alphas) + Optional(ZeroOrMore(arg)) + rbracket
I see your python script... And fold.Yeah, it looks like we should skip any flattening of the files in the text folder, since they are all one-entry-per-line already and the "comments" are actually literal text. But standard diffs should be able to handle those files once they survive the pre-processing stage.
:)
Maybe button is having an issue w parsing files where tokens and comments NEED to be in the same line. Such as the text subfolder and some items in the data subfolder?
Maybe button is having an issue w parsing files where tokens and comments NEED to be on the same line. Such as the text subfolder and some items in the data subfolder?
No flattening, I just ran the code as it came.
I hope I haven't killed this thread/discouraged anybody from working on it. As you may have noticed I don't have a ton of time to work on this myself so please don't think that I've claimed the project or anything."I don't have a ton of time to work on this" basically sums up my position, except I'm also bumping up against the limits of my programming ability. My sneaky plan is to wait until my semester course covers something relevant, and then retry - but if someone else finishes first I'm not going to be sad!
I know everyone is talking about automation and the best script methods to merge mods together, but why not have one person do it by hand? I have done it dozens of times, and its quite easy if you take minor mods and vanilla DF as base. Since you say that you don't even want to try MDF + Genesis + LfR or something similar, it might be that the entire process of writing the code to merge mods is much more work than merging them by hand.
I mean... this thread is over 2 weeks old. In that time I could have merged 100 minor mods into a modpack, using simple on/off toggles on the [OBJECT:X] line at the top, and using specific filenames, with only the current visual-basics based LNP/Starter Pack GUI as base. And I dont have coding knowledge, I would copy+paste the AQUIFER button you have for this.
I dont mean to intrude, but I have the feeling that you are overcomplicating stuff. Although I have to add that new additions would require recompiling the launcher. Splinterz would probably find a way around that, even using the system I described, by using a dropdown menu and an external txt that modders could add their mods to, he did the same for MDF.
Suggestion: Add a new tab to your normal DF Starter Pack, with 10 very popular minor mods, and wait for peoples feedback? That's what I would do.
Yes its ugly. And primitive. But you know that only because you have modding/programming background knowledge. Users dont care.
Ah, so you want a random person who has this Mod Pack to download any mod he likes from DFFD, drag+drop it into some folder, and have the Mod Pack automatically add it to its list of available modules that it installs/deinstalls. Thats a pretty neat idea. :)Yep, that's it exactly :D
Only downside I see is that it requires more initiative than I attribute to the average currently-non-mod-using player. It would still require an active mindset of looking through the forum, seeing a mod they like by finding it in the modding board and reading the description, going to DFFD, download it, unpack it, drag+drop it into the Mod Loader. Reminds me of Nexus Mods, Oblivion or Skyrim. ^^
And cross compability for linux/mac is a big plus of course. :)
My concept for the way load order works is uh, very similar, to the elder scrolls model - user chooses order, gets feedback on compatibility, retry until it works or you realise it's not going to work at all
the 2 weeks of conversation is due toQuoteMy concept for the way load order works is uh, very similar, to the elder scrolls model - user chooses order, gets feedback on compatibility, retry until it works or you realise it's not going to work at all
you're talking about a mod tool for another game in who knows what language that can't just be "ported" over to DF.
So... I would propose something simple first.
v1.0
Static Mods (scripts, parsed raws [either python parsed or batch/shell parsed], and patch files)
Then...
v2.0
Do a mod merger (maybe using a system similar to Meph's search/replace concept; mods that don't merge though, would have to be figured out somehow so the proper flags wouldn't conflict)
Then maybe an
v3.0
advanced object merger.
We're waiting for a mansion when all we need is a shack atm.
We literally have ALL the tools to create a batch based mod system. User clicks on a batch file or linux shell script, then gets a bunch of text prompts asking what mod he wants to load. Then the mod is applied, just before the game is started from the gui or batch file.
We could EVEN HOST A GITHUB repository of all the mods so a user can go and download them :)
Hell, we could use github to create our PATCH files for us.
BTW, this is just a proof of concept, but it could easily be done rather than having a user have to hunt mods down and parse and merge.
However, in the meantime, I'd propose just including a few mods, say 5 to 10 in an initial release, and work on a repo where users can upload their mods. Hell, the theory is easy enough for anyone to implements. The batch file could simply list a subfolder's dir contents where the mods are stored.... and parse from their. The batch file would automagically parse and patch the mod compared to a base vanilla version, which is entirely possible using diff3.
I'm merely suggesting if we're having a hold up based on what is ready vs not ready. Currently, no one is 100% onboard with the way diff3 does merging. It can drop lines that are meant to be merged together.
So, since that requires more advanced merging techniques, I was proposing in lieu of a fully working solution, to merely keep diff3 to applying tilesets atm and we only load 1 base mod.
I do see one issue with tilesets, and that's when we have mods that bring in new creatures, plants, etc... we can't rely on a universal tileset applier for all mods... (for the most part it might work though).
As to modularity:
I think if modders keep their changes to new_files vs changing vanilla files. It would help with merging mods... However, a lot of mods mod objects in vanilla files... so unless the object was removed and put into an external file to be modded...
I'm merely suggesting if we're having a hold up based on what is ready vs not ready. Currently, no one is 100% onboard with the way diff3 does merging. It can drop lines that are meant to be merged together.
So, since that requires more advanced merging techniques, I was proposing in lieu of a fully working solution, to merely keep diff3 to applying tilesets atm and we only load 1 base mod.
I do see one issue with tilesets, and that's when we have mods that bring in new creatures, plants, etc... we can't rely on a universal tileset applier for all mods... (for the most part it might work though).
As to modularity:
I think if modders keep their changes to new_files vs changing vanilla files. It would help with merging mods... However, a lot of mods mod objects in vanilla files... so unless the object was removed and put into an external file to be modded...
Argh, didn't mean to imply we shouldn't go forward with a one-mod-only version to start. I just don't see that as a viable end-goal.
"I was proposing in lieu of a fully working solution, to merely keep diff3 to applying tilesets atm and we only load 1 base mod."
Argh, didn't mean to imply we shouldn't go forward with a one-mod-only version to start. I just don't see that as a viable end-goal.
I didn't intend it to be an end-goal, but rather a start
Btw your idea of using blank placeholders for heavy diff files is similar in concept to if a file does not exist in vanilla. How would one code that, like, right now. Is that possible without having to implement ###comments... how would that be handled/flagged appropriately?
I think another cause for issue is the fact that some of us have ideas in one language (primarily myself in batch scripting and command lines) and others using python. I'm all for python; however, the py is weak in this one ;) It does have the added affect that while we discuss the possible "end-case" scenario's on the thread... anyone one of us could have completed a simple batch solution in our own right that could have served as a base 1. I do think if python is more versatile, to go that route vs shell scripting. Just ensuring that the end client dependencies are met when running python (without conflict) are ensured, then go for it (DFHack does it somehow).
Note:
The fear of a tool "never" getting better should never stop one from implementing it. Things can be proof of concept until we get better features implemented. A python based gui could be in order next.
I'm not 100% sure how your python script works.
I would imagine the concept would do 1 of two things.
A 'ModTool' (whatever name) which scan two folders
One for dropped in raw's, say
\ImportedMods
and one for already created patchffiles and maybe a matching zip of binary files that need to be included
\PatchFiles
<snip>
The one exception to merges for atm should be "tileset" patching on top of mods. Similar to how the linux java based lnp works. However, we could simply do a diff3 comparison using a common ancestor. There hopefully won't be any conflicts, and a matching diff3 output file can be created using the concept.
The beauty of this system, is it leaves the complicated [possibly broken] merging of mods up to mod mergers who know how to hand implement merged mods. As we know, it's not a pretty process. That way when a mod is intelligently merged, we have a format for them to create a patch file from it. That patch file can BE PASSED AROUND
because the whole point of this thread is that we have a common input format that any tool or version of a tool can use, which won't change with improved logic
hrmmm...
Well. I can see how basing it on a base would break if you added beards to females and every mod afterwards didn't (basically the mods would say, no we took off beards) when your intended behaviour is to draw what the mods changed from what they used as their base. Which we all "assume" is vanilla, or at least [we assume we] can have have a common ancestor derived; which is not always true, such as with total conversion mods.
However. I guess it could be "rebased" on a bearded version as a base... but the mod load concept I had would break?
I was under the "mis-assumption" that since LNP pretty much shipped as a static build, that we could start with a static build of mods. Things would most likely break if players tried to mod the vanilla base after we checked the mods worked with a common ancestor system. However, in theory we shouldn't have to 'check' them if we plan on allowing importing of them.
If I loaded mods into an imported folder (via the whole mod set), derived diff's from based off vanilla, and tried to apply it to a non vanilla base... then I think there would be an issue at that point. Because then all diff's would be contextual vs unified and then you'd have those issues because you don't have a common ancestor anymore.
Well either way. I'm glad I learned some stuff about 3 way merges and kdiff3 at the very least. My brain has been hardwired lately with github thinking, so that's why I'm so pro patches. However, merge issues such as I've had with github would arise if applying to different bases.
The beards-as-base problem isn't really a problem, tbh. You'd just have to change your installation strategy. Instead of tweaking beards and then installing, the installer should consider any deviations from vanilla in the user's raw files a mod, and install that mod last and/or with highest priority.
It does point to the biggest problem, though, which is multifeatured mods. Because mod installation is currently such a hassle, mods start packaging more and more features together, when an automatic mod installer would be significantly simpler if each feature were packaged as its own individual mod. Then we wouldn't have to worry about whether any given change is significant to the mod or not: if it deviates from the raws, it's significant.I am all for modular mods, but the manifest is useful to declare dependencies and known these-two-mods-do-not-play-well-together cases.
I suppose we could just assume every change is significant to the mod, fail on any conflict, and hope that modders change their packaging to be more, well, modular. That would be a lot more helpful than a manifest file.
I haven't read the whole thread, but is there a reason you don't just use git as a versioning system?
I haven't read the whole thread, but is there a reason you don't just use git as a versioning system?
OK, I stayed up later than I should have and did some more work on the surrounding logic. I'm happy to call it version 0.2 now, and I've rewritten the readme as well. Anyone want to test it on some simple or not-so-simple mods?
https://github.com/PeridexisErrant/Py-Mod-Loader
OK, I stayed up later than I should have and did some more work on the surrounding logic. I'm happy to call it version 0.2 now, and I've rewritten the readme as well. Anyone want to test it on some simple or not-so-simple mods?
https://github.com/PeridexisErrant/Py-Mod-Loader
I don't use Soundsense (or any sound at all), but if sound packs can be dropped in then those should be allowed as well.
PeridexisErrant, I played around a bit with you mod loader.
Notes (Using the following mod packs: All Races Playable, DF Fine Polish, Plantfixes load in that order):
I got a improper indentation error, which I fixed.
I had to figure out whether it was python2 or python3 (Python3). Might want to include a note in the readme to that effect.
I had to figure out where to place the mod files (in LNP/Mods/) Might want to include a note in the readme to that effect.
Upon merging it one set of complete set of raws in temp/raw/ and in temp/raw/objects/, which were different from each other.
Upon merging it initially erased all files in the Plantfixes mod due the folder being named objects in stead of the standard raw.
After deleting the excess raws, I successfully created a pocket embark on a pocket world (With a human civ, no less, thanks to a successfully merged all races playable mod)
I have checked to see if the merges were 'clean' (that is, none of the feature broke anything from the other mods that they shouldn't have) but it appears to works as advertised, but a bit buggy.
If you need any other help PeridexisErrant, let me know and I will be happy to help with my Rusty Competent Python Scripter and my Rusty Competent C Programmer skills.
Yes, any variation from the vanilla file structure will mean files have to be copied instead of merged - which massively increases the chance of conflicts. You can't really avoid new files with some mods, but trying to 'fix' the folder structure should make them easier to spot.This looks like a very good start, though I'm not sure why it couldn't handle stuff outside the raw folder (like Stonesense XML or a new embark profile) so long as it doesn't attempt any flattening there.
<?xml version="1.0" ?>
<!-- This XML contains enough information to make an educated guess about how two mods would merge *without processing any files.*
If it sees multiple mods affecting the same file, it notifies the user that it has to try a merge to figure out compatibility.
This means that the tool can download a bunch of these manifests in bulk to preview mods. -->
<modpack token="Example"> <!-- The token is the name of the folder holding the mod, and how mods refer to one another. -->
<title>The Example Mod</title>
<version>0</version><subversion>90</subversion> <!-- Would display as "v0.90". Can also add a <revision> tag for "v0.90r2" -->
<author>Dirst</author>
<thread>http://www.bay12forums.com/smf/index.php?topic=000000</thread>
<download>http://dffd.wimbli.com/file.php?id=0000</download>
<description>This mod is an example used to illustrate the optional manifest file.</description>
<dependencies>
<dfhack/>
<mod token="Basic" min_version="0" min_subversion="01" order="before"/>
</dependencies>
<compatibility>
<core max_version="40" max_subversion="07"> <!-- The lack of min_ will match anything v40.07 or earlier. -->
<color>red</color>
<message level="3">The Example Mod is not compatible with Dwarf Fortress before v40.08.</message>
</core>
<core min_version="40" min_subversion="08" max_version="40" max_subversion="11">
<color>green</color>
</core>
<core min_version"40" min_subversion="12">
<color>orange</color>
<message level="3">The Example Mod has not been tested with this version of Dwarf Fortress.</message>
</core>
<dfhack min_version="40" min_subversion="08" min_revision="2" max_version="40" max_subversion="10" max_revision="1">
<color>green</color>
<message level="1">"The Example Mod" employs DFHack's interactionTrigger and lua scripts.</message>
</dfhack>
<dfhack min_version="40" min_subversion="10" min_revision="2">
<color>orange</color>
<message level="3">The Example Mod has not been tested with this version of DFHack.</message>
<message level="1">"The Example Mod" employs DFHack's interactionTrigger and lua scripts.</message>
</dfhack>
<mod token="CLA" min_version="40" max_version="40">
<color>green</color>
</mod>
<mod token="Ironhand" min_version="40" max_version="40">
<color>green</color>
</mod>
<mod token="MasterworkDF"> <!-- Lack of min_ and max_ will match any version -->
<color>red</color>
<message level="3">The Example Mod is known to be incompatible with Masterwork DF.</message>
</mod>
<mod token="Mayday" min_version="40" max_version="40">
<color>green</color>
</mod>
<mod token="OldGenesis">
<color>red</color>
<message level="3">The Example Mod is known to be incompatible with Old Genesis.</message>
</mod>
<mod token="Phoebus" min_version="40" min_subversion="07" max_version="40">
<color>green</color>
</mod>
<mod token="Spacefox" min_version="40" max_version="40">
<color>green</color>
</mod>
<mod token="Other" min_version="1" min_subversion="0" max_version="1" max_subversion="35">
<color>yellow</color>
<message level="2">There have been no reports of conflicts with this version of That Other Mod.</message>
<message level="1">"The Example Mod" & "That Other Mod" both add to entity_default.txt, but do not appear to affect one another's edits.</message>
</mod>
<mod token="Basic" min_version="0" min_subversion="01" max_version="0" max_subversion="20">
<color>green</color>
<message level="1">"The Example Mod" is an extension of "The Basic Mod" and will not function without it.</message>
</mod>
<mod token="Basic" min_version="0" min_subversion="21">
<color>orange</color>
<message level="3">"The Example Mod" has not been tested with this version of "The Basic Mod".</message>
<message level="1">"The Example Mod" is an extension of "The Basic Mod" and will not function without it.</message>
</mod>
</compatibility>
<manifest>
<!-- Upon merge, any files with no path info are copied into an LNP documentation subfolder named after the token. -->
<!-- The parse attribute defaults to "flatten" which also strips out anything outside of []s. -->
<file parse="text">manifest.xml</file> <!-- Encode linefeeds if needed, but do not flatten this file. -->
<file parse="text">readme.txt</file>
<file parse="none">The Example Mod v0.90.pdf</file> <!-- Pass this file exactly as-is. -->
<file parse="none">data/art/font.TTF</file>
<file>data/init/embark_profiles.txt</file> <!-- Should be able to add a profile to the end of the user's set. -->
<file parse="text">data/init/overrides.txt</file> <!-- TWBT uses # to comment out inactive lines, so can't trust flattening. -->
<file parse="text">raw/onLoad.init</file> <!-- Mods that use DFHack will probably want to add lines to this file. -->
<file>raw/graphics/graphics_example.txt</file>
<file parse="none">raw/graphics/example/example.png</file>
<file>raw/objects/creature_example.txt</file>
<file>raw/objects/entity_default.txt</file> <!-- Many mods will need to add lines into this file. -->
<file>raw/objects/interaction_example.txt</file>
<file>raw/objects/reaction_example.txt</file>
<file parse="text">raw/objects/text/secret_wisdom.txt</file>
<file parse="text">raw/scripts/example-script.lua</file>
<file parse="text">stonesense/index.txt</file> <!-- Any Stonesense content will need to add a line to this file. -->
<file parse="text">stonesense/example/index.txt</file>
<file parse="none">stonesense/example/example.png</file>
<file parse="text">stonesense/example/example.xml</file>
</manifest>
</modpack>
Hrm. I was sort of hoping for a way to put more... granular merging instructions into a manifest file. For example, take my vampire fodder minor mod (one of the modular modpack I uploaded for testing), which replaces a number of creatures' BLOOD or ICHOR with a custom BLOOD2 or ICHOR2. It would be super sweet if I could tell the merger that, if another mod messes with BLOOD for [these creature objects], please change it so it messes with BLOOD2 instead.The goal is to parse things that could individually function if just dropped onto the DF folder. Sometime after v1.0, it might make sense to fold in something more elaborate such as scripting. But I think it'd be easier to user Rubble as a back end for that rather than re-inventing the wheel.
Now that I write that out, though, it sounds unrealistically ambitious, so never mind.
Skimmed over your script and the logic looks OK,but it's not really runnable right now >>. Herp derp it's supposed to be Python 3.
I think your observation about mods which put new entities in new files being more error-prone than those which put new entities in existing files is flawed. AFAICT the reason you think this is that you don't have any dupe-checking logic between raw files - but dupes could just as easily be placed in separate vanilla raw files as they could in separate modded raw files.