Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1] 2

Author Topic: Rubble Script Loader: In-save DFHack script and module loader.  (Read 3115 times)

milo christiansen

  • Bay Watcher
  • Something generic here
    • View Profile

The Rubble script loader is a small piece of code Rubble uses to load DFHack scripts and script modules from the save directory. Without this loader you can still load commands from the save directory, but not modules.

The thing is, if you want your mod to have lots of DFHack powered reactions/workshops/etc you need a lot of scripts, and most of those scripts will either duplicate each other or you will have custom modules with shared APIs. The problem is "how do you install these modules?" By default they need to be placed in the "hack/lua" directory, which works fine most of the time, but for custom modules you either need to distribute a preinstalled version of your mod or you need to make sure your users install the required scripts. Another big problem is the fact that you can only share saves with other users that have the required modules installed, which is something of a pain.

This loader fixes all that.

This loader was written for use with Rubble (hence the name) but there is no real reason it can't be used by itself, so here it is (complete with extra comments to make up for not having the Rubble documentation).

The new loader (needed to use scripts from Rubble):
Code: [Select]
--
-- Rubble pseudo-module loader.
--
-- This code provides support for functionality sorta like `dfhack.script_environment`, but with a
-- syntax and usage much more like the Lua default "require" mechanism. In addition automatic
-- unloading is provided for modules that need it.
--
-- This loader also provides "live reloading", basically this makes the scripting system think the world
-- was unloaded, then reloaded, refreshing the scripts. This is invaluable when debugging a module.
-- Sadly this feature does not refresh the entire scripting state, as that would be basically impossible.
--
-- A pseudo-module is structured mostly like a module that you would use with the default require, but
-- with the following differences:
--
-- * Use `rubble.mkmodule` rather than `mkmodule`
-- * You can provide an optional onUnload function that will be called when the world is unloaded.
-- * You can provide an optional onStateChange function that will be called for all state change events
--   *except* `SC_WORLD_LOADED` and `SC_WORLD_UNLOADED` (don't look at me, it's a DFHack bug).
--
-- Using a pseudo-module is the same as using a default module, with the following difference:
--
-- * Use `rubble.require` instead of `require`.
--
-- `rubble.require` is fairly talented, if it can't find a module it will try `dfhack.script_environment`
-- and finally the default `require`, so you can just use it in all cases and it will load the module no
-- matter what underlying module mechanism was used.
--

-- Important Globals and Loader Functions

-- This little song-and-dance creates a new global table named "rubble".
dfhack.BASE_G.rubble = {}

-- Don't touch, internal.
rubble.__modules = {}

-- Don't touch, internal.
function rubble.__load_module(name)
local modfile = SAVE_PATH.."/raw/modules/"..name..".lua"

-- If the given pseudo module does not exist first try "dfhack.script_environment" and then "require".
if not dfhack.filesystem.exists(modfile) then
if not dfhack.findScript(name) then
return require(name), nil
end
return dfhack.script_environment(name), nil
end

env = {}
setmetatable(env, { __index = dfhack.BASE_G })

local f, perr = loadfile(modfile, 't', env)
if f then
local ok, module = pcall(f)
if not ok or not module then
return nil, module
end
rubble.__modules[name] = module
return module, nil
end
return nil, perr
end

-- Registers a single value as a module, useful for making small modules for special purposes.
-- Unlike the other module functions you can overwrite existing modules with this.
function rubble.forcemodule(name, module)
rubble.__modules[name] = module
end

-- Use exactly like "mkmodule"
-- If called with the name of an existing module it will return a reference to the existing module.
function rubble.mkmodule(name)
if rubble.__modules[name] ~= nil then
return rubble.__modules[name]
end

env = {}
setmetatable(env, { __index = dfhack.BASE_G })
return env
end

-- "Extend" an existing module.
-- Makes a new module that has the API of the old module, plus whatever you add to it,
-- the old module is not modified.
-- This should work even if the parent uses "dfhack.script_environment" or the default
-- "require" mechanism.
function rubble.extendmodule(parent, name)
if rubble.__modules[name] ~= nil then
return rubble.__modules[name]
end

local parentmod = rubble.__modules[parent]
if parentmod == nil then
-- rubble.require errors out if the module does not exist in any form.
parentmod = rubble.require(parent)
end

env = {}
setmetatable(env, { __index = parentmod })
return env
end

-- Use like "require", but for modules made with "rubble.mkmodule".
-- If the module does not exist this fails over to trying to load the module with
-- "dfhack.script_environment", then the default "require". If both of those fail
-- loading aborts with an error.
function rubble.require(name)
if rubble.__modules[name] == nil then
local mod, err = rubble.__load_module(name)
if err ~= nil then
qerror(err)
end
return mod
end
return rubble.__modules[name]
end

-- Refresh any DFHack startup scripts and Rubble pseudo-modules.
-- Some scripts may not like this! In general the only problems will be scripts
-- that start global actions and do not handle being called multiple times gracefully.
-- If your script can be called twice in a row with no ill effect then it should be
-- fine.
-- I don't/can't fully refresh the scripting system, as there is far too much room for
-- error, so in general the only things that get refreshed are pseudo modules and save
-- init scripts, which is generally sufficient, particularly since this is for use with
-- Rubble, which makes heavy use of both.
--
-- Oddly DFHack does not appear to call `onStateChange` for the `SC_WORLD_LOADED` *or*
-- `SC_WORLD_UNLOADED` events. Upon further consideration there is really no reason to,
-- as scripts are run on load, and `onUnload` is called when the world is unloaded, but
-- this behavior seems strange and may be a bug.
function rubble.refresh()
-- Amazingly this is all that is needed.
-- For some odd reason unloading is carried out for both events, I guess as insurance
-- for missed unload events.
dfhack.onStateChange.DFHACK_PER_SAVE(SC_WORLD_LOADED)
end

function onUnload()
for _, module in pairs(rubble.__modules) do
if module.onUnload ~= nil then
module.onUnload()
end
end

rubble.__modules = {}
end

function onStateChange(state)
for _, module in pairs(rubble.__modules) do
if module.onStateChange ~= nil then
module.onStateChange(state)
end
end
end

The old loader (needed if you want to use some of the old scripts I have posted)
Code: [Select]
----------------------------------------------------------------------------------------------------
--Begin-Rubble-Script-Loader------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------

-- This code will load any Lua scripts from "raw/dfhack" whenever you load a world.
--
-- I feel safe in saying that, with my work on powered workshops, I probably have more experience with
-- loading lots of DFHack script at once than any other modder. This experience has shown me that the
-- default mechanisms are cumbersome when you are loading more than 4-5 scripts (and I often load
-- many, many more than that). This loader in intended to make it easy to load lots of scripts with
-- minimal modder effort.
--
-- Scripts that end with ".mod.lua" will be loaded as modules (you can use them with rubble.require)
-- whereas scripts that end with ".start.lua" will be loaded directly when the world loads.
-- You can use module files ("raw/dfhack/*.mod.lua") with normal commands (scripts in "raw/scripts"),
-- not just start scripts ("raw/dfhack/*.start.lua").
--
-- You do not need to worry about when a world loads or unloads (in most cases), as the loader takes
-- care of these details for you, just create an unload function (as described later) if you need to
-- and let to loader do the work.
--
-- ".mod.lua" files may have an "onUnload" function that is automatically called when the world is
-- unloaded, ".start.lua" files may add a function to the "rubble.unloaders" table to accomplish the
-- same thing. This is a convenience, as you could do the same thing by hooking the state change event
-- yourself.
--
-- This loader is somewhat similar to "dfhack.script_environment", but it is far more flexible, with
-- special support for cleanup, refreshing (during development), and automatic startup without having
-- to register scripts individually.
--
-- When you have 50+ scripts not having to list them all in the onload.init file is a great time saver
-- and it can also prevent bugs related to renaming a script and forgetting to update the onload.init
-- or forgetting to list a required script. Also being able to easily and quickly refresh the scripts
-- (via the ":lua rubble.reload_scripts()" command) is an invaluable help during mod development, as
-- it makes it easy to make changes to a script and see them immediately in game.
--
-- While most (if not all) of the things this loader can do can also be done with the vanilla mechanisms,
-- this loader does them in a more elegant and simple fashion. Since I use this code with all my mods
-- (most of which include large amounts of DFHack Lua code) it is highly optimized for ease of use.

-- Important Globals and Loader Functions

-- This little song-and-dance creates a new global table named "rubble".
dfhack.BASE_G.rubble = {}

-- The save directory path, stored here as a little used convenience.
rubble.savedir = SAVE_PATH

-- Don't touch, internal.
rubble.modules = {}

-- You can add functions to this table to be called when the world is unloaded, not needed for modules
-- (they have their own onUnload functions), but useful for start scripts.
rubble.unloaders = {}

-- Don't touch, internal.
function rubble.load_script(name)
env = {}
setmetatable(env, { __index = dfhack.BASE_G })

local f, perr = loadfile(name, 't', env)
if f then
return safecall(f)
end
dfhack.printerr("    Error: "..perr)
end

-- Don't touch, internal.
function rubble.load_module(name)
env = {}
setmetatable(env, { __index = dfhack.BASE_G })

local f, perr = loadfile(rubble.savedir.."/raw/dfhack/"..name..".mod.lua", 't', env)
if f then
local ok, module = pcall(f)
if not ok or not module then
return nil, module
end
rubble.modules[name] = module
return module, nil
end
return nil, perr
end

-- Registers a single value as a module, useful for making small modules for special purposes.
-- Unlike the other module functions you can overwrite existing modules with this.
function rubble.forcemodule(name, module)
rubble.modules[name] = module
end

-- Use exactly like "mkmodule"
-- If called with the name of an existing module it will return a reference to the existing module.
function rubble.mkmodule(name)
if rubble.modules[name] ~= nil then
return rubble.modules[name]
end

env = {}
setmetatable(env, { __index = dfhack.BASE_G })
return env
end

-- Use like "require", but for modules made with "rubble.mkmodule"
function rubble.require(name, path)
if rubble.modules[name] == nil then
local mod, err = rubble.load_module(name)
if err ~= nil then
qerror(err)
end
return mod
end
return rubble.modules[name]
end

-- Called when the world is unloaded to take care of any cleanup that may be needed.
function rubble.unload_scripts()
for _, unload in pairs(rubble.unloaders) do
unload()
end

for _, module in pairs(rubble.modules) do
if module.onUnload ~= nil then
module.onUnload()
end
end

rubble.modules = {}
rubble.unloaders = {}
end

-- This function will load all the scripts, feel free to call this as often as you like on loaded
-- worlds to refresh the scripts.
function rubble.reload_scripts()
rubble.unload_scripts()

local scrlist = dfhack.internal.getDir(rubble.savedir.."/raw/dfhack/")
if scrlist then
table.sort(scrlist)
for i,name in ipairs(scrlist) do
if string.match(name,'%.start.lua$') then
print("  "..name)
rubble.load_script(rubble.savedir.."/raw/dfhack/"..name)
end
end
else
print("  No startup scripts installed.")
end
end
rubble.reload_scripts()

-- Use this instead of onUnload so that other users of init.lua can use onUnload without problems.
dfhack.onStateChange.Rubble_Loader = function(code)
if code == SC_WORLD_UNLOADED then
rubble.unload_scripts()
dfhack.onStateChange.Rubble_Loader = nil
end
end

----------------------------------------------------------------------------------------------------
--End-Rubble-Script-Loader--------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------

From now on any script modules I upload will use this loader, as it is too hard to maintain Rubble and vanilla versions of everything.
« Last Edit: October 30, 2015, 02:33:30 pm by milo christiansen »
Logged
Rubble 8 - The most powerful modding suite in existence!
After all, coke is for furnaces, not for snorting.
You're not true dwarven royalty unless you own the complete 'Signature Collection' baby-bone bedroom set from NOKEAS

Roses

  • Bay Watcher
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #1 on: July 13, 2015, 02:20:44 pm »

You can now use dfhack.script_environment('blah').function(args) where 'blah' can be in the raw folder. For example, if you wanted to put hack/lua/utils.lua in the raw/scripts folder you could just use
Code: [Select]
utils = dfhack.script_environment('utils')Instead of
Code: [Select]
utils = require 'utils'
Logged

lethosor

  • Bay Watcher
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #2 on: July 13, 2015, 04:08:49 pm »

require() and script_environment() search in different places.
Logged
DFHack - Dwarf Manipulator (Lua) - DF Wiki talk

There was a typo in the siegers' campfire code. When the fires went out, so did the game.

Roses

  • Bay Watcher
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #3 on: July 13, 2015, 05:07:16 pm »

Right, but you can use script_environment like modules can't you? There by replacing the need for anything in the hack/lua folder.
Logged

lethosor

  • Bay Watcher
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #4 on: July 13, 2015, 05:28:18 pm »

Sure, but putting things in hack/lua prevents them from being treated as scripts by DFHack, which could be desirable for things that aren't actually intended to be used as scripts.
Logged
DFHack - Dwarf Manipulator (Lua) - DF Wiki talk

There was a typo in the siegers' campfire code. When the fires went out, so did the game.

expwnent

  • Bay Watcher
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #5 on: July 15, 2015, 02:46:29 am »

As of the most recent release, DFHack already searches in the raw folder of the current save for scripts. First it searches there, then it searches in df/objects/raw then it searches in hack/scripts. The script_environment function works just as well as modules, but it allows circular dependencies and correctly updates if you unload a save and load a save with a different version of the same script, whereas the module version will not.
Logged

milo christiansen

  • Bay Watcher
  • Something generic here
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #6 on: July 18, 2015, 10:27:08 am »

script_environment():

I already know about that, but it is not as easy to use as my loader. My loader handles loading, unloading, refreshing, etc in an easy to use way that parallels the vanilla module syntax.

One of the nicest thing about this system is that scripts are detected and loaded automatically, no need for an entry in the onload.init file or the like.

Anyway, frankly I don't care if you would rather I did things differently, because this loader was designed to solve problems that I feel safe in saying no one else has ever had to worry about (yet). If you ever have to load 50+ scripts at once (every time the world loads) you will quickly come to understand that the module loader part of this is, in many ways, a side show (admittedly a powerful sideshow).

The next version of Rubble will come with an improved version of this loader (which I will post here as well).

Logged
Rubble 8 - The most powerful modding suite in existence!
After all, coke is for furnaces, not for snorting.
You're not true dwarven royalty unless you own the complete 'Signature Collection' baby-bone bedroom set from NOKEAS

expwnent

  • Bay Watcher
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #7 on: July 19, 2015, 04:01:09 am »

Most of the time someone implements something they do a script version. The main insight behind script_environment is that there is no need for people to create a separate script version and library/module version. They can just do it all at once in one file. So instead of having syndrome-util and add-syndrome it's possible to combine those into one script (not currently implemented for backwards compatibility reasons but it will be at some point). Scripts will be loaded only as needed at runtime instead of all at once at the start. Also new in the most recent version is that it no longer recompiles every script just before running it. It only recompiles if necessary, which should greatly improve things.

What problems exactly are you trying to solve with this? I have no idea what you mean by "side show".

It is not a matter of how you do things, it's just that we've worked on similar things and if your way is better then it should be incorporated into DFHack proper.
Logged

milo christiansen

  • Bay Watcher
  • Something generic here
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #8 on: July 20, 2015, 10:28:08 am »

I use this to load what I call "start scripts", scripts that run automatically when the world is loaded.

Start scripts do not have to be listed in onload.init, which makes them very easy to use. If you take a look at the powered workshop addon that come with Rubble you will see a simple trend in my work, all core functionality (APIs and the like) are in modules, registering workshops and reactions with plugins is in start scripts. This way each component has it's own script, making each script short and easy to understand, at the cost of having LOTS of scripts. When you have 50+ scripts listing them by hand becomes a real chore :)

Another important feature that the newest version of this loader provides is the ability to register "unloaders", simple functions that are called when the world is unloaded. This is much easier to use then hooking the state change event, or using eventfuls onUnload event.

"script_environment" wasn't available when I first wrote the loader, and I like how much easier to use my loader is, so I probably won't change it now. That and if I ever did drop my loader I would have to reimplement start scripts and unloaders anyway...

Of cource this loader is in no way incompatible with "script_environment", I do not expect everyone to use it for their own work, but if you want to use my modules you will have to have it installed. This is because Rubble uses it, and I do not have time to maintain Rubble and non-Rubble versions of my work.

EDIT: I just posted a new version today, enjoy!
« Last Edit: July 20, 2015, 10:41:47 am by milo christiansen »
Logged
Rubble 8 - The most powerful modding suite in existence!
After all, coke is for furnaces, not for snorting.
You're not true dwarven royalty unless you own the complete 'Signature Collection' baby-bone bedroom set from NOKEAS

expwnent

  • Bay Watcher
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #9 on: July 21, 2015, 12:31:24 am »

https://github.com/DFHack/dfhack/blob/master/Lua%20API.rst#save-init-script

This is also implemented in "vanilla" DFHack. It's open source so you can use whatever you want internally of course. I just wanted to see if there were any features that needed to be pulled into the main project.
Logged

milo christiansen

  • Bay Watcher
  • Something generic here
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #10 on: July 21, 2015, 04:45:03 pm »

https://github.com/DFHack/dfhack/blob/master/Lua%20API.rst#save-init-script

This is also implemented in "vanilla" DFHack. It's open source so you can use whatever you want internally of course. I just wanted to see if there were any features that needed to be pulled into the main project.

Um, what? I use that script to power my loader, so why are you mentioning it? My loader is basically a layer OVER that functionality.
Logged
Rubble 8 - The most powerful modding suite in existence!
After all, coke is for furnaces, not for snorting.
You're not true dwarven royalty unless you own the complete 'Signature Collection' baby-bone bedroom set from NOKEAS

expwnent

  • Bay Watcher
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #11 on: July 22, 2015, 01:54:39 am »

The init.d folder already does everything you've said for scripts, and raw/scripts supports custom modules with script_environment.
Logged

lethosor

  • Bay Watcher
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #12 on: July 22, 2015, 09:12:08 am »

I don't think the unload handlers and reload_scripts() would be as easy to implement with script_environment - the latter would require clearing dfhack.internal.scripts, as well as the four other tables used in prior versions of DFHack. For something as large as Rubble, a custom loader also has the advantage of not polluting the scripts directory, which is especially useful since (from what I can tell) most Rubble modules aren't intended to be user-runnable scripts.
Logged
DFHack - Dwarf Manipulator (Lua) - DF Wiki talk

There was a typo in the siegers' campfire code. When the fires went out, so did the game.

expwnent

  • Bay Watcher
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #13 on: July 22, 2015, 09:39:06 am »

Unload handlers are already implemented for scripts in init.d. I'm not sure what the purpose of reload_scripts is. As for polluting the scripts folder, you can just put them in a subfolder and it won't be directly listed by the "ls" command.
Logged

milo christiansen

  • Bay Watcher
  • Something generic here
    • View Profile
Re: Rubble Script Loader: In-save DFHack script and module loader.
« Reply #14 on: July 22, 2015, 09:44:41 am »

"reload_scripts" refreshes all the scripts, perfect for use when debugging.

I missed the init directory, that would indeed be comparable with my start scripts.
Logged
Rubble 8 - The most powerful modding suite in existence!
After all, coke is for furnaces, not for snorting.
You're not true dwarven royalty unless you own the complete 'Signature Collection' baby-bone bedroom set from NOKEAS
Pages: [1] 2