Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 [2] 3 4 ... 35

Author Topic: PyLNP 0.13b - Cross-platform Lazy Newb Pack port with graphics pack patching  (Read 156700 times)

Pidgeot

  • Bay Watcher
    • View Profile

I'm very glad people are showing interest! I'm going on vacation for two weeks (starting tomorrow), so I won't be able to code anything until I get back, but I'll try to keep an eye on this thread, and I appreciate all your feedback.

This looks great!  I'd love to see this become the standard, both because having a properly cross-platform launcher is awesome and because I can actually do some coding in python.  PTW with interest.

If you have any specifics that would make it easier for you to start using it, feel free to ask. ;)

Loading graphics packs from a diff is an excellent idea.  Besides saving space, it's a solid step toward a mod loader using the same mechanism.  It would probably only support small to moderate raw-only mods, but this would include everything from Accelerated DF to Modest Mod and a host of minor tweaks.  It should even be able to check compatibility, and only load non-conflicting mods. 

It's going to require some thought and probably some experimentation to figure out what's the best route - the most straight-forward way would simply be to have a standard diff-format file and then first *try* to patch (and erroring out if the patch couldn't be applied), but it *may* be preferable to do a custom format and exploit the hierarchical structure imposed by indentation in the raws - I'm not currently sure if the indentation has any semantic meaning for the game code itself, or if it's only useful to a human. It's one of the things I need to consider carefully; a standard diff would be much simpler to do (Python already has diff-stuff built in), but a custom format would make it easier to use that idea for mods (and it would be possible to gracefully handle the case where two different mods want the same change to occur).

Of course, you would probably also want a way of removing a mod again - and it would be best if a complete rollback wasn't required to remove a patch applied early in the sequence - so there's some challenge in figuring out exactly how to do that. (Ideas, anyone?)

For updates:  I haven't done much with the idea of updating by a diff, as there's a lot that could go wrong with deleting something important if the parsing goes wrong.  However the latest windows launcher does include an update notification tool; it just checks the DFFD version page (~70B) and if the version string is different notifies the user.  You could potentially extend something similar to auto-download the new pack from a configurable address and then push all data to the new pack.  To reduce the odds of losing user data, I offer the idea of a top-level folder "User Generated Content" - designed to copy to a new pack. 

That is probably what that will start as: a text file containing a version string and a URL to read for the new version, with a notification popup if a newer version was found (and there probably needs to be a download URL somewhere too).

Regarding DFHack stuff:  I would leave *any* development on this from until we have a stable version of 40_xx and DFHack for that version has matured, as it's bound to change significantly.  This goes double for graphics stuff; recent TwbT screenshots include an isometric view, I've got stonesense for 34.11 running with printmode:standard, and so on. 

Yeah, that's one of the reasons I'm not sure how to best approach the issue. Maybe some kind of "requirements.txt" file which specifies valid values for certain settings? Of course, you would need a mechanism to check for conflicts between different mods/utilities.

For mod loading, you might want to check out this.

It's likely going to be something along these lines, but it's going to be slightly different by going to a patch-based system, which I'd like to do. I'm not yet sure how to best handle that.
« Last Edit: July 22, 2014, 03:34:35 pm by Pidgeot »
Logged

shaver

  • Bay Watcher
    • View Profile

PTW, as they say.

I have some experience with delta-patching of systems, interested to see where you come out on it.
Logged

PeridexisErrant

  • Bay Watcher
  • Dai stihˇ, Hrasht.
    • View Profile

I like the direction that the update and dfhack segments seems to be going; if this is going to be a standard cross-platform thing the top priority is basic but flexible functionality. 

It's probably also a good idea to select a standard format for holding configuration data early - in the windows version it's now mostly going into a text file, but using a defined XML standard would probably be a better choice (not that I have any experience with this).  Keeping it extensible and well-formatted is important. 

This looks great!  I'd love to see this become the standard.
If you have any specifics that would make it easier for you to start using it, feel free to ask. ;)

Well, I learned my lesson with Dricus' java GUI - I'll stick with whichever has superior features and stability.  From my reading of the OP you've already got most of it!  Still to do: 
 - In utilities pane, don't show path or extension (ie "DwarfTherapist" instead of "Dwarf Therapist 23.2/DwarfTherapist.exe")
 - Allow pack authors to configure the contents of the folders and links menu
 - The DFHack tab (I don't actually care about this for 40.xx, and probably won't for a few months)
 - Update notification tool
 - Show which graphics pack is currently installed (current GUI just shows tileset name)
 - Preserve non-graphics settings when changing graphics packs (I asume this will just happen once diffs are used)
Completing the items on this list would see me move over, purely for the sake of a single launcher on all platforms - it should focus development nicely.  Other features not available on windows would be an inducement to move sooner - likewise my ability to understand the code!

Edit:  I'm going to add these as issues on bitbucket, so we can keep track of the ones that aren't already there.


Loading graphics packs from a diff is an excellent idea.  Besides saving space, it's a solid step toward a mod loader using the same mechanism.  It would probably only support small to moderate raw-only mods, but this would include everything from Accelerated DF to Modest Mod and a host of minor tweaks.  It should even be able to check compatibility, and only load non-conflicting mods. 
It's going to require some thought and probably some experimentation to figure out what's the best route - the most straight-forward way would simply be to have a standard diff-format file and then first *try* to patch (and erroring out if the patch couldn't be applied), but it *may* be preferable to do a custom format and exploit the hierarchical structure imposed by indentation in the raws - I'm not currently sure if the indentation has any semantic meaning for the game code itself, or if it's only useful to a human. It's one of the things I need to consider carefully; a standard diff would be much simpler to do (Python already has diff-stuff built in), but a custom format would make it easier to use that idea for mods (and it would be possible to gracefully handle the case where two different mods want the same change to occur).

Of course, you would probably also want a way of removing a mod again - and it would be best if a complete rollback wasn't required to remove a patch applied early in the sequence - so there's some challenge in figuring out exactly how to do that. (Ideas, anyone?)

I would prioritise getting a working system in for graphics packs, and then extend it to mods after all core functionality is in. 

A graphics pack in the current LNP consists of two parts.  From the data folder, the contents of data/art/ delete and replace the installed folder; then from data/init/ init.txt, d_init.txt, and colors.txt overwrite the installed files (GUI'd settings are then restored).  The raw folder then replaces the installed raw folder. 

In my pack (unzipped), each graphics option is about 5MB of which 4MB is raws - with minimal changes.  Using a diff for this would save a significant amount of space and stay reasonably modular.  In all cases diffs should be compared to the pure vanilla raws (and other text files) exactly as they come out of the Bay12 download.  The init files could also be done as a diff, since for graphics you only need to change [graphics:yes/no] and the four [*font:foo] tags (leaving the ubiqitious settings changes to a different diff.  To extend this system to mods would then be very simple, if the mods were written (or rewritten) to make the minimum effective change - using diffs from vanilla would mean that reorganising raws is generally a bad idea, which does bias this system to small mods (unless someone wants to rebase the whole thing, Meph). 

To modify your installed mods, I would not go near removing diffs from the middle of a stack - rather rebuild the whole thing without whatever mod you want to remove.  This dodges all kinds of problems with compatibility, as it should be checked every time. 
« Last Edit: July 23, 2014, 02:10:01 am by PeridexisErrant »
Logged
I maintain the DF Starter Pack - over a million downloads and still counting!
 Donations here.

doulos05

  • Escaped Lunatic
    • View Profile

In the midst of making an account to report a bug on the now un-maintained Linux LNP, I saw this. I felt like I'd taken a Silver warhammer to the gut (though that could be the stitches/meds from the operation...)! I'm an (extremely amateur) python programmer and I love cross-platform stuff! I'll be downloading the mods in the Starter Pack and fiddling around with it as time permits. Additionally, I'll read over your code and see if I can help at all. Yay, Python!
Logged

SirPrefect

  • Escaped Lunatic
    • View Profile

PTW
Logged

PeridexisErrant

  • Bay Watcher
  • Dai stihˇ, Hrasht.
    • View Profile

Two quick things: 

1)  Have you considered moving this to GitHub?  Every open source DF program I can think of is over there, bar this one. 

2)  To open up the use of this a bit, how about branding it the "Dwarf Fortress Launcher" rather than Lazy Newb Pack?  That might open the door to wider use in modpacks, for example. 
Logged
I maintain the DF Starter Pack - over a million downloads and still counting!
 Donations here.

Nopenope

  • Bay Watcher
    • View Profile

Because there's already a "launcher" on the utilities forum, maybe.
Logged

shaver

  • Bay Watcher
    • View Profile

Because there's already a "launcher" on the utilities forum, maybe.

Runner? Manager? Concierge? Overmind?
Logged

Pidgeot

  • Bay Watcher
    • View Profile

Two quick things: 

1)  Have you considered moving this to GitHub?  Every open source DF program I can think of is over there, bar this one. 
Dricus' launcher was also on Bitbucket.  :P

Basically, I chose Bitbucket because I like it better than GitHub. It's just a little more pleasant to work with, IMO.

2)  To open up the use of this a bit, how about branding it the "Dwarf Fortress Launcher" rather than Lazy Newb Pack?  That might open the door to wider use in modpacks, for example.
In the long run, that should probably happen, but then I also need to change the banner, and I have zero skills to create a new one. Suggestions are welcome, though. :)

mifki

  • Bay Watcher
  • works secretly...
    • View Profile
    • mifki

Sorry guys that I spam with this thread with my ideas again.
Why not use node-webkit, for example for a launcher? It would be truly cross-platform without need to install Python or anything, easier to develop (just html+javascript) and easier to build more complex UI, eg. you can have a webpage with colormaps/tilesets with previews to choose very easily.

Pidgeot

  • Bay Watcher
    • View Profile
Re: PyLNP - Cross-platform Lazy Newb Pack port with graphics pack patching
« Reply #25 on: August 01, 2014, 03:58:27 am »

You don't need Python for this launcher either as long as you use the pre-built executable - only if you run directly from source. Also, node-webkit does not appear to support single-file binaries.

You also tie yourself down to *actually using a browser*. I have deliberately designed the launcher to have the UI as dumb as possible - if someone wants to, they can implement e.g. a terminal-based UI on top of the base code and use it with Dwarf Fortress in text mode. I don't see how that would be done with node-webkit.

salithus

  • Bay Watcher
  • gottagofast
    • View Profile
Re: PyLNP - Cross-platform Lazy Newb Pack port with graphics pack patching
« Reply #26 on: August 05, 2014, 09:41:25 pm »

I've got a background in software development, but am admittedly lost when it comes to DF stuff. I'm used to very traditional centralized source control (currently doing C# stuff w/ svn). What's the best way to get started and contribute? Looking at the project page there are 6 issues and a few I could probably tackle.

I *think* I should:

1) Fork python-lnp.
2) Make my updates.
3) Make a pull request.
4) ???
5) Profit.

E: Also does anyone have a recommendation for an IDE on Windows 8.1? I'm using SourceTree and N++ right now.
« Last Edit: August 05, 2014, 10:01:19 pm by salithus »
Logged

Billy Jack

  • Bay Watcher
  • Baywatch Watcher
    • View Profile
Re: PyLNP - Cross-platform Lazy Newb Pack port with graphics pack patching
« Reply #27 on: August 06, 2014, 10:49:06 pm »

Loading graphics packs from a diff is an excellent idea.  Besides saving space, it's a solid step toward a mod loader using the same mechanism.  It would probably only support small to moderate raw-only mods, but this would include everything from Accelerated DF to Modest Mod and a host of minor tweaks.  It should even be able to check compatibility, and only load non-conflicting mods. 

It's going to require some thought and probably some experimentation to figure out what's the best route - the most straight-forward way would simply be to have a standard diff-format file and then first *try* to patch (and erroring out if the patch couldn't be applied), but it *may* be preferable to do a custom format and exploit the hierarchical structure imposed by indentation in the raws - I'm not currently sure if the indentation has any semantic meaning for the game code itself, or if it's only useful to a human. It's one of the things I need to consider carefully; a standard diff would be much simpler to do (Python already has diff-stuff built in), but a custom format would make it easier to use that idea for mods (and it would be possible to gracefully handle the case where two different mods want the same change to occur).

Of course, you would probably also want a way of removing a mod again - and it would be best if a complete rollback wasn't required to remove a patch applied early in the sequence - so there's some challenge in figuring out exactly how to do that. (Ideas, anyone?)

For updates:  I haven't done much with the idea of updating by a diff, as there's a lot that could go wrong with deleting something important if the parsing goes wrong.  However the latest windows launcher does include an update notification tool; it just checks the DFFD version page (~70B) and if the version string is different notifies the user.  You could potentially extend something similar to auto-download the new pack from a configurable address and then push all data to the new pack.  To reduce the odds of losing user data, I offer the idea of a top-level folder "User Generated Content" - designed to copy to a new pack. 

Check out this VB code that I wrote to manage different tilesets. Breaking them down to only needing a file defining the tile and color values.
http://www.bay12forums.com/smf/index.php?topic=133506.msg4779833#msg4779833

If you find the concepts / methods of use, feel free to use them as you convert it to Python. I make use of VB's dictionary object, which is an associated array in Python, if I understand correctly.
Logged
Give a dwarf a fish, you feed him for a season.
Give a dwarf a Fishpond, couple of buckets, build a Fishery, and enable Fishfarming labor; you feed him for a lifetime. (And get someone to clean and prep the fish)

Pidgeot

  • Bay Watcher
    • View Profile
Re: PyLNP - Cross-platform Lazy Newb Pack port with graphics pack patching
« Reply #28 on: August 08, 2014, 07:13:31 am »

I came back from vacation yesterday, but don't expect too much of me before next week - I'm pretty busy this weekend.

I've got a background in software development, but am admittedly lost when it comes to DF stuff. I'm used to very traditional centralized source control (currently doing C# stuff w/ svn). What's the best way to get started and contribute? Looking at the project page there are 6 issues and a few I could probably tackle.

I *think* I should:

1) Fork python-lnp.
2) Make my updates.
3) Make a pull request.
4) ???
5) Profit.

Yep, pretty much. In other words, what you're already doing. :P

E: Also does anyone have a recommendation for an IDE on Windows 8.1? I'm using SourceTree and N++ right now.

Personally, I'm just using Vim with a plug-in for Intellisense-like behavior, but that is definitely not a solution for everyone.

Check out this VB code that I wrote to manage different tilesets. Breaking them down to only needing a file defining the tile and color values.
http://www.bay12forums.com/smf/index.php?topic=133506.msg4779833#msg4779833

If you find the concepts / methods of use, feel free to use them as you convert it to Python. I make use of VB's dictionary object, which is an associated array in Python, if I understand correctly.

This is more or less the basic idea I have in mind, but the format would probably need to be a bit more flexible to allow for insertion and deletion of new tags. Python has some diff-related things built-in, I need to see if that can be adapted to the DF configuration format.

salithus

  • Bay Watcher
  • gottagofast
    • View Profile
Re: PyLNP - Cross-platform Lazy Newb Pack port with graphics pack patching
« Reply #29 on: August 08, 2014, 11:21:45 am »

I came back from vacation yesterday, but don't expect too much of me before next week - I'm pretty busy this weekend.

Sweet. I got your feedback on my pull request - I've never worked with Python before so expect more of the same until I get my sea legs :D
Logged
Pages: 1 [2] 3 4 ... 35