Bay 12 Games Forum

Dwarf Fortress => DF General Discussion => Topic started by: Markavian on May 03, 2020, 10:07:08 am

Title: Future of the Dwarf Fortress Map Archive
Post by: Markavian on May 03, 2020, 10:07:08 am
Hi I'm Markavian, (aka John Beech), and once upon a time in 2008 I made a website called the DF Map Archive (https://mkv25.net/dfma/) featuring a flash based viewer in collaboration with an encoder programmed by ShadowLord. I then continued to build out the site on my domain. You and the community may have used it from time to time...

Between then and now, I went from hobby coder, to aspiring software engineer, to where I am now; a software team lead at a big international broadcaster, married to my beautiful wifey (who is also an engineer), and adopted two cats Ezio, and Icarus. Basically life happened.

Meanwhile, the rest of the world moved on as well; Flash doesn't work on mobile, and is being blocked or deprecated by most browser vendors. This means that the map decoder / viewer doesn't work for most peple, and the the sizeable map archive 11800 maps, and 33400 points of interest are at risk to the future internet community.

With problems, come solutions... and I'd like to call out and thank the work of Jacob Blomquist, who has made a P5.js and HTML5 Canvas version (https://github.com/JacobRBlomquist/DFMA-Viewer-HTML5) of the map viewer, that should run from modern browsers. He contacted me recently, and reminded me of the on going need to support the development of the map archive.

So that brings me to this post...

Spoiler (click to show/hide)

https://github.com/df-map-archive/dfma-map-file-archive/

DFMA Map File Archive

Motivation
To help share, and preserve Dwarf Fortress stories for aeons to come.

Purpose
The DFMA is place to upload maps, tell stories, tracks points of interest, comments, and community favourites - bringing to life fortresses through interactive snapshots submitted by the community.

Underlying Technology
The Dwarf Fortress Map Archive (DFMA) is a repository and online viewer for player created maps exported from Dwarf Fortress. The site revolves around an optimized file format (.FDF-MAP), which can produce files 1/2000th the size of the ouputted bitmaps (for a 28 layer map) and 1/20th the size of equivalent PNG files. This makes them small enough to upload and transfer across the web without chewing up bandwidth.

Map File Archive
The map file archive is an archive of dwarf fortress maps (.fdf-map files) and any associated descriptive data, such as author, submitter, fortress name, map year, map season, submission date, points of interest, and description.

There are gaps in this data, this is a stalled project that needs love, work, and attention.

Project Roadmap
Any/all future work is being tracked through the shared github project page for through the df-map-archive organisation on github.

See: https://github.com/orgs/df-map-archive/projects/1

Timeline
2017 November
- Initial idea to create HTML/JS version of Map Archive discussed with Casey Link
- Created df-map-archive organisation on github
- Created dfma-map-file-archive namespace for archiving maps

2018 April
- Download and preserve .fdf-map files from https://mkv25.net/dfma/ to local machine
- Archiving stalled due to file size restrictions on repositories - lost motivation

2019 February
- Archiving stalled due to file size restrictions on repositories
- Looking for solutions; may just pay the cost

2020 May
- Jacob Blomquist creates a new P5.js and HTML5 Canvas map viewer for fdf-maps, and releases it on an MIT license: http://blomquist.xyz/dfma/ / https://github.com/JacobRBlomquist/DFMA-Viewer-HTML5
- Created a new DF Map Archiive project space to track and prioritise work

New Plan
- Break repo down into multi year backups, e.g. pre-2008, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, and 2019
- Automate process of creating multi-repo backups using template script
- Communicate intentions on forum

Footnote
The map archive has slowed in its use, but is an excellent resource for early Dwarf Fortress gameplay, and has allowed many great stories to be shared over the years. I hope that by providing this archive of the website that we can build a new site that will continue to support these stories into the future.

Fey Demands
There's a lot on my shoulders here. I want to try and open up the archive and provide long term support to the community by backing stuff up in Github as a place where other hackers, coders, programmers, engineers, etc. can go use and make stuff. It's the community data after all, my website is just hanging it all together.

I'm not look for money, or adverts, or any programming expertise. I think I'm just in need of support at this stage. I've got the outline of a plan worked out, and I've got lots of data to move and push around, and hopefully I'll have lots of exciting news to share with people. Maybe its just the ideas and inspiration I need, and that'll hopefully make this all worth while.

Also while I'm here, and incase they read this, congrats to Tarn, Zach, (and the community) for keeping Dwarf Fortress going for all these years. Among all the games I've embraced DF has carved a very precious rune on my heart.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Uthimienure on May 03, 2020, 10:31:10 am
While I've only played for a couple of years, it is very clear there is great value to your project!
A while back I did check out the DFMA and enjoyed browsing it.
Thank you for your labor of love and I sincerely hope you can maintain the motivation to carry this through.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Blomquistador on May 03, 2020, 05:34:42 pm
My name is Jacob Blomquist (Blomquistador).
I wanted to revive the DFMA mostly because I don't want these amazing builds and fortresses to be lost just because flash has lost support.
I also wanted to revive this project so that it would be ready for the eventual Steam release of DF.
I'm a longtime player of Dwarf Fortress, but this is the first contribution I've made to the community.

This is also a rare opportunity for us to add features and functionality to the map viewer. I would be happy to hear any contribution requests.

All in all I hope that this will revive interest in the DFMA and allow for more DF history to be shared amongst the community.

I also want to echo Markavian in thanking Tarn and Zach Adams for building such a profound and influential game that has touched the hearts of so many. I hope that the Steam release will go well and that we'll see more newcomers who can tackle the learning curve.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Salmeuk on May 03, 2020, 08:33:30 pm
Wow, I had my heart all set to be broken, expecting this thread to be the announcement that the archive would be finally shutting down . . but no!

I've uploaded maps to DFMA since it's creation. It's wonderful to search back and see all the strange and wonderful things I was building so long ago. And before the early 3d visualizers there was DFMA's ability to see a vertical y-slice of your upload, allowing all the strange megaproject sculptures to be visualized despite the impenetrable ASCII. Indispensable, really. 

So, from the bottom of my heart, thank you greatly for your donated time, effort, and hosting bandwidth!

I created a thread a few years back that you might enjoy reading, since many people shared their enjoyment of DFMA. (http://www.bay12forums.com/smf/index.php?topic=159965.0)

Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Markavian on May 04, 2020, 02:21:50 pm
Wow, I had my heart all set to be broken, expecting this thread to be the announcement that the archive would be finally shutting down . . but no!

I've uploaded maps to DFMA since it's creation. It's wonderful to search back and see all the strange and wonderful things I was building so long ago. And before the early 3d visualizers there was DFMA's ability to see a vertical y-slice of your upload, allowing all the strange megaproject sculptures to be visualized despite the impenetrable ASCII. Indispensable, really. 

So, from the bottom of my heart, thank you greatly for your donated time, effort, and hosting bandwidth!

I created a thread a few years back that you might enjoy reading, since many people shared their enjoyment of DFMA. (http://www.bay12forums.com/smf/index.php?topic=159965.0)

Thanks Salmeuk, I started reading through the first page, and that's really heart warming, thank you again :) fingers crossed I can get the site back open for business.
I'm getting stoked by the DF Steam release, feels like a big time for the community.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Fleeting Frames on May 08, 2020, 12:46:35 pm
I was expentant of the archive being abandoned (already so, in fact, with your latest uploads being decade ago and latest cmment being half a decade ago on DFMA) with flash support dropped. It makes me glad this is only not so, but that there's already a translating to HTML5 project that I was unaware of (gj & gz to Blomquistador).

Additionally, whenever you're ready, you might want to post the update onto reddit as well; while DFMA is linked on the sidebar I still see maps shared as collection of images on imgur, which may make sense for certain graphic set cases but is frequently done with ascii as well. Additional exposure would lead to additional users, I expect.

Though tbf I have to recheck "how did I use this again" every time I upload a map which might hurt usage numbers, so seeing if it can be simplified might be worth consideration -  for example mifki made 3d viewer (http://assets.mifki.com/df3dview/) which simplifies uploading to single copy-paste into dfhack console.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Markavian on May 09, 2020, 04:30:25 am
I was expentant of the archive being abandoned (already so, in fact, with your latest uploads being decade ago and latest cmment being half a decade ago on DFMA) with flash support dropped. It makes me glad this is only not so, but that there's already a translating to HTML5 project that I was unaware of (gj & gz to Blomquistador).

Additionally, whenever you're ready, you might want to post the update onto reddit as well; while DFMA is linked on the sidebar I still see maps shared as collection of images on imgur, which may make sense for certain graphic set cases but is frequently done with ascii as well. Additional exposure would lead to additional users, I expect.

Though tbf I have to recheck "how did I use this again" every time I upload a map which might hurt usage numbers, so seeing if it can be simplified might be worth consideration -  for example mifki made 3d viewer (http://assets.mifki.com/df3dview/) which simplifies uploading to single copy-paste into dfhack console.

Wow, that 3D viewer is amazing. I think I always wanted to do something like that with the existing data on the map archive. I had experimented with an isometric viewer using a custom tileset, but I could never really iron out the bugs in the system. I think once I get over the hill of "fixing the map archive", it would definitely be worth looking at the "upload my fortress" journey - make it more drag and drop ... and lets be real, people's bandwidth is much better these days, if we tried, we could probably do something with the actual game/map data (like the exporter script) rather then just compressing bitmaps. I think the data already in the DFMA (.fdf-map files) would be compatible with the 3D viewer as well - we've got colour information and layered blocks, so rendering cubes should be easy.

Good shout on reddit. I think I came back here to the forums first to dip my toe back into the community.

My update for this week:
- Updated the roadmap - got lots of ideas on the project page, got some more to add now :)
- Recovered the SVN repo for the old DFMA, wasn't much in there, but I've got a snapshot of the code dfma website in github now
- Started linting the dfma website using modern techniques, its slow going, I've got another 60 files to go, its quite painful - setup a deployment pipeline to make site updates easier / more controlled
- Started this morning putting together a spike for Jacob's decoder/viewer in https://github.com/df-map-archive/dfma-html5-map-viewer

Jacob made a few changes as well:
- Added progress bar and async
- Changed scaling factors
- Fixed a bug with smooth zooming
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Blomquistador on May 09, 2020, 01:03:22 pm
...... and lets be real, people's bandwidth is much better these days, if we tried, we could probably do something with the actual game/map data (like the exporter script) rather then just compressing bitmaps.

At the very least it would be nice to implement the .BMP compressor online so that people wouldn't need to download the compressor to upload .fdf-map files.
Creating a DFHack script that exports even more information (such as an entity list, water flow, etc) would be amazing and we could do so much with it.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Markavian on May 16, 2020, 12:10:42 pm
...... and lets be real, people's bandwidth is much better these days, if we tried, we could probably do something with the actual game/map data (like the exporter script) rather then just compressing bitmaps.

At the very least it would be nice to implement the .BMP compressor online so that people wouldn't need to download the compressor to upload .fdf-map files.
Creating a DFHack script that exports even more information (such as an entity list, water flow, etc) would be amazing and we could do so much with it.

Completely agree; something to add to the R&D list...

Weekly update, mostly for my own benefit:
- Jacob did an update to the viewer; now packaged a bit more friendlier
- I set up a build pipeline to deploy the new viewer code to the DFMA website : https://github.com/df-map-archive/dfma-html5-map-viewer
- I've got a "working" wrapper of Jacob's viewer code hosted on my website - it looks identical; but I'm now in a position to start feeding in DF map IDs
  - See: https://mkv25.net/dfma/xdfmadev/parcel/index.html

What's next:
- I'm still fighting with PHP Linting of the old website - I need to do this so I can start quality control around the map viewer changes - I want to avoid pushing broken code to the website when I have a limited amount of time to fix things
- Ready to start integrating DFMA Maps with Jacob's HTML viewer, using the above code/project
- Need to work out how Points of Interest will be supported with the new viewer - its not enough to just load a map (that should be easy), we also need a way to navigate using the existing PoI system

Project page:
- https://github.com/orgs/df-map-archive/projects/1



Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Fleeting Frames on May 17, 2020, 06:07:43 am
Oh, that's some fast progress. Few obvious notes that jump out to me:

- Display-wise, it seems the previous "multilevel" view in underground areas is gone, though.

- Navigation, might want to set the maps to be entirely navigatable with both mouse and keyboard (I'd suggest key placement that enables one to move around with one hand in both cases).

- POIs, current flash viewer had weird thing where logged out it directs you to a new page while logged in it updates the flash viewer. Limitation arising from lack of cookies, I assume? (Mind the GDPR.)
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Starver on May 17, 2020, 07:54:55 am
I've been meaning to PTW this thread for a while now, still not found a good thing to justify it, though.

The 'best' qualifying comment I've yet come up with is

Which is why I'm PingTW. With absolutely no need for any direct response to me on the above rambling screed that is mostly only talking about things you have no control over, and I'm obviously just venting for the sake of it.  :P


I really should consider using the DFMA more, though, to make it obvious I support it, and you. Maybe when I've decided I've got a good 0.47.x embark going, with my Super-Turtlable design philosophy nicely established, or something. *shrug*
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: lethosor on May 18, 2020, 08:43:39 pm
Wow, I had my heart all set to be broken, expecting this thread to be the announcement that the archive would be finally shutting down . . but no!
Same here. DFMA was one of the first DF sites I signed up on. Glad to see it's still alive!

Regarding the map export side of things: I've done some work with DFHack, as well as cross-platform DF-related tools in Go, and could help out there if needed. The latter approach would almost certainly be a command-line tool, and I guess a DFHack plugin/script would be too, although people might be more accustomed to that. (Is the .fdf-map tool command-line-based? I've never actually used it, as I don't run Windows.)

I would be willing to take a look at frontend stuff too, although it looks like you might have that covered pretty well. (And despite my free time having increased in the past couple months, the number of side projects I've taken on has also increased, so I can't make too many commitments, haha.)
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Fleeting Frames on May 19, 2020, 05:04:38 am
I don't run windows (to export maps; still have copy if I need it) either; I run the .fdf-map export tool in linux with mono --debug iirc. Though it tends to complain about nr of different tiles; which is fair enough given TWBT multilevel & transparency mean there's far less repeating tiles.

PS: Also, "I've done some work with dfhack" is amusing to hear :P (, given that you seem to be one of the most active dfhack core devs here).
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Warmist on May 20, 2020, 02:00:16 am
Wow, I had my heart all set to be broken, expecting this thread to be the announcement that the archive would be finally shutting down . . but no!
Same here. DFMA was one of the first DF sites I signed up on. Glad to see it's still alive!

Regarding the map export side of things: I've done some work with DFHack, as well as cross-platform DF-related tools in Go, and could help out there if needed. The latter approach would almost certainly be a command-line tool, and I guess a DFHack plugin/script would be too, although people might be more accustomed to that. (Is the .fdf-map tool command-line-based? I've never actually used it, as I don't run Windows.)

I would be willing to take a look at frontend stuff too, although it looks like you might have that covered pretty well. (And despite my free time having increased in the past couple months, the number of side projects I've taken on has also increased, so I can't make too many commitments, haha.)

Ditto.

Composed a few mails and then deleted them to Markavian with title: let's rewrite CMV viewer in js. Mostly because i'm not a webdev and my js would be no-libs-scary mess. However "Warmist is ecstatic upon hearing this news".

Edit: i wonder how possible would it be to integrate something into forums? Direct iframe with movie viewer would be awesome!
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Markavian on May 20, 2020, 04:46:09 am
Oh, that's some fast progress. Few obvious notes that jump out to me:
- Display-wise, it seems the previous "multilevel" view in underground areas is gone, though.
- Navigation, might want to set the maps to be entirely navigatable with both mouse and keyboard (I'd suggest key placement that enables one to move around with one hand in both cases).
- POIs, current flash viewer had weird thing where logged out it directs you to a new page while logged in it updates the flash viewer. Limitation arising from lack of cookies, I assume? (Mind the GDPR.)

Display-wise: I'd forgotten about the multi-level view; I'll let Jacob know and we can get support for that back in. Loading maps from custom URLs and zoomTo are what I've asked him to look at next. I could help him directly with viewer development, but I figure while development is fresh in his mind, I can focus on tidying up the site, and sort out the direct integration with PHP + Database.

Navigation: I've let him know about the onscreen mouse controls; I can always remap the keyboard in DFMA, and eventually make this a user-profile setting if people want.

POIs: Yes I hated that bug, I never got to the bottom of what caused it, some sort of browser change - the links should dynamically update the map, but they "fall back" to a full page refresh.

@Starver - that whole comment is fascinating. I can totally imagine it.

Wow, I had my heart all set to be broken, expecting this thread to be the announcement that the archive would be finally shutting down . . but no!
Same here. DFMA was one of the first DF sites I signed up on. Glad to see it's still alive!

Me too. Keeping the server alive is the easy bit; plenty of money, hardly any time - I'm working hard, sacrificing computer games at the moment.

Ditto.

Composed a few mails and then deleted them to Markavian with title: let's rewrite CMV viewer in js. Mostly because i'm not a webdev and my js would be no-libs-scary mess. However "Warmist is ecstatic upon hearing this news".

Edit: i wonder how possible would it be to integrate something into forums? Direct iframe with movie viewer would be awesome!

Oh please send the emails :D I'll add cards to the project page :) I think technology has moved on sufficiently to do a purely HTML/JS/Canvas solution - Jacob's map viewer is proof of that. My .NET CMV editor is... somewhere, that has a decoder in it. Found it: https://github.com/Markavian/CMV-Editor/blob/master/CMV/CMV.cs

Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Blomquistador on May 20, 2020, 10:38:02 am
Thanks Markavian for working so hard on the back end of things for this project.

Here's an update for what I'm currently working on with the viewer and what the future might hold.

In Progress
Spoiler (click to show/hide)

ToDo
Spoiler (click to show/hide)

Potential Added Features
Spoiler (click to show/hide)

Other Side Projects
Spoiler (click to show/hide)

Let me know if you have any suggestions for features/projects that might help grow the DFMA.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: lethosor on May 20, 2020, 10:02:57 pm
https://github.com/BenLubar/cmvjs is a pure-JS CMV renderer (if nothing else, the tile-rendering logic might be useful). I believe BenLubar was using it for some df-ai streams at some point.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Markavian on May 23, 2020, 06:27:09 am
https://github.com/BenLubar/cmvjs is a pure-JS CMV renderer (if nothing else, the tile-rendering logic might be useful). I believe BenLubar was using it for some df-ai streams at some point.

Can definitely make us of this; I'll see if I can get in touch with Ben before integrating it with DFMA - but the MIT license checks out - so should be able to do that so long as I link to the repo/license from the website.

Thanks @Blomquistador for sharing the update. I'm back to linting PHP this morning after a busy week at work; and waiting on that zoomTo feature for the map viewer MVP :D
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Blomquistador on May 26, 2020, 09:41:12 am
UPDATE

I won't be able to spend as much time as I have been on this. My brother has had a near-fatal accident and is in intensive care. Family comes first, you know how it is.

I've been currently just working on this in my free time at work. Once I go back to work I'll be able to work on the viewer a bit more.
Right now there is a "Zoom To" function but it is currently not functional.

Thanks everyone!
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Blomquistador on May 29, 2020, 01:20:38 pm
UPDATE 2

Looks like my brother will make a full recovery. All is well that ends well, as they say.

Anyway, I've updated the viewer and there is now keyboard-only input.
Use the 8.6.4.2 keys on the numpad to move one tile at a time. The 'z' key is a modifier which will cause it to move 10 at a time.
(I couldn't use shift due to the way HTML5 Canvas works with the DOM).

I've also added a "ZoomTo" function in preparation to implementing the POI system.

See it live at https://blomquist.xyz/dfma (https://blomquist.xyz/dfma)
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Markavian on May 31, 2020, 08:33:59 am
Looks like my brother will make a full recovery. All is well that ends well, as they say.

Thanks for the update Jacob; I wish your brother the best for his recovery.

I've managed to finish linting the DFMA code base; 106 files, 16,692 lines (up from 14,074 lines)... and am now in a position to put some basic tests around page generation. I've also found and fixed a few SQL bugs that I found while reading through everything line by line.

I think I'm now in a place to start surfacing the data and writing an integration with Jacob's viewer.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Markavian on June 10, 2020, 03:26:06 am
Update after a slow week(s)

Managed to get functional tests in place, supported by github actions... so the website now has some basic test coverage for surface level user journeys. It was very painful setting up mysql to run on a docker image using github actions; but I figured it out... so now I can spin up a version of the website and run tests against it as part of a continuous integration pipeline.

Meanwhile, a bit quiet from Jacob - I'm going to fork the viewer and rewrite the interface to make it easier to integrate with DFMA;
- Need a way to specify the map file (url) before load (customise the preload function)
- Need a way to navigate to a specific point on the map (using the zoomTo function)
- Need a way to customise key config to support a wider combination of keys (wasd, up,down,left,right, etc.)
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Markavian on June 11, 2020, 05:32:44 am
Progress
- See: https://mkv25.net/dfma/xdfmadev/parcel/index.html
- Source: https://github.com/df-map-archive/dfma-html5-map-viewer
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Uthimienure on June 11, 2020, 06:53:07 am
Progress
- See: https://mkv25.net/dfma/xdfmadev/parcel/index.html
- Source: https://github.com/df-map-archive/dfma-html5-map-viewer

Now that's cool.  Thanks for what you're doing  :)
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Fleeting Frames on June 12, 2020, 04:41:19 am
Drag and drop: Nice ease-of-use feature; I imagine that's going to be useful with mass-selecting and dropping a gallery.

Also found a "bug": xy remains constant: cue confusion with dropping 1x1 fort that was entirely out of viewrange.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Loud Whispers on June 13, 2020, 05:47:35 pm
posting to watch this noble endeavour
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Markavian on June 14, 2020, 04:51:50 pm
Done since last update:
- Fixed the NaN bug when loading starting PoI - turned out to be the canvas width that hadn't loaded yet; switched to fix value
- Created an architecture diagram for the viewer (https://github.com/df-map-archive/dfma-html5-map-viewer/tree/master/docs) - outlines plans for different components within codebase

In progress:
- Restructure file structure to match architecture diagram - made some good progress here, got separate 'Drag & Drop' and 'User Input', giving the 'Renderer' a clearly defined viewState... simpler code split across different files - missing some basic tests around new files

Do next:
- Support rendering front and side views

Code base is in a really healthy place; finding it easy to sit down and chip away and issues on the board now. Getting the infrastructure setup after 10 years of "its all in my head" helps a lot.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Blomquistador on June 25, 2020, 10:43:27 am
I'm Back.
I've created a new branch and PR on the repo to start work on front and side view as well as UI improvements.

Here's how it will work:

Spoiler (click to show/hide)
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Markavian on August 10, 2020, 04:02:32 pm
Right... I'm back too.

Long story; short version - successfully completed on a house purchase, moved in, got my new office set up.
Once house move settles down I'l be back into code - my focus is still fixing up the DFMA - so back to the project page to figure out what's next:


Got this diagram to explain the new architecture before I moved house:
(https://github.com/df-map-archive/dfma-html5-map-viewer/raw/master/docs/diagrams/architecture.drawio.svg)

So in a good place; but had to take a break to move house <3 🏡
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Fleeting Frames on August 10, 2020, 04:25:31 pm
Encouraging to see activity!

PS: Control layer or somewhere might want to include "Make POIs" & other such <add new map functionality>.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Salmeuk on August 16, 2020, 03:26:46 pm
Congrats on the new house!

I know we discussed this before, but are you planning to extend tileset support when the new graphical interface comes out? Also, have any of the newest devlogs shown features that might conflict with this current development?
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Markavian on August 23, 2020, 03:04:37 am
Hey all,

Project update:

- No progress, got in touch with Jacob via email

Up next:
- Deploy bundled map viewer to an agreed location that can be used by the DFMA PHP Website (https://github.com/df-map-archive/dfma-html5-map-viewer/issues/21)
- Use bundled map viewer with the DFMA Map Viewer behind a hidden feature toggle (http://Use bundled map viewer with the DFMA Map Viewer behind a hidden feature toggle)
- Support rendering front and side views (https://github.com/df-map-archive/dfma-html5-map-viewer/issues/12)

Personal update:

- Moving house has been a big distraction - finally got my room sorted out / setup / tidied - in a place where I think I can code again

RE: Comments

Quote
PS: Control layer or somewhere might want to include "Make POIs" & other such <add new map functionality>.

Perhaps - the intent of the control layer is storing the state of the map and moving things around - there is a separate data/domain layer that will be created to authenticate/authorise users to submit data to the DFMA database, which will be controlled by the API. So it might be that functions like "make POI" or "upload map" are handled by a separate system e.g. right now there's an entry point called "addmap.php" in a separate private DFMA PHP project that uses "addmap.form.inc.php" to handle that behaviour.

Quote
Congrats on the new house!

Thanks, its amazing~ worth working/saving up for :)

Quote
I know we discussed this before, but are you planning to extend tileset support when the new graphical interface comes out? Also, have any of the newest devlogs shown features that might conflict with this current development?

I would hope that extended tilesets are supported by default. The way the map encoder and decoder works is independent of Dwarf Fortress functionality. At present the main input is stacked bitmaps organised in tiles. The compression works best where there is heavy tile reuse. The more unique the maps are (the less tile reuse there is), the bigger the map file will end up. But because the map compressor uses the output pixels, it should support any tileset that Dwarf Fortress supports - if that makes sense?

If we allow people to upload different data sets, we'll need to store and render that data ... differently.

Quote
have any of the newest devlogs shown features that might conflict with this current development?

I am not sure; I need to read up on the devlogs, but what I've seen so far makes me think we'll be ok. Also, my current focus is on the historical map data - making that accessible for posterity - once that's fixed we can turn back in the direction of the future of the DF community and the archive.



Title: Re: Future of the Dwarf Fortress Map Archive
Post by: MiguelT on March 11, 2022, 09:36:12 pm
Sorry for necroing.

Is this project still alive?

I've been using the DFMapCompressor but I miss the DF Map Archive.

I'm asking just to know if I can have some hope, rs.

If it is the case, well, it was great back in the days, thanks.
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: Blomquistador on May 14, 2022, 07:19:47 pm
Is this project still alive?
Yes! Progress has been slow but we are continuing to develop this.
As we speak I am working on UI buttons for the map viewer.
After that, I will dive into front/side views and optimization potentially.

I apologize for being absent for the past while. Life has a way of getting away from you.
Looking forward to continued progress.

Time to ☼STRIKE THE EARTH!☼
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: maximaximal on July 14, 2022, 05:17:19 am
So, after some small commits to the new JS-based web viewer, I present a larger contribution, which hopefully makes using the map compressor more accessible to a broader audience.

There now is DFMapCompressorPP (https://github.com/maximaximal/DFMapCompressorPP). This is a new implementation of the good old C#-based map compressor. The benefits of the new version are:
I created this new tool to give some more love to the very nice .fdf-map format. With this, there are less barriers to entry to compress map images, also for Mac and Linux users. Just in time for the steam version releasing soon :D
In the future, I'd also invest into a native map viewer based on SDL2, which runs locally also without a browser, to fully replicate the original map compressor's features. This should then also easily enable different viewing modes of maps.

Additionally to the native compressor, I also created a PHP-based renderer (https://github.com/maximaximal/DFMapCompressorPP/tree/main/php) for .fdf-map files. This could in the future be used to enable social-graph-based previews for the DFMA. If anybody wants to try stuff out with this PHP implementation, I would of course be happy about it! The PHP file also supports downloading maps from external sources, so when giving it the correct URLs, it directly renders files from the official DFMA page.

I would be happy for inputs and your experiences with these new tools! The web-based compressor is a product of the last few days, if it works well for you shoot me a message or post a reply! :)
Title: Re: Future of the Dwarf Fortress Map Archive
Post by: clinodev on July 15, 2022, 04:32:24 am
[I accidentally posted in the wrong thread, sorry!]