Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 [2] 3 4

Author Topic: DFFD Downtime Explanation  (Read 16635 times)

Janus

  • Bay Watcher
  • huffi muffi guffi
    • View Profile
    • Dwarf Fortress File Depot
Re: DFFD Downtime Explanation
« Reply #15 on: November 19, 2022, 01:15:09 pm »

Wow. MySQL is that bad? Or is this a Virtuozzo VServer or similar/file system/underlying issue?
As mentioned, there's a fair chance that the database was initially recoverable, slightly damaged but likely repairable. A further mistake on my part made in haste (which I won't go into) rendered it ultimately unrecoverable.
Logged
Tomas asked Dolgan, "What place is this?"
The dwarf puffed on his pipe. "It is a glory hole, laddie. When my people mined this area, we fashioned many such areas."
     - Raymond E. Feist, Magician: Apprentice  (Riftwar Saga)

Ralpha

  • Bay Watcher
    • View Profile
Re: DFFD Downtime Explanation
« Reply #16 on: November 19, 2022, 03:47:24 pm »

It looks like the Wayback Machine indexed the site quite a few times. Although this does not include the download themselves.
https://web.archive.org/web/20220905221931/https://dffd.bay12games.com/index.php
(last update September 5, 2022)

Where the archive files also stored in the DB? (most likely not)

I'm curious how big the archive of the whole server is? How many GB have been uploaded?
Logged

Gruust

  • Escaped Lunatic
    • View Profile
Re: DFFD Downtime Explanation
« Reply #17 on: November 19, 2022, 03:53:35 pm »

file creation times might help to rebuild the IDs...
Logged

Janus

  • Bay Watcher
  • huffi muffi guffi
    • View Profile
    • Dwarf Fortress File Depot
Re: DFFD Downtime Explanation
« Reply #18 on: November 19, 2022, 06:14:18 pm »

It looks like the Wayback Machine indexed the site quite a few times. Although this does not include the download themselves.
https://web.archive.org/web/20220905221931/https://dffd.bay12games.com/index.php
(last update September 5, 2022)

Where the archive files also stored in the DB? (most likely not)

I'm curious how big the archive of the whole server is? How many GB have been uploaded?
The Wayback Machine archive could be useful for people to recover the file descriptions and some other data for whatever they cover. Technically I might could work up a script to scrape certain info for each file listing and user, assuming they're all stored there, but it would take a ton of work and time. And a lot of it would then require further processing. Like file descriptions needing to be converted back to BBCode from HTML; yuck. And since the user information wouldn't have their email address or password, there would be no reliable way to recover the actual accounts, so no real value to those.
Probably simplest for me to just provide a link the the Wayback Machine for people as a way to recover their listing data manually.

The files aren't stored in the database. As mentioned, all of the up to date user uploaded files are safe and will be maintained. The collective size of all the uploaded files is about 372 GB.

file creation times might help to rebuild the IDs...
No need, the user uploaded files are stored with their file ID as the filename on the server. The actual filename which you end up downloading them as is stored in the database.

And on that note, slightly better news. Since I have the abbreviated JSON file data available for each file, stored as actual static files, I should be able to use the data in those to restore some information to the database for each currently orphaned upload. The correct download filename, the author's name (not their ID, but I can match those at least for user accounts that existed back in 2015, otherwise just show an unlinked username), the last update timestamp, the file version if one was provided, and the DF version the file was for. And the file size, which could alternately be determined by checking the file itself, but no need with that info available here.
It will still be missing their file listing title, description, category ID, original upload date, file homepage info, rating, download and view counts, and a couple other bits of info.

So for orphaned files uploaded by those user accounts that existed back in 2015, I should hopefully for the most part be able to link those file listings back to your accounts.

The work continues.
« Last Edit: November 19, 2022, 06:36:25 pm by Janus »
Logged
Tomas asked Dolgan, "What place is this?"
The dwarf puffed on his pipe. "It is a glory hole, laddie. When my people mined this area, we fashioned many such areas."
     - Raymond E. Feist, Magician: Apprentice  (Riftwar Saga)

txtsd

  • Bay Watcher
    • View Profile
    • I have a quest!
Re: DFFD Downtime Explanation
« Reply #19 on: November 21, 2022, 12:20:47 am »

Sucks when things like these happen. I can only imagine the toll it's taking on you.

Please let me know if you'd like help setting up the new server, maintaining up-to-date security standards, or just menial help in general.
Logged

Xelor

  • Escaped Lunatic
    • View Profile
Re: DFFD Downtime Explanation
« Reply #20 on: November 21, 2022, 06:23:16 am »

Wow i have 0 mods as i just got into this game really a darn shame i don't get to play with all these epic mods for vanilla :/ please seend help!!
Logged

Gruust

  • Escaped Lunatic
    • View Profile
Re: DFFD Downtime Explanation
« Reply #21 on: November 21, 2022, 06:44:06 am »

There is a starter pack avl from other file servers, too.
Logged

Janus

  • Bay Watcher
  • huffi muffi guffi
    • View Profile
    • Dwarf Fortress File Depot
Re: DFFD Downtime Explanation
« Reply #22 on: November 21, 2022, 07:43:57 am »

I'm down to tying up a few loose ends. The older file listings which had updated files since 2015 have been updated to indicate the proper file update date. The missing newer file listings have been recreated based on available data on the server. Well, other than 24 remaining files that I need to look into which had no stored JSON data but do have a download file, somehow.

Still a few more things to go before I'm ready to reopen the site. Maybe later today, probably no later than late tomorrow.
Logged
Tomas asked Dolgan, "What place is this?"
The dwarf puffed on his pipe. "It is a glory hole, laddie. When my people mined this area, we fashioned many such areas."
     - Raymond E. Feist, Magician: Apprentice  (Riftwar Saga)

gendernihilist

  • Escaped Lunatic
    • View Profile
Re: DFFD Downtime Explanation
« Reply #23 on: November 21, 2022, 11:26:39 am »

I'm down to tying up a few loose ends. The older file listings which had updated files since 2015 have been updated to indicate the proper file update date. The missing newer file listings have been recreated based on available data on the server. Well, other than 24 remaining files that I need to look into which had no stored JSON data but do have a download file, somehow.

Still a few more things to go before I'm ready to reopen the site. Maybe later today, probably no later than late tomorrow.

Thank you for your hard work!
Logged

darkhog

  • Bay Watcher
  • JAGIELSKI is PERFECTION
    • View Profile
    • Jagielski Gaming YT channel
Re: DFFD Downtime Explanation
« Reply #24 on: November 21, 2022, 03:17:20 pm »

Maybe the DB could be "recovered" by parsing HTML dumps of the site from archive.org? As in, parse page for each file, then extract every bit of info from the HTML in order to rebuild the DB. Basically the same way you build the HTML representation of the site that then gets sent to the browser, but in reverse.
Logged
I am a dwarf and I'm digging a hole. Diggy Diggy hole, diggy diggy hole.

If I say something funny, don't ask if you can sig it. Just do it (though credit me).

Janus

  • Bay Watcher
  • huffi muffi guffi
    • View Profile
    • Dwarf Fortress File Depot
Re: DFFD Downtime Explanation
« Reply #25 on: November 21, 2022, 03:28:17 pm »

Maybe the DB could be "recovered" by parsing HTML dumps of the site from archive.org? As in, parse page for each file, then extract every bit of info from the HTML in order to rebuild the DB. Basically the same way you build the HTML representation of the site that then gets sent to the browser, but in reverse.
The only thing useful to extract like that at this point would be the file's title, description, and homepage link. Even then, I've looked over some file listings over there and there are many that were never archived. Others only have a fairly old archived copy that is very likely out of date as well.
I don't really see the point in putting in work and time to scrape for that. I'll be providing a link for people to check their file listings there in case they want to manually update their file listing.

EDIT:
One other potentially useful thing I forgot to mention above is the category the file was in. Still not worthwhile in my opinion; I have all of the orphaned files in a new category to differentiate them anyway.
« Last Edit: November 21, 2022, 03:51:51 pm by Janus »
Logged
Tomas asked Dolgan, "What place is this?"
The dwarf puffed on his pipe. "It is a glory hole, laddie. When my people mined this area, we fashioned many such areas."
     - Raymond E. Feist, Magician: Apprentice  (Riftwar Saga)

Janus

  • Bay Watcher
  • huffi muffi guffi
    • View Profile
    • Dwarf Fortress File Depot
Re: DFFD Downtime Explanation
« Reply #26 on: November 21, 2022, 04:15:44 pm »

The site is now open again.

You may be interested in some Q&A about recovering your account or file listings.

Note that all orphaned files are in a new category.
Orphaned files do have the name of the uploader, and if their account exists from back in 2015, they retain ownership of those listings. Otherwise, see the above Q&A link for info on reclaiming file listings.
Older file listings that were rolled back to their 2015 state do have correct new update timestamps, download file related info (size, filename, SHA256), and file version and DF version info. Other info in these listings remains as it was back in early 2015.
Logged
Tomas asked Dolgan, "What place is this?"
The dwarf puffed on his pipe. "It is a glory hole, laddie. When my people mined this area, we fashioned many such areas."
     - Raymond E. Feist, Magician: Apprentice  (Riftwar Saga)

Gruust

  • Escaped Lunatic
    • View Profile
Re: DFFD Downtime Explanation
« Reply #27 on: November 21, 2022, 04:25:18 pm »

TY!
Logged

brewer bob

  • Bay Watcher
  • euphoric due to inebriation
    • View Profile
Re: DFFD Downtime Explanation
« Reply #28 on: November 21, 2022, 04:39:17 pm »

Thanks for all the effort!

voliol

  • Bay Watcher
    • View Profile
    • Website
Re: DFFD Downtime Explanation
« Reply #29 on: November 21, 2022, 05:36:32 pm »

Thanks for all the work! It's nice seeing links to orphaned files seemingly still work, even if the file descriptions are gone. :)
Pages: 1 [2] 3 4