Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Parisbre56]

Here's a Java program that transcribes any thread of this forum into a single file for offline reading! It processes about 260 pages a minute. Time might vary depending on connection speed and site traffic, as well as your preferences (check the menu).

The program can also be ordered to only keep the posts of certain users, great for reading only the GM's posts or finding the Toad's posts in the Future of the Fortress thread.

The program can combine multiple threads into one, arranging the posts chronologically.

If the output file ends in .html, the program will split the output into multiple pages based on user preferences (check the menu).
If the output file ends in .txt, the program will keep the output as a single plain text file, great for reading in an e-book or for processing with regular expressions or programs like grep.

Here's the program: B12_PostProcessor.jar (10 MB)
Should work on most common OSs (Windows, Linux, Mac) and architectures (x86, x86_64) with an up to date Java VM installed.

Spoiler: To do list (click to show/hide)

Bugfixing
Rewrite it to make the code more manageable.
Make it understand and store more info.
Make it download multiple pages at a time to save time, since processing a single page doesn't take long. It's establishing a connection and downloading that are the real time sink. Done
Make it able to login. Done
Make it able to download images. Done
Make it download theme information and images so that things don't look so black and white when offline. Done; needs some bugfixing but nothing serious

Long term:
Make it combine multiple threads into one.  Done; needs to understand dates other than %a %d-%m-%Y, %H:%M:%S
Make it prettier.
Option to divide output to multiple files.  Done
Filter options: Remove OOC (()), Only include posts with certain Text types (Italics, Bold, Underlined, Coloured, etc), Include posts with character speech "", mix and match rules
Output options: minimize post size,
plain text output Done; maybe add a few more options and clean up output

Very Long term
Make it able to work on other forums.

Here's the source code if anyone wants to mess with it. (30 MB, Eclipse project)

Changelog:
1/4/2014:New version can combine multiple files into one based on time. Unfortunately, it can only process times in the %a %d-%m-%Y, %H:%M:%S format. So make sure to log in and change your date format to that if you want to use it until I get around and fix it.
28/3/2014: Ability to split output to multiple files. New options menu. Various bugfixes.
2/11/2013: New version can save output as a "lightweight" .txt file.
1/11/2013: Fixed an out of memory error that occurred in 32-bit windows JVMs when processing more than 1069 pages
23/10/2013: New version can login and can download images and forum theme images. It also utilizes multiple downloader threads to reduce download time. Finally, it has a better GUI.
11/10/2013: New version should work on most common OSs (Windows, Linux, Mac) and architectures (x86, x86_64) with an up to data Java VM installed.
10/10/2013: Made the program create a window that acts as a terminal. This means that you can now just double click the file instead of having to launch it from the terminal. Bad thing is, it only works on 64-bit Linux now. I'll fix that tomorrow. Should be an easy fix (famous last words).

[Xantalos]

Hells yeahs PTW.

[miauw62]

Does this work for all Simple-Machine powered forums? (or could it be editted to do so?)

Either way, this is fucking awesome.

[Lectorog]

Man oh man. This has so much impractical use. PTW. I'll be actually using it if optional theme formatting and image downloading is implemented.

[Parisbre56]

Does this work for all Simple-Machine powered forums? (or could it be edited to do so?)

Theoretically, with a bit of modifying, yeah, but I haven't tried it yet.

Changed the first post.

EDIT: A new version is up!

EDIT2: Updated with some speed ups and bug fixing.

[kisame12794]

You glorious bastard. PTW.

[Parisbre56]

New version is up. It should now work on most common OSs (Windows, Linux, Mac) and architectures (x86, x86_64) with an up to data Java VM installed.
Tested it on 64-bit Windows and Linux. Tell me if it doesn't work on your system and I'll see what I can do.

It took some messing around with ANT and the help of those two pages:
http://mchr3k.github.io/swtjar/
http://timeme.eclipselabs.org.codespot.com/git.wiki/SWT.wiki
Check them out if you're planning on creating a cross platform Java app that uses SWT to generate its interface. Or just copy my source code and rewrite the Main class to suit your needs.

Now to start rewriting the program to make it "smarter".

[Person]

I think I'll be having some forum games to read through once this thing does images.

[Elephant Parade]

This does look interesting.

Edit: Now to use this to read the Homestuck thread. Except not, because that would take far too long. I like how it preserves the formatting.

[Parisbre56]

Spoiler (click to show/hide)

Version 2 is coming along nicely. At this rate, I should be ready for a release near the end of the week or sometime in the next week. Most of my work so far has gone to designing a simple GUI and a login system. Next up is upgrading the downloader itself to download images, downloading through multiple connections to increase download speed and making it smarter so that it can filter messages according to their content.

Speaking of filters, as of right now, I'm working on two filters. One that removes out of character content by removing any text enclosed in (( )) and another that checks for types of text like Italics, Bold, Underlined, Coloured, etc. Any ideas for other filters you would like to have?

[sjm9876]

This looks very interesting. PTW

[miauw62]

Removal of all the shit inbetween posts. Just a name and a link to the profile, maybe an avatar instead of a name, an image, contact details, the topic reply name, the date of the reply, the signature, the little report to moderator and IP logged thingie in the bottom left etc etc.

[Armok]

This is great

If we're making wishlists, what I could really use is if it striped away all the formatting and signatures and junk and outputted a plain textfile viewable on kindle. Just name, post date, main text of post, repeat.

[LordSlowpoke]

PTW

[Parisbre56]

Bug squashing took more time than expected (mostly due to personal issues and my lack of experience with working with SWT), but lo and behold:

Spoiler (click to show/hide)

A speed of about 260 pages per minutes (which translates to 3800 posts per minute with 15 posts per page, speed should increase with higher settings) with 6 threads downloading and processing data (I'll probably add an option to increase the number of threads).
Now all that's left is the ability to download images and it'll be ready for the next release.