Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[funkydwarf]

Hello everyone!
First off you shouldn't waste your time reading this. If you do, thank you ahead of time.  Ok now I wont feel bad writing alot! It kinda needs some back info.

 
Ok, I got these three industry web pages I log into cause I am stupidly traveling the path to become an actor in film and tv in Los Angeles. Yes yes, impossible Im dumb, millions of people trying,etc,etc.

Anyhow part of the work part of acting is monitoring three industry webpages to submit for parts. Actors access, backstage, and la casting. So they are not the most modern pages. They do weird things like use a shopping cart to select parts to apply for then "checkout" for money or free with monthly sub.  Not the money part, everyone want to extract from the acting throngs. Just the weird use of 2002 era web shopping cart code.

A big part for my , no credit, small time butt is submitting like 5 mins after everything gets posted.

So I am trying to setup an rss feed type deal where I can just leave some client running to watch all three. Thing is its like a huge amount and I just want the filtered ones that fit my description, and each site has a built in way to save a default filter.

I made a shortcut with three destinations and it opens all three webpages, But I need to click various amounts of logins and the filter page gets a click for "filter by these" for each to get the list up. If I link to an address already filtered it stops me at the login page, different problems for each of the three.

So I want to make a client or script I can run to open all three windows, filtered and logged in.

So is this something I could use python for? I thought of an autoit script, but that would be a pain and thats a hacky way to deal. Although simple and in my wheelhouse already.

Python? Go? Java? C#?    I finally got off programming tutorials and made my own thing from scratch in autoit, and feel like I am ready for the next level and thought some sort of alert-er web browser would be a good next thing.

Maybe I should fork an open source browser and hack it into what I want. 

I dunno. I know there are alot of varied programming experienced people in all sorts of specialized languages around and would love to hear some opinions.

If you read my drivel, thank you very much!


Oh duh, I run windows on my main so cross platform is best but I think I would build this for my debian server to run all the time and email me when a new one comes in.

Thanks Again guys!

[Ire]

I used to be an adventurer like you until I put an adderall into me

[Reelya]

I'd recommend Greasemonkey if you use Firefox, or Tampermonkey if you use Chrome. They're extensions.

Then you write your code in Javascript as an extension to the page, and it can refresh the page, click on things etc. It's handy because it's all sitting on top of the client-end of the page so you don't have to do any server trickery and it just appears to be a normal user. You could do this as server-side stuff logging into those pages, however, then you're doing "browser spoofing" which is more clearly a type of attack and could get blocked by Cloudflare and stuff. It's much easier to just have an actual browser tab open that's reading the page and periodically refreshing things. It looks more "normal" to the other server. If you can have a browser actually running then it makes things much simpler, like I said, by writing light-weight page extensions in JavaScript for the page you're targeting.

Then, you have a server page that the Greasemonkey/JavaScript is talking to via Ajax / xhttml requests to upload any gleaned data. That can get as fancy as you like and could be written in anything.

I do this stuff all the time, PM me and I'd help you out.

[funkydwarf]

Of course theres some sort of extension for this! Thanks for that, I will def check out grease monkey, it sounds perfect.  I am so daft! Thanks for the offer of help, I'll get to it and if I have any problems and google and stacktrace come up dry I may take you up on that!

[Reelya]

Let me give you some starting pointers since I've been doing this a few years:

you have something like this at the top of your greasemonkey script:

Code: [Select]

// ==UserScript==
// @name          Bay12 Topic Filter
// @namespace     *
// @include       *www.bay12forums.com*
// @grant         none
// ==/UserScript==
Then whatever follows is your JavaScript. Anything that's sitting there and not in a named function will be excuted on page load. However, running things directly on page load can be iffy, depending on the page, because some pages use their own JavaScript to load resources after the base page is loaded. So for those instances, you need to use timeouts / timers to run at intervals and look for things that have loaded. You can have this statement:

  setTimeout(MyFunction, Milliseconds);

And it will run your MyFunction after # Milliseconds. To repeat a check you can do two things:

  setInterval(MyFunction, Milliseconds);

Or

function MyFunction()
{
    // do stuff
 
  setTimeout(MyFunction, Milliseconds);  // call MyFunction again a bit later
}

As for interacting with elements on the page, the most general way to find stuff on the page is to use the document.querySelector (gets the first match) or document.querySelectorAll (gets an array of matches) commands. This is good since you only need that one command then, and you can base everything off the css selectors info from web sources. There are also a suite of commands of the type "GetElementByID" and "GetElementsByID" but I never use those, as they're exactly the same as just querySelector("#ID") or querySelectorAll("#ID"), so using querySelector exclusively just mean many less commands to memorize.

So you need to manually load the page, use the element inspector to look at the elements you want to read then work out what series of querySelector commands and JavaScript will isolate the element you want to get data from. The whole platform lends itself to rapid iterations however. edit and save your script, then refresh the page, note any issue, then iterate.

What you get back will be a DOM element or an array of DOM elements (for querySelectorAll). You should check for null here, since "no match" will return null and using a null variable will crash your script (however any callbacks you set up will still happen). Once you've got DOM elements that are non-null then you can use "." properties such as ".innerHTML" and ".value" to extract or change their properties.

Use "alert" or better "console.log" to make printouts for debugging purposes. Shift+Ctrl+J is the browser console in Firefox, not sure about Chrome. You need to have that console information so you can get diagnostic info about any errors you've incurred.

[funkydwarf]

Wow. You have really saved me some time here, thanks so much! I'm gonna hammer this out after I finish up an audition today and then make a self taped audition..for what i need you pretty much laid it out,i justto need to work through the logic,( ie the fun part ((no,not !!fun!!)))I really appreciate the info, it will be put to good use.

Thanks again!