Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

It works for me.

6 (31.6%)

It does not work for me.

1 (5.3%)

I'd be willing to help test it.

6 (31.6%)

I'm not interested in testing it.

1 (5.3%)

I might be willing to help test it.

5 (26.3%)

Total Members Voted: 14

[Angle]

I'm tempted to just start with something simple like SocialDebate or TrueDebate - we can easily change the name at this stage.

Remember that this is intended for more than just debates- I also want this to be useful for things like what we're doing right now - planning projects, discussing ideas, etc. Really, I'd like to call it dscourse, but someone already made off with that name.

Also, Angle, could you update the OP with the google doc and other information? We should sig this and make it obvious that it's open-source and open to participation

Done!

[Anvilfolk]

The server could also host the web front-end and manage users and their authentication.
It would also make changes to the DB through an API accessible from the front-end.
+calculations

The interfaces grabs information from the DB and shows it to the user.

How would the interface authenticate with the DB?
If the interface can directly change the DB (voting for example) you could use the login/password from the interface to make any changes you want.

I feel like all of this functionality - authentication, getting information and modification - should be in the libraries we provide to users, including authentication. I don't know too much about this, but it'd be kind of like a PHP session. When you authenticate, you get a "ticket" which is yours alone, and valid while you keep using it. It times out after a little bit, or you store that in a cookie-like situation. Encrypted communications should ensure nobody else gets that ticket.

Having a web frontend makes everybody HAVE to use that particular webserver. I don't see why this can't be handled by whatever interface you decide to use If you're using a web interface, you use the web login. If you're using it on your smartphone, you login from your smartphone's application.

I have no experience at all with server-client relations and all of that stuff, but I think computation should not be done by the server but the client. When more and more people start accessing the server you would probably get serious performance problems pretty soon.

You can do client-side computation for things like, "how do I display the debate", but for calculating debate outcomes you need something dedicated. Like BoboJack said, different clients have different architectures, and they give different results. Interfaces are also proned to be hacked (especially if they're open-source...), so that you can't really trust them. At the very least, you'd need server-side verification of the the results, which is actually faster than calculating everything, so that is an option.

It all amounts to the underlying infrastructure. A debate for 3 people can run everything on a mobile phone. A debate on Bay12 needs more infrastructure. It's up to whoever is hosting to handle that.


The algorithm we have right now does not handle incremental changes. There's some minor optimisations you can do, but we still need to recompute almost everything whenever there's an update (including a single vote). This is a limitation that it would be amazing to get over, so I can put up a description of the algorithm at some point for you to review. The idea's actually pretty simple. You can modify the algorithm, but you need to prove that it'll still do what we want.

I think at the moment we can handle a few thousand arguments in a couple of seconds. Like I've told those I've talked to, we've had someone working on optimising these things for a few months, and this is what he came up with, so I believe at the very least it's a good starting point. Eventually people should be able to use higher-end machines with multiple cores (the algorithm is highly parallelizable), which should make things better.

Remember that this is intended for more than just debates- I also want this to be useful for things like what we're doing right now - planning projects, discussing ideas, etc. Really, I'd like to call it dscourse, but someone already made off with that name.

I was using debate in the wider sense - any discussion where points are made and argued over. For instance, I'd say we are engaging in a nice debate about how to organise an application. I suppose people might have some fairly strict notion of what a debate is. We should definitely include the idea that it's "social", because of the voting and the fact that outcomes depend on the context in which discussions are carried over.

Funny how coming up with a name is usually one of the hardest parts



Gimmie a few minutes and I'll put up a little infrastructure diagram of how I think most interactions can go.

[Anvilfolk]

Here we go:

Spoiler: Overall architecture (click to show/hide)

Let me explain what's happening there.

Lots of users are accessing Server #1. Each of them can be using a different application, or interface, as we have been calling them. To obtain information from or modify a debate (I'm used to using this term, sorry!), the applications/interfaces use a very small client-side library with minimal functionality. You can log-in, ask for debates, and submit modifications. That's should be almost everything.

Once the requests arrive at a server, there is a server-side interface. This parses the requests, and talks to the database. It makes sure logins are correct, that votes are accurate, etc.

In this case, Database #1 contains a small debate that is being affected by a debate stored in Database #2. Upon identifying this, the server-side interface from Server #1 will figure out where the other debate is stored (Server #2), and talk to the interface responsible for it to get the necessary data. ***

There is only one SAA Computation node, who is responsible for computing debate outcomes. He is authoritative over both Server #1 as well as Server #2. That means he will actually talk to both databases and recalculate debates within those databases.



*** An alternative version to this is that the client finds out that there are relevant arguments stored in other databases, and the client goes and grabs them. This removes communication overhead from the server-side interface, of which there can be a LOT if there are many clients logged on. Instead, each client is doing a little bit more work, which is fine. DNS has similar mechanisms. We can figure out which one we want to implement, and possibly both.


The difficult part will be figuring out how SAA authorities work together, and how those edges between databases under different SAA authorities are dealt with. At this point we have our hands full though, and I believe that is something we can deal with in the future.



I hope the diagram makes these ideas a lot clearer to everyone! Let me know if you have any questions!

Also, the diagram was made in yEd, and I can provide the base file if you want to play around with it.

[Angle]

Right now we're considering what to name the project. I peronally lean toward Agora, which is the greek word for forum.

[Anvilfolk]

Working name GO!

I'd love some feedback on the architecture. I've put that up along with a description in the online google doc. We are creating a GitHub organisation for this, and from there we can create repositories for all major components!

[BoboJack]

I like how the calculation server is different from the server side interface.
I'm not so sure about the database set up.
If database #1 and #2 are referencing the same big web, why are they separated?
Every query done by the server side interface now has to query two databases.
Communication/keeping things consistent between the databases seems really hard and would have to be hand coded.
I would rather follow these instructions.
We would have one logical server, to which all the server side interfaces reference to
and the logical server would be made out of many servers which would communicate in one of the ways described in the link.

[tompliss]

I love "Agora"

About the computation thing, remember one thing, guys : you don't want the user to download every argument at once in his client for it to filter out what he want to read.
Pretty much the same idea about having pages in a classic forum thread.

[Anvilfolk]

Good point, BoboJack!

That's not exactly what I was thinking of here though. What we should make possible, eventually, is for an argument here on Bay12 to reference or be referenced by something on some other website unrelated to Bay12. More specifically, imagine that Agora has been deployed at bay12forums.com and at nytimes.com - there are lots of topics in common in both. I would love to be able to use a nytimes.com argument to attack a bay12forums.com argument, for instance. If you can do this, all of a sudden Agora is infinitely more powerful and interesting. This goes hand in hand with the development of the Semantic Web, which is huge in academia right now (including the guy who made the WWW), and is often said to be the future of the internet.

However this should definitely not be a major concern for initial versions, but we should keep it in mind as we develop the architecture.

What you suggest with PostgreSQL is wicked awesome though. I imagine we can definitely have a package that installs Agora on a PostgreSQL database rather than a mySQL one. As long as there's some install script we can provide with it, and a server-side interface that knows how to communicate with it, it sounds like postgreSQL could be more efficient! That's the beauty of server-side interfaces. They hide whatever is behind!

tompliss: yeah, I figure we should definitely be able to keep some forum-like organisation as well. I am guessing that we might use JSON/BSON like BoboJack suggested to store this type of structure. Because JSON is so flexible, it can also be used to store different structures. But again, this needs a whole lot more thought. We need to adapt to forum-like structures, article-like structures (news sites, blogs), etc.

I think the client-side library and the server-side interface should have some sane default values as to how many arguments you can download at once.


Quick notes:
- Angle created a GitHub organisation for the project. Post your GitHub id on the google doc "participants" section and you'll be added.
- If I am being too verbose, let me know and I'll cut down on it.
- Once Eagleon responds, and if he agrees with the architecture, let's go ahead and create the repositories for each component and start a'hackin'

[BoboJack]

That's not exactly what I was thinking of here though. What we should make possible, eventually, is for an argument here on Bay12 to reference or be referenced by something on some other website unrelated to Bay12.

So you want something like the pods from Diaspora? Is that even possible?
How would searches work? Every search by a user makes a request to all pods?

let's go ahead and create the repositories for each component and start a'hackin'

We still need to decide on one language/framework to use.

[Anvilfolk]

You've always got one entry point. If you're part of the bay12forums.com community, you contact the bay12forums.com server.

However, Bay12's database might refer to NYTimes arguments. If argument IDs contain location information, this becomes much easier, and kind of like how DNS works. Hence my suggestion of a node ID being D1:A1, for argument ID 1 from database ID 1. D1 could be "nytimes.com", and A1 could be a random number. Thus, when you see an incoming attack from D1:A1, you know you need to contact the nytimes.com authorities to get information about that debate.

Like I said, contacting "secondary" servers when browsing debates could be done either server-side or client-side. I'd go for client-side to avoid overworking servers.


As far as the near-future roadmap, here's my suggestion:

• Develop an independent system, i.e. don't try to integrate it into SMF, phpBB, Blogspot, etc, yet
• Go for the path of least resistance tech wise: mySQL is the most easily accessible database, and for languages we should use our available skillsets
• Before implementing anything, we need the database schema, including how we store the forum-like structure
• Once the basic database structure is up, we need the communication protocol
• With the communication protocol, implement ONE server-side interface and a couple of client-side libraries in different languages
• Implement one or two client interfaces

The SAA stuff I can probably code super quickly, so don't worry about that.

Right now we need to figure out database and comms protocol.

[Eagleon]

- Once Eagleon responds, and if he agrees with the architecture, let's go ahead and create the repositories for each component and start a'hackin'

Erp - I didn't realize it hinged on me. Absolutely, I have no complaints as to the organization, but for one thing - in the doc you mention "Server-side interface is the “site layer”" but everything below Database (starting at Site Layer) was me determining what the database needed to send out to the clients. Rather than send out site-level information within each node, it made more sense to encapsulate all nodes and edges from a site with that information. It's a quibble (I'm not disagreeing to a server interface), but it'll help if we end up having multiple different types of 'sites' (could end up accessing offline databases in specific use scenarios) and reduces redundancy in the other objects.

I love "Agora"

About the computation thing, remember one thing, guys : you don't want the user to download every argument at once in his client for it to filter out what he want to read.
Pretty much the same idea about having pages in a classic forum thread.

It should be possible to send out portions of the graph in little sub-networks, by picking a node, traversing a certain distance, and sending one more set of edges past that number of nodes in the graph. The server could also operate on these sub-networks as they're built and packaged, to do fancy math things, but anyway- So if the client can handle having dangling edges to unknown groups of nodes, it can either ignore them or use them to request more. The problem is this adds more for the server to do - processing at the expense of bandwidth. My mind is going to fractal database formats O.o Not helping, brain.

You also run into synchronization problems if nodes or edges are destroyed, which should only be confusing to the client, not the server. The server would just have to be telling the client whether the changes were successful, or blocked by an inconsistency.

PostgreSQL talks about load-balancing - if the server load is heavy, maybe we could even do this automatically based on demand, only giving the client as much of the local graph as the server can keep up with and still provide other active users OK service.

[Anvilfolk]

Eagleon: no worries! Since everyone has their own specific ideas, we just need to make sure everyone's in agreement before turning proposals into decisions.


By site-level information you mean a forum-like structure, or a newsite like structure, or blog-like structure, right? If so, I think the current proposal by BoboJack where that information is stored in a flexible JSON format column in the argument/node table in the database should work for what you mean.

When you access the server-side interface, you can request the "meta-information", which clients should know how to parse easily.

I feel like a generic hierarchical organisation could be enough for most sites, if not all, sites. Let's call these "forums", to keep with intuition. Every Agora-powered website has a main "forum". Any forum can have subforums, which are also forums. Any forum can also have "threads". Threads cannot have sub-threads or sub-forums. Instead, all arguments are proposed within a thread.

This mimics a forum like Bay12 perfectly. It should also be able to capture a blogging website, where the Blog is a forum, and each article is directly a thread (which is used by the poster, and commenters alike).

We can also say that a thread can belong to multiple forums, which allows us to implement a blog post as belonging to several "categories".

[Eagleon]

Sounds great. I just want to make sure the database structure is rock-solid, since I know that updating them to a new structure can be nightmarish, and then you have to make sure the clients are reading it right, etc, etc. I haven't had time to look at JSON, should probably get on that.

Also - the google doc is getting to be quite slow to load, maybe we should split it off into documentation for the different sections (architecture proposals, database, interface, project overview and outline/references to each). That way people without github can quickly see roughly what's going on - or does their wiki system allow anonymous browsing?

[Anvilfolk]

We just created an empty GitHub repo to host the wiki (turns out GitHub Organizations don't have their own wikis, only their repos do). It's Agora-Project/AgoraDevDocs.

I'll try to create a couple of empty pages to set up the structure, and we can start migrating there. Since it's open-source, I can't see why it wouldn't be public!

[Anvilfolk]

Here's a proposal for how we should organise further developments.


The Google Doc is where we post and discuss proposals more persistently. We can use the really cool comment system to identify and resolve issues as they come along.

Once there is a quorum on a proposal, it gets transferred over to the wiki, and becomes an official decision.


If you guys agree, we can start transferring and removing things from the Google Doc into the wiki. Which, by the way, is becoming really cool