Slashdot Log In
Algorithm Rates Trustworthiness of Wikipedia Pages
Posted by
CowboyNeal
on Fri Aug 31, 2007 06:38 AM
from the getting-it-right dept.
from the getting-it-right dept.
paleshadows writes "Researchers at UCSC developed a tool that measures the trustworthiness of each Wikipedia page. Roughly speaking, the algorithm analyzes the entire 7-year user-editing-history and utilizes the longevity of the content to learn which contributors are the most reliable: If your contribution lasts, you gain 'reputation,' whereas if it's edited out, your reputation falls. The trustworthiness of a newly inserted text is a function of the reputation of all its authors, a heuristic that turned out to be successful in identifying poor content. The interested reader can take a look at this demonstration (random page with white/orange background marking trusted/untrusted text, respectively; note "random page" link at the left for more demo pages), this
presentation (pdf), and this paper (pdf)."
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Light Bulb Moment (Score:5, Funny)
algorithmic argumentum ad verecundiam (Score:2)
http://en.wikipedia.org/wiki/Appeal_to_authority [wikipedia.org]
Re:Light Bulb Moment (Score:5, Interesting)
Anyway, i guess it'll be another pagerank or slashdot filter affair. People trying to beat it, devs trying to make it better.
The plus is, there is not only wikipedia. You can always search the rest of the web.
The minus is, you search the rest of the web with google which is equivalent if not worse.
We need a good search engine on top of a tor network, and bandwidth to make it run smooth. Not many other way to achieve real net freedom.
Parent
Re: (Score:3, Insightful)
Can you explain yourself a little more? I don't see how Tor would improve the quality of information being searched for. (Not arguing, just interested in your ideas)
Re:Light Bulb Moment (Score:4, Insightful)
Nope. If you post one misdeed and that gets edited out, such is life but shouldn't affect your credibility that much because everyone is always getting edited out a few times in the long run.
However, if you edit hundreds or thousands of different articles and people leave you alone, o great guru, you're good.
Wikipedia's ultimate strength depends on the community's desire for good information, readiness to stomp on crap, and will to contribute. Conversely, Wikipedia would decay if people didn't give a rat's ass about Wikipedia and let it go to ruin like an unweeded garden. This mechanism of quality control needs to be applied down the hierarchy of categories, subcategories, and articles. It's understandable that certain areas will have more pristine content overall while other areas will be populated with childish and wanton ideas. Thus, a contributor evaluation program can be tested.
Parent
Seems a bit dangerous (Score:5, Insightful)
And the editor wars start
Re:Seems a bit dangerous (Score:4, Insightful)
Parent
Re: (Score:3, Insightful)
No, it won't gain a better reputation in the eyes of professors (at least decent professors) for two reasons:
1) It's an inherently flawed algorithm and easily gameable. It's useful as a very vague unreliable data-point, and not much else.
2) Wikipedia is not a source for academic research, and never will be. If it's anything to academics, it's a place to go to get some clues on how to proceed with their
Re: (Score:3, Informative)
-~~~~
Re: (Score:3, Insightful)
but they get a whole new meaning when it makes sense to find all edits by an editor, delete them, and then rewrite them as your own...
Re: (Score:3, Informative)
We're not talking about Wikipedia's concept of authorship, here, but the tool's. The tool tracks who first wrote something and doesn't re-assign authorship because it was removed (e.g. by a vandal) and then restored.
You would have to remove what they wrote and then restore it in your own words in such a way that your edit was good enough to be retained by the community. In which case, the system worked.
Overall, I think it would be an excellen
Hmmm... A reputation metric... (Score:4, Funny)
Re: (Score:2)
Re: (Score:2)
I wonder whether nominating an editor on Wikipedia a "karma whore" will result in a net increase or decrease of "reputation" for the nominee. :-)
Godwin's Second Law (Score:3, Insightful)
(Godwin didn't publish this, but I might get around to editing his Wikipedia entry to say that he did).
7 years??? (Score:3, Interesting)
Re: (Score:2, Informative)
How can we trust the Wikimedia software if it corrupts the edit database?
Re: (Score:2, Informative)
RTFA!
Doesn't take into account common myths (Score:5, Interesting)
So, if there is a myth that a lot of people believe is true, then it will stay up there as it is not challenged. So, it still gets reputation, and therefore more credibility, making it more likely that the myth will be perpetrated.
Also, if someone hasn't noticed something that is wrong on an esoteric entry, it will also be given credibility, and once again be more likely to be considered to be fact.
While you could add voting to the algorithm to have people vote on whether it is true, that still gets destroyed by someone who just votes because they think it's true, not because they have verified it.
Either way, it potentially gives additional credibility to something that may be very wrong.
It doesn't have to be perfect (Score:5, Insightful)
No algorithm, except maybe personally checking every single article yourself, will ever be perfect. I suspect that the stuff you talk about will be very rare exceptions, not the rule. In fact, one of the reasons that it is so rare is because people who know what the actual truth of a matter is can post it, cite it, and show it for all to see that some common misconception is, in fact, a misconception. This is much better than, say, a dead tree encyclopedia where, if something incorrect gets printed, it will likely stay that way forever in almost every copy that's out there. (And, incidentally, no such algorithm can exist, since dead tree encyclopedias generally don't include citations and/or articles' editing histories.)
The goal wasn't to create a 100% perfect algorithm, it was to create an algorithm that provides a relatively accurate model and that works in the vast majority of cases. I don't see any reason this shouldn't fit the bill just fine.
Parent
Re: (Score:2, Insightful)
Re: (Score:2)
Layne
Seems to work ... (Score:5, Funny)
hmmm... (Score:5, Funny)
#REDIRECT (Score:5, Insightful)
Re: (Score:2)
I dunno about this system. (Score:5, Insightful)
And, of course, there is the potential for abuse. If the software could intelligently track reversions and somehow ascribe to those events a neutral sort of rep, that would probably help the system out.
As it stands, they're essentially trying to objectively judge "correctness" of facts without knowing the actual facts to check. That's somewhat like polling a college class for answers and assigning grades based on how many other people DON'T say that they disagree with a certain person in any way.
Re: (Score:2)
I guess it's a start, but pure longevity of content isn't the best metric for trustworthiness.
Re: (Score:2)
I suspect this heuristic measures.... (Score:5, Insightful)
If I edit a history page of a small rural village near where I live, I can guarantee that it will remain unaltered. None of the five people who have any knowledge or interest in this subject have a computer.
If I edit an item on Microsoft attitude to standards, or the US occupation of Iraq, I'm going to be flamed the minute the page is saved, unless I say something so banal that noone can find anything interesting in it.
But my Microsoft page might be accurate, and my village history a tissue of lies....
AfD: nn (Score:3, Insightful)
If I edit a history page of a small rural village near where I live, I can guarantee that it will remain unaltered. None of the five people who have any knowledge or interest in this subject have a computer.
If nobody else who has a computer cares, then it's less likely that your edits can be backed up with reliable sources [wikipedia.org]. In fact, people might be justified in nominating the article for deletion on grounds of lack of potential sources [wikipedia.org].
Tuned for Subject Matter (Score:5, Insightful)
Afterall just because someone is a reliable expert at editing the wikipedia entries on Professional Wrestling [wikipedia.org] or Superheroes [wikipedia.org] doesn't necessarily mean we should trust their edits on, for instance, the sensitive issues of Tibetan sovereignty [wikipedia.org].
Unpopular but neutral points of view? (Score:5, Interesting)
Re: (Score:2)
Re: (Score:2)
Tyranny of the majority (Score:5, Insightful)
That said, I can't help but believe that this tool is a net positive because it makes points of debate more visible. One could even argue that it literally highlights the frontiers of human knowledge. That is, high-trust (white) text is well known material and highlighted (orange) text represents contentious or uncertain conclusions.
Re:Tyranny of the majority (Score:5, Insightful)
Yes, this system demonstrates the correlation between the content and the majority opinion, not between the content and the correct information (assuming such objectively exists).
Of course, if you take as an axiom that the majority opinion will, in general, be more reliable than the latest random change by a serial mis-editor, then the correlation with majority opinion is a useful guideline.
Something that might be rather more effective, though perhaps less practical, is for Wikipedia to bootstrap the process much as Slashdot once did: start with a small number of designated "experts", hand-picked, and give them disproportionate reputation. Then consider secondary effects when adjusting reputation: not just whether something was later edited, but the reputation of the editor, and the size of the edit.
This doesn't avoid the underlying theoretical flaw of the whole idea, though, which is simply that in a community-written site like a wiki, edits are not necessarily bad things. Someone might simply be replacing the phrase "(an example would be useful here)" with a suitable example. This would be supporting content that was already worthwhile and correct, not indicating that the previous version was "untrustworthy".
Parent
Algorithms are handy (Score:2, Offtopic)
For instance, Google has a strong brand, despite their hideous logo and "Don't be evil" slogan, because the consumer experience is so good. Coca-Cola, on the other hand, score big with their logo's distinctive cursive script, despite ongoing critisms of its health effects and numerous allegations of wrongdoing by the compa
Re: (Score:2)
A reasonable first step... (Score:2, Funny)
Goddamn... (Score:5, Funny)
Don't Care. (Score:2, Insightful)
I do not trust wikipedia on any "divisive issue" (Score:2)
too many zealots rule certain categories and unfortunately too many of the same are the very powers that be.
This will promote one thing (Score:2, Insightful)
Should be called "stability" (Score:3, Insightful)
Algorithm doesn't prove what it thinks it does (Score:3)
It's progress over edit counts (Score:3, Interesting)
One big problem with Wikipedia has been that editor status, and promotion to "adminship", is based on edit counts, the number of times someone has changed something. The editors with huge edit counts don't generally write much; it takes too long. Slashdot karma is a more useful metric than edit counts, but Wikipedia doesn't have anything like karma.
I'd suggested on Wikipedia that we needed a metric for editors like "amount of new text that lasted at least 90 days without deletion". This UCSC thing is a similar metric.
Spelling Mistakes? (Score:3, Insightful)
In particular I'm worried that the system will undervalue the information from people whose edits are frequently cleaned up by others even if that content is left unchanged.
Re: (Score:3, Funny)
Re: (Score:3)