Large Scale Web Apps Built on Open Source 213
prostoalex writes "Brad Fitzpatrick presented at OSCON with on overview of his little project. Interesting facts about the evolution of the Livejournal back-end architecture."
It's currently a problem of access to gigabits through punybaud. -- J. C. R. Licklider
Salesforce.com (Score:4, Informative)
Re:Salesforce.com (Score:2, Informative)
http://www.onlamp.com/pub/a/onlamp/2001/01/25/l
Re:Salesforce.com (Score:4, Funny)
Re:Salesforce.com (Score:2, Funny)
Re:Salesforce.com (Score:2, Insightful)
Server: Resin/3.0.s040331
Re:Salesforce.com (Score:3, Informative)
My bet on a fully open system would be
Re:Salesforce.com (Score:2)
p.s. "BAPP" is pronounced Batman-style
Re:Salesforce.com (Score:2)
FreeBSD instead of generic termed BSD.
Re:Salesforce.com (Score:4, Funny)
BINGO!!!
lamp! (Score:2, Informative)
CB$@#--C
Re:lamp! (Score:2, Informative)
LAMP is php (and linux, apache, mysql) and last I checked Slashdot used mod_perl...
but it is still open source.
Re:lamp! (Score:3, Informative)
I thought the P means any or all of the P language: PHP, Python, Perl
Re:lamp! (Score:5, Informative)
Re:lamp! (Score:2)
That's OpenBSD with Apache, PostgreSQL, and Python from OpenBSD's packages repository. Fast, secure, and correct.
Re: (Score:2)
Re:lamp! (Score:2, Informative)
Hmmm, this post is modded up as informative, when it should really be modded down as misinformative (if only there were such an option).
Somebody please mod up one of the child posts that correct the parent by pointing out that the "P" in LAMP represents Perl, Python, or PHP. Please DON'T give me the karma, give it to those still posting at 1. Thanks.
Speaking of slashdot... (Score:2)
About a month or so ago, slashdot was regularly dying while fetching pages. Anybody know what was actually causing the problem? I suspected it was Mysql, but don't know.
In any case, it seems to have quieted down some.
Re:Speaking of slashdot... (Score:2)
CB
Re:Speaking of slashdot... (Score:2, Informative)
Re:Speaking of slashdot... (Score:2, Informative)
/. effect (Score:2, Funny)
Typical Livejournal (Score:5, Funny)
With MILK!!!! OMG!!
Re:Typical Livejournal (Score:5, Funny)
From link --
speakin of french and korea did u no they both opposed the war in iraq? 1 is a comunist country and the othr is a no-fight-anytime country. mabye there in this 2gethr to squash the american gold medals an make ppl think there strong! HEY KOREA WE BLEW U UP IN WW2 W/ TEH ATOM BOMB WE'LL DO IT AGAIN. an french ppl suck.
Re:Typical Livejournal (Score:2)
That's just sad. That someone is that wrong on their history about such a major point. I would post a correction, but I'd rather not post my IP addr to the world on her blog. I'll have to do it when I get home from work.
-nB
Re:Typical Livejournal (Score:5, Funny)
without a doubt Jesus would've been the best president ever. When he was only 12 years old he went to the temple to preach to the Jews and was just amazing everyone. Think how much better the U.S.A. would be if Bill Clinton wasn't Bill Clinto but instead was Jesus? Do you think we'd be fighting a way? NO!! We'd all be loving each other because thats what Jesus was about! LOVE! There isn't enough love in the world! Jesus would also be great because hes not only the son of god, HES A PRINCE OF PEACE!!! Hed probably do things differently. Even George W Bush could learn from Jesus (and thats why George W Bush is a christian and we need to keep him in office!!)
LiveJournal -- Convincing teenage girls that someone cares about what they have to say since 1999.
Re:Typical Livejournal (Score:3, Insightful)
We laugh about this but the really scary part is that there are a lot of people who think like her. People hate Bush so much because of the war but I am much more scared about his connection to the zealots of the religous right. The war in Ir
Re:Typical Livejournal (Score:2)
or is zealot == nimrod?
-nB
Re:Typical Livejournal (Score:3, Insightful)
And the right likes to call the left communists. Calling them zealots wouldn't make much sense.
And I had nothing to do with making this girl think there are anti-christian views in this country. She gets plenty of that from the preachers. If you don't thi
Re:Typical Livejournal (Score:2)
Even George W Bush could learn from Jesus (and thats why George W Bush is a christian and we need to keep him in office!!)
John Kerry is also a (Catholic) Christian. So I guess she must believe both candidates are qualified for the presidency.
Re:Typical Livejournal (Score:2)
9/11 - NEVR FORGET. in case ur wonderin what 9/11 is let me tell u. early in the morning at like 8 or sumthin some evil muslim ppl took over two planes and crashd them in to teh world trade center. the world trade center is a place where ppl do business and stuff so they were attack our econonmy. then anoter plane crashed into the white house an tried to kill sum military ppl. a bunch of ppl died and we should nevr 4get them.
Re:Typical Livejournal (Score:2)
Re:Typical Livejournal (Score:5, Funny)
Brad Fitzpatrick apparently agrees with your take on LJ, judging from the sample user data shown on page 24 of the presentation:
Obligatory SOD reference (Score:2)
But I think their 'Ballad of Michael Hutchens' off Bigger Than The Devil was the absolute best.
Java, Tomcat, Apache on UNIX (Score:2, Interesting)
We are using Oracle as the database, and Solaris as the UNIX, but we could be using MySQL and Linux.
In fact, we are investigating that right now
Re:Java, Tomcat, Apache on UNIX (Score:5, Interesting)
Re:Java, Tomcat, Apache on UNIX (Score:2)
Re:Hope you sent in a patch (Score:2)
Re:Java, Tomcat, Apache on UNIX (Score:3, Interesting)
That's 100% open source, people... and we are talking large corporate intraweb apps and such.
I work mostly with financial institutions... they prefer IBM backed Linux servers with WebSphere... but still like eclipse (or WSAD, which is eclipse with a Websphere test server plugin), and a commercial DB (oracle, DB2, or informix are popular)... but the
Re:Java, Tomcat, Apache on UNIX (Score:2)
I was just wondering how it compares to other existing commercial DBs out there.
Re:Java, Tomcat, Apache on UNIX (Score:2)
Not sure about popular, but it runs on some fairly big IBM hardware, and would historically be used by larger organizations. I guess with the porting to Linux, they are hoping for more small scale business.
Uh, the Web itself (Score:4, Insightful)
Uh, like, you mean the Web itself? That's large scale, certainly was built, and is most certainly built on open source.
So, yeah, I reckon it can be done. I'm using the proof-of-concept to submit this comment.
layers -- kind a like the onion (Score:4, Informative)
The most important aspect of the web is that the interface of the different layers were well defined and exposed...not that each line of code in the different layers is exposed.
Re:Uh, the Web itself (Score:2, Interesting)
The LJ folks faced scaling problems and had financial limits on how much money they could throw at the problem. So they used smarts and OS software instead of huge piles of money. They also built some new tools that are OS themselves, thus contributing back to the community (I hate that phrase, but this is Slashdot).
The presentation is actually interesting technically, and good news for Linux/MySQL/Perl/etc.
(I gue
Re: (Score:2)
Re:Uh, the Web itself (Score:2)
Re:Humor is dead (Score:2)
Re: (Score:2)
Re:Uh, the Web itself (Score:2)
.sxi format? (Score:3, Funny)
Re:.sxi format? (Score:5, Informative)
In the off chance that you are, it's one of the OpenOffice.org formats, inheritted from StarOffice... it's supposed to be their answer to MS PowerPoint.
Re:.sxi format? (Score:2)
What are you a Office luser ?
Re:.sxi format? (Score:3, Funny)
translation: "what, do you have a job? I wish I was part of the corporate world. Better get back to checking the classifieds..."
Re: (Score:2)
Re:.sxi format? (Score:3, Informative)
Re:.sxi format? (Score:2)
Re:.sxi format? (Score:2)
Re:.sxi format? (Score:2)
PASSWORD="" (Score:2)
Re:.sxi format? (Score:2)
opensource sxi? (Score:2, Funny)
Re:opensource sxi? (Score:3, Informative)
helixcommunity.org is another big one... (Score:5, Informative)
It's powered by GForge [gforge.org], so it's backed by PHP and PostgreSQL [postgresql.org].
There are a bunch of other sites running GForge listed here [gforge.org]...
GForge is your own personal SourceForge (Score:2)
Re:helixcommunity.org is another big one... (Score:2)
Not being one to want to re-invent the wheel, do you know of anyone packaging that for Fedora Core?
Re:helixcommunity.org is another big one... (Score:2)
It just seems like setting up a departmental development server is a common enough task that there should be an easy-to-setup Open Source solution.
Ah well, there I go dreaming again.
(Thanks for the info!)
Maypole! (Score:5, Informative)
Livejournal Images (Score:5, Funny)
http://www.livejournal.com/stats/latest-img.bml [livejournal.com]
Hint - its a constantly updating list of all the new images posted to journals. After a while you give up waiting for a hot chick to post and decide crazy survey graphics are as good as it gets. And then some hot chick posts her birthday party pictures, but she's only 14 and suddenly you wish you'd spent the day doing something else.
Re:Livejournal Images (Score:3, Interesting)
Re:Livejournal Images (Score:2, Informative)
Porn (Score:5, Interesting)
Re:Porn (Score:3, Funny)
Re:Porn (Score:2)
Re:Porn (Score:4, Funny)
Porn (Score:4, Interesting)
I worked at a huge (now defunct) porn site.
The funny thing is, I'm pretty sure the interesting mod is about working for a porn site, and has nothing to do with the hardware or software (for those who even read that far... ;)
Large scale? (Score:2)
Per Hour: 6818
Per Minute: 114
That's 2 inserts a second, and maybe a hundred queries a second. Quite honestly, that could be handled by MySQL & PHP. Definitely not what I'd call "large scale".
Re:Large scale? (Score:2, Informative)
Your assumption would be correct if it was 1 select for each page view, but since there are about 4-5 just for 1 page view (userpic, friends, info, etc) then that number is misleading.. Fortunally most of that static content is memcached and not hitting the DB's.
Re:Large scale? (Score:5, Interesting)
Anyway, those are only the number of entries being posted. For every entry being posted, there are a ton of inserts actually going on:
* log2 table to contain some metadata about the entry
* logtext2 table to contain the actual text
* logprop2 table (multiple rows, 3-5) containing other metadata about entry
So, four times the traffic, about 6 inserts each, 2400 updates per second--and that's just for posting entries. We get a lot more traffic from people posting comments (which also do 3 or 4 update/inserts each comment), plus people editing their userinfo, uploading new userpics,
While LiveJournal definitely isn't a huge site, it's not a lightweight, and definitely doing pretty good for having around 80 machines and doing 30-40 million fully dynamic page views a day.
Zope Enterprise Objects (Score:5, Informative)
Server timed out (Score:3, Funny)
WARNING!!! Goat Link!!! (Score:2, Funny)
What about Livejournal? (Score:4, Interesting)
Re:What about Livejournal? (Score:2)
Do you even watch the evolution of the site, or are you just throwing stuff out for the hell of it?
Re:What about Livejournal? (Score:2)
Re: (Score:2)
The last 3 projects I've been on were built on OSS (Score:2)
Is there something interesing here?
LJ - Memcached - Wikipedia (Score:4, Informative)
Re:LJ - Memcached - Wikipedia (Score:2, Informative)
Kenny (Score:5, Funny)
"Inside LiveJournal's Backend" (Score:2)
Inside LiveJournal's Backend or, "holy hell that's a lot of hits!"
Believe me, it is taking all my strength to avoid making a certain obvious joke about this title..
Re:Get a clue (Score:5, Insightful)
This is for people with absolutely no budget and infinite traffic. This is how to live through that and come out winning like Brad apparently has.
Re:Get a clue (Score:2, Informative)
I guess Amazon.com [masonhq.com] is one of those not-properly-designed websites that doesn't do anything real?
Re:Get a clue (Score:3, Insightful)
Simple lessons:
-replicating database al
Re:Get a clue (Score:2)
They could/should have moved to a much better DBMS. Although the DBMS licensing fee would've been non-trivial it would have meant SIGNIFICANTLY reduced hardware costs and much much less application code development. I even suggested this several years ago but I was told that licensing costs were prohi
Re:Get a clue (Score:2, Interesting)
I was thinking mostly of Sybase Replication Server combined with Sybase ASE or Oracle 10g/Oracle Clustering, things that would go really, really nicely in the environment and workload the LiveJournal folk are experie
Re:Get a clue (Score:5, Insightful)
Give it up slashdot crowd. mod_perl is not a valid technology for a large scale website! Perl was designed for a task, and that task was NOT enterprise application development.
Spoken like someone who has never had to build a very large site (doing "real" work) completely in Perl/mod_perl. I can tell you that it most certainly can scale to enterprise needs. Did this guy do it right? I don't think so either but he most certainly learned a valuable lesson. Hopefully other people will study what he has done and improve their own systems based on his work.
For the record, Java wasn't built for enterprise application development either. As with Perl, people discovered that Java had a future there and here we are today.
A properly designed website with n-tier sepperation will be able to handle a large load and scale infinitly. You'll note that large websites who actually do real things besides logging people's daily problems don't use mod_perl and a thousand servers. There's a reason for this.
You're assuming two dangerous things... (1) That you can't have n-tier and Perl. And (2) that large mod_perl sites require lots of servers. To believe any of these things is to demonstrate your horrific misunderstanding of computer science in general. I pity the company that lets you design their architecture. Wait, no I don't.... I'll gladly take their money for fixing your mistakes.
Oh yeah, and let us not forget some other languages that are showing promise... specifically Python+Zope. In fact, I know of several people implementing n-tier applications with PHP on the front, Python in the middle and PostgreSQL in the back with much success.
And for the record, here [amazon.com] are [ticketmaster.com] some [etoys.com] large [rent.com] companies [find-job.net] and [redhat.com] sites [redhat.com] heavily [mobile.de] using [afp-direct.com] mod_perl [imdb.com].
Want more [apache.org]?
Re:Get a clue (Score:2)
Also PHP -> PL/PgSQL -> Postgres
I count PL/PgSQL and postgres different tiers because they have different functions and in the case of one system I'm working on all database interactions are moderated by PL/PgSQL stored procedures. They could just as easily be PL/Python or PL/Perl stored procedures if I wanted them to be.
Re:Get a clue (Score:2)
That's interesting, what you have certainly provides the ideal MVC separation but I'm not sure that it would technically qualify as 3-tier. Only because you couldn't scale up or swap-out the PL/PgSQL without also affecting Postgres.
<crazy mode>
That being said, it might be possible though. (And this is
Re:Get a clue (Score:2)
How do they do to make the PHP front talk to the Python middle layer?
I love python and I've been trying to use Python in the front too which it isn't too good at. PHP+Python sounds interesting.
Re:Get a clue (Score:2)
Oh, yes it is. [slashdot.org]
Re:Oooh, handwaving. (Score:2)
Re:LiveJournal isn't THAT large and fairly slow (Score:2)