Stories
Slash Boxes
Comments
Search

News for nerds, stuff that matters

Slashdot Log In

Log In

Create Account  |  Retrieve Password

Coral P2P Cache Enters Public Beta

Posted by timothy on Sat Aug 28, 2004 07:59 PM
from the will-it-scale dept.
The Internet IT
Eloquence writes "infoAnarchy reports that Coral, a peer-to-peer webcaching system, has gone into public beta. Currently the Coral node network is hosted on Planet-Lab, a large scale distributed research network of 400 servers. You can use Coral right now by appending "nyud.net:8090" to a hostname. View Slashdot through Coral. Is this the end of the Slashdot effect?"
+ -
story
This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Coral P2P Cache Enters Public Beta 50 Comments More /

 Full
 Abbreviated
 Hidden
More
Loading... please wait.

  • Slashdotted already (Score:5, Funny)

    by Rexz (724700) on Saturday August 28 2004, @08:00PM (#10099843)
    Just kidding.
    • > Just kidding.
      yea, but its true :)

    • Re:Slashdotted already (Score:2, Informative)

      by Anonymous Coward
      No, it really is down, at least from here:

      Windows:

      C:\>ping slashdot.org.nyud.net
      Ping request could not find host slashdot.org.nyud.net. Please check the name and try again.

      Linux:

      ~$ping slashdot.org.nyud.net
      ping: unknown host slashdot.org.nyud.net

      ~$dig slashdot.org.nyud.net
      ; <<>> DiG 9.2.1 <<>> slashdot.org.nyud.net
      ;; global options: printcmd
      ;; connection timed out; no servers could be reached

      Seems their nameservers have some kind of problem. I am in the Midwest, going t

    • "Invalid domain name in packet" (Score:5, Informative)

      by Anonymous Coward on Saturday August 28 2004, @08:46PM (#10100090)
      It really DOESN'T work for a lot of people.

      The problem is that it doesn't seem to be compatible with Microsoft DNS severs. Below is a copy of the DNS log when I issue a query here, on my LAN which has a Microsoft DNS server running on Windows 2000, which then forwards through the University of Wisconsin. You can see that at the end it says "The DNS server encountered an invalid domain name." Perhaps someone who knows more about DNS can tell where the problem is?

      Rcv 10.76.0.2 0004 Q [0001 D NOERROR] (8)slashdot(3)org(4)nyud(3)net(0)
      UDP question info at 014D5A0C
      Socket = 384
      Remote addr 10.76.0.2, port 1263
      Time Query=4338128, Queued=0, Expire=0
      Buf length = 0x0200 (512)
      Msg length = 0x0027 (39)
      Message:
      XID 0x0004
      Flags 0x0100 QR 0 (question) OPCODE 0 (QUERY) AA 0 TC 0 RD 1 RA 0 Z 0 RCODE 0 (NOERROR)
      QCOUNT 0x1 ACOUNT 0x0 NSCOUNT 0x0 ARCOUNT 0x0
      Offset = 0x000c, RR count = 0
      Name "(8)slashdot(3)org(4)nyud(3)net(0)"
      QTYPE A (1)
      QCLASS 1
      ANSWER SECTION:
      AUTHORITY SECTION:
      ADDITIONAL SECTION:

      Snd 144.92.254.254 39b0 Q [0001 D NOERROR] (8)slashdot(3)org(4)nyud(3)net(0)
      UDP question info at 0109200C
      Socket = 408
      Remote addr 144.92.254.254, port 53
      Time Query=0, Queued=0, Expire=0
      Buf length = 0x0200 (512)
      Msg length = 0x0027 (39)
      Message:
      XID 0x39b0
      Flags 0x0100 QR 0 (question) OPCODE 0 (QUERY) AA 0 TC 0 RD 1 RA 0 Z 0 RCODE 0 (NOERROR)
      QCOUNT 0x1 ACOUNT 0x0 NSCOUNT 0x0 ARCOUNT 0x0
      Offset = 0x000c, RR count = 0
      Name "(8)slashdot(3)org(4)nyud(3)net(0)"
      QTYPE A (1)
      QCLASS 1
      ANSWER SECTION:
      AUTHORITY SECTION:
      ADDITIONAL SECTION:

      Rcv 144.92.254.254 39b0 R Q [8081 DR NOERROR] (8)slashdot(3)org(4)nyud(3)net(0)
      UDP response info at 012DB8AC
      Socket = 408
      Remote addr 144.92.254.254, port 53
      Time Query=4338128, Queued=0, Expire=0
      Buf length = 0x0200 (512)
      Msg length = 0x00e0 (224)
      Message:
      XID 0x39b0
      Flags 0x8180 QR 1 (response) OPCODE 0 (QUERY) AA 0 TC 0 RD 1 RA 1 Z 0 RCODE 0 (NOERROR)
      QCOUNT 0x1 ACOUNT 0x4 NSCOUNT 0x2 ARCOUNT 0x2
      Offset = 0x000c, RR count = 0
      Name "(8)slashdot(3)org(4)nyud(3)net(0)"
      QTYPE A (1)
      QCLASS 1
      ANSWER SECTION:
      Offset = 0x0027, RR count = 0
      Name "[C019](4)nyud(3)net(0)"
      TYPE 39 (39) CLASS 1 TTL 1333 DLEN 25
      DATA Unknown resource record type 39 at 012DBC41.
      Offset = 0x004c, RR count = 1
      Name "[C00C](8)slashdot(3)org(4)nyud(3)net(0)"
      TYPE CNAME (5)
      CLASS 1 TTL 0 DLEN 15
      DATA (8)slashdot(3)org[C033](4)http(2)l2(2)l1(2)l0(5)n yucd(3)net(0)
      Offset = 0x0067, RR count = 2
      Name "[C058](8)slashdot(3)org[C033](4)http(2)l2(2)l1(2) l0(5)nyucd(3)net(0)"
      TYPE CNAME (5)
      CLASS 1 TTL 1335 DLEN 2
      DATA [C033](4)http(2)l2(2)l1(2)l0(5)nyucd(3)net(0)
      &am p;n bsp; Offset = 0x0075, RR count = 3
      Name "[C033](4)http(2)l2(2)l1(2)l0(5)nyucd(3)net(0)"
      TYPE A (1)
      CLASS 1 TTL 60 DLEN 4
      DATA 139.91.70.71
      AUTHORITY SECTION:
      Offset = 0x0085, RR count = 0
      Name "[C038](2)l2(2)l1(2)l0(5)nyucd(3)net(0)"
      TYPE NS (2)
      CLASS 1 TTL 1991 DLEN 19
      DATA (3)139(2)91(2)70(2)71(3)ip4[C041](5)nyucd(3)net(0 )
      Offset = 0x00a4, RR count = 1
      Name "[C038](2)l2(2)l1(2)l0(5)nyucd(3)net(0)"
      TYPE NS (2)
      CLASS 1 TTL 1991 DLEN 16
      DATA (3)141(3)213(1)4(3)202[C09E](3)ip4[C041](5)nyucd( 3)net(0)
      ADDITIONAL SECTION:
      Offset = 0x00c0, RR count = 0
      Name "[C091](3)139(2)91(2)70(2)71(3)ip4[C041](5)nyucd(3 )net(0)"
      TYPE A (1)
      CLASS 1 TTL 603196 DLEN 4
      DATA

      • Re:"Invalid domain name in packet" (Score:5, Informative)

        by mfreed (217310) on Saturday August 28 2004, @09:20PM (#10100249) Homepage
        It appears that the Windows 2000 DNS server you are using is not aware of DNAME records (RFC 2672 [faqs.org]):

        Name "[C019](4)nyud(3)net(0)"
        TYPE 39 (39) CLASS 1 TTL 1333 DLEN 25
        DATA Unknown resource record type 39 at 012DBC41.
        We use these types of records to aid in redirecting resolvers to nearby Coral proxies (by mapping nyud.net to a "hierarchical" name http.l2.l1.l0.nyucd.net. The goal is that once you find a "nearby" server, you should remain nearby.

        Given that the DNAME RFC is from 1999, it appears that some old DNS servers do not handle this record type well. We'll look into some alternatives or work-arounds. (Perhaps you can contact me directly to see if subsequent changes can fix your problem.)

        Thanks for the detailed report!
        --mike

    • Check out their logs...

      Coral Statistics [nyu.edu]

      ...note the recent blip?

  • Google (Score:3, Informative)

    by asd-Strom (792539) on Saturday August 28 2004, @08:01PM (#10099851)
    Google cache has been a good helper to me for some time.
    So this is not so new to me regarding slashdot effects.

    • Re:Google (Score:5, Informative)

      by bogie (31020) on Saturday August 28 2004, @10:48PM (#10100637) Journal
      Google cache tip for you. There is a bookmarklet for Firefox where you simply click the bookmarklet and Google's cache of the page opens up. Its a nice feature to have at your fingertips. You can get the code at the very bottom of the following page, just drag it to your personal toolbar.

      http://www.rentzsch.com/notes/googleCacheHacking

      If the page won't load at all thus negating the above just use the following example to load a page.

      http://google.com/search?q=cache:www.slashdot.or g

      • Re:Google (Score:5, Informative)

        by doofsmack (537722) * <slashdot@@@doofsmack...com> on Sunday August 29 2004, @01:46AM (#10101159) Homepage
        Talking about bookmarklets, I just wrote a quick little bookmarklet to redirect you to the Coral cache of the current page. Here it is:

        javascript:location.href=location.href.replace(/ht tp\:\/\/([a-zA-Z\.]+)\/(.*)/, "http://$1.nyud.net:8090/$2");void(0)

        And if slashdot's tendency to insert spaces in long strings screws that up, try grabbing it from here [gotdoof.com]

        • Re:Google (Score:3, Informative)

          by Gherald (682277)
          >Are you saying with Coral that you can?

          With Coral you can get it cached just by asking for it. Of course, the Coral pcs have to connect to it at least once.

          You cannot get google to cache a page at your request -- no matter how hard you try :)

  • Anyone see the irony? (Score:5, Funny)

    by bigberk (547360) <bigberk@users.pc9.org> on Saturday August 28 2004, @08:01PM (#10099858)
    Of, well, slashdoting the solution to slashdotting? Really cool idea though. Nice!

  • Dear Lord (Score:4, Funny)

    by over_exposed (623791) on Saturday August 28 2004, @08:02PM (#10099864) Homepage
    I hope this isn't the end of the /. effect! What would we do w/o webservers crashing under tremendous loads?!? WE NEED the /. effect! I hope this technology crashes and burns...

    Then again it might not be so bad....

  • self-referential slashdotting (Score:5, Funny)

    by Shaheen (313) on Saturday August 28 2004, @08:02PM (#10099866) Homepage
    so it's like this... people click on a link on slashdot, which gets farmed out to the p2p network to get the cached copy, but there's so many people clicking the link to get the cached copy that they are only slashdotting their own computers since they are all part of the p2p network too! now we can all collectively feel the slashdot effect!

    oh, first post?

  • files (Score:3, Interesting)

    by Coneasfast (690509) on Saturday August 28 2004, @08:02PM (#10099868)
    you can ensure that your readers can still access a certain web page or files, when the multitude of readers would otherwise overload the website and make the content unavailable.

    well apparently all html content, including files, will be cached. this is a great way to get around downloading from snail-pace sites, (although i will be checking md5sums)
  • http://www.scs.cs.nyu.edu.nyud.net:8090/coral/ if we do /. it...

  • Not too good for websites (Score:4, Insightful)

    by chrispyman (710460) on Saturday August 28 2004, @08:04PM (#10099881)
    While their system would be pretty good (supposing it can withstand a slashdotting) for cacheing large files, it's not very useful for websites. Websites usually have lots of additional images, links, and whatnot, and as is currently, the system doesn't rewrite URLs.

  • In case Coral gets slashdotted (Score:5, Funny)

    by Rushuru (135939) on Saturday August 28 2004, @08:05PM (#10099885)
    In case Coral gets slashdotted, use this mirror [slashdot.org] to view slashdot
    • It might sound better if you wrote " moddo appu onegai". The pronoun "me" made it sound awkward to me. It would sound fine in English without the pronoun (e.g. "mod up please"), and since the Japanese dislike pronouns in the first place, you might as well take it out. Plus, without the pronoun it sounds like more traditional katakana usage rather than forced English.

      I'm not a native speaker though, so ymmv. :)
  • "Is this the end of the Slashdot effect?"

    haha no - only the lateral shifting of the slashdot effect to your local lan as some dope sets up a cache server in your office. Im sure the ./ colo guys at exodus would love for you to run one :).

  • Hosting companies'll hate this.... (Score:3, Interesting)

    by rsilvergun (571051) on Saturday August 28 2004, @08:09PM (#10099907)
    as will ISPs if it takes off. Right now with bandwidth usage centralized it's pretty easy to bill for it. If you decentralize it with p2p via millions of always on unmetered clients/servers it gets hard, if not impossible. I kinda hope it doesn't take off, since if it does it could end unmetered Internet access...

      • imagine if we all used our max upload bandwidth 24/hrs a day. ISP would need to modify their networks to work around this. At least I assume they would. As it is, many 'unmetered' isps will start sending you nastygrams if you make heavy use of your upload bandwidth, but otherwise look the other way when you run a server. Keep in mind that all these p2p apps violate most IPS' TOS (mine doesn't let you run a server of any kind, and while there are places where enforcement of that would be silly, there's still
      • Erm? If you mean the "5mb webspace" places, no, they don't charge for bandwidth. They just cancel your account if you post porn or anything else that will get heavily downloaded.

        As far as I know, anybody in the 0.5gig/month or over (all the way up to the backbone carrierers, which have to have peering agreements as an exception to the rule of charging for bandwidth) charges per megabyte.

  • Also a proxy... (Score:4, Interesting)

    by jelevy01 (574941) on Saturday August 28 2004, @08:09PM (#10099908)
    This would also by pass any restricted sites your company may be blocking...

    • Re:Also a proxy... (Score:5, Informative)

      by interiot (50685) on Saturday August 28 2004, @09:35PM (#10100326) Homepage
      There are actually a lot of sites out there that will let you access arbitrary content from elsewhere. Most corporate restricting proxies will block at least some of them (but it's impossible to get all of them). So something that could be as high-profile as Coral is less useful compared to some of the more obtuse of these:
      • google cache (this has been periodically blocked at my company)
      • the internet archive [archive.org]
      • online translation sites (eg. if it's an english site, have the translator go from japanese to english... none of the words will be recognized as japanese, so it will pass them all as-is)
      • several others I'm forgetting at the moment...
  • No, after the FBI has a gander at the servers, and puts them in a truck and drives off, the Slashdot effect will be alive and well.
  • This is a Content Distributon network of cooperating servers colloborating to exchange information and 'level out' excess demand by distributing reqiests among n servers. Like Akamai's EdgeSuite. based on a quick read of the front page. The providors of content in their network are never the consumers if content. thus i don't know why they call it peer-to-peer? anyone?
  • Some friends and I have an approx 10 MBytes application we want to distribute over the Internet, looking into hosting costs we see that it would cost us a bundle. So does coral let us serve our file to a slashdot-like crowd without breaking the bank?
    • Bittorrent is your friend. It's as common as AIM or IRC these days, instead of pulling the whole file from a central server, only the first few need to use a server host, and everyone else shares with each other. Most big linux distros do it with 650 MB files, or for large video files. No reason it wouldn't work for you.

      Here, I'll even link you to a good client that will give you a nice GUI for starting out. Another Bittorent Client [sourceforge.net] for all OSes.

  • Only the top page? (Score:5, Interesting)

    by News for nerds (448130) on Saturday August 28 2004, @08:24PM (#10099977) Homepage
    http://slashdot.org.nyud.net:8090/ caches only the /. homepage. Doesn't it analyze hyperlinks?

    • Re:Only the top page? (Score:4, Informative)

      by mothz (788133) on Saturday August 28 2004, @08:43PM (#10100073)
      Doesn't it analyze hyperlinks?

      All the links on Slashdot have the format
      <a href="//slashdot.org/blahblahblah">
      so that they will always link back to Slashdot. Most websites just use "blahblahblah" or "/blahblahblah" for their links. For example, links on google.com.nyud.net [nyud.net] are fully functional.
      • so that they will always link back to Slashdot. Most websites just use "blahblahblah" or "/blahblahblah" for their links. For example, links on

        Well...all they have to do is have some modifying code like CGI-Proxy [jmarshall.com] does....

  • Is it possible to combine this with bittorrent (Score:3, Insightful)

    by Danathar (267989) on Saturday August 28 2004, @08:31PM (#10100007) Journal
    Many times it seems a bittorrent tracker is down due to bandwidth issues. If I "corralized" it...could this alleiviate the problem?
    • The whole point of a tracker is that it's updated constantly with which chunks each person has available. A cache, by definition, doesn't interact with the original site so you couldn't send your own information. Nobody would know to download chunks from you, and therefore their software would be less likely to send you chunks.

      You could conceivably design a distributed tracker, but this isn't it. Anyway, there would doubtless be synchronization issues that would greatly decrease the network's overall performance.

  • Stats! Slashdot has it REALLY working! (Score:5, Informative)

    by Danathar (267989) on Saturday August 28 2004, @08:34PM (#10100024) Journal
    http://www.scs.cs.nyu.edu/coral/stats/

  • Work for CmdrTaco (Score:5, Interesting)

    by Dreadlord (671979) on Saturday August 28 2004, @08:43PM (#10100076) Journal
    Goatse-links trolls will be back, with slashcode showing the same domain for [nyud.net] every [nyud.net] link [nyud.net], I think CmdrTaco has some work to do now.

  • Stupid story submitter... :-) (Score:5, Funny)

    by Jugalator (259273) on Saturday August 28 2004, @08:46PM (#10100086) Journal
    To save their bandwidth, you should've linked to their mirror!
    http://www.nyud.net.nyud.net:8090 [nyud.net]

    • Re:Stupid story submitter... :-) (Score:5, Funny)

      by interiot (50685) on Saturday August 28 2004, @09:42PM (#10100355) Homepage
      In fact, it lets you specify port numbers as well, so you can use the mirror-of-the-mirror too:

      http://www.nyud.net.nyud.net.8090.nyud.net:8090/ [nyud.net]

      Or the mirror-of-the-mirror-of-the-mirror:

      http://www.nyud.net.nyud.net.8090.nyud.net.8090.ny ud.net:8090/ [nyud.net]

      They should have posted THAT link to slashdot to see how well the system faired.

      • Hackable? (Score:3, Interesting)

        by arth1 (260657)
        The practice of allowing portnumbers seems dangerous. I can imagine links like http://localhost.19.nyud.net:8090/ or http://loghost.515.nyud.net:8090/ being used for nefarious purposes.

        Regards,
        --
        *Art

        • Re:Hackable? (Score:3, Interesting)

          by arth1 (260657)
          Yep, accessing http://localhost.22.nyud.net:8090/ returned:

          SSH-1.99-OpenSSH_3.5p1 via: SSH-1.99-OpenSSH_3.5p1 216.165.109.81:8090 (CoralWebPrx/0.1 (See http://www.scs.cs.nyu.edu/coral/))
          accept-ranges: none
          connection: close

          That definitely doesn't look too good, security-wise, when you can get access to inside services through their proxy.

          Regards,
          --
          *Art
            • Okay. Apparently localhost is now blocked, at least it didn't give me the reply you guys got.

              That's the Microsoft way of securing things -- blocking single exploits as they are found. That doesn't solve the design problem of the proxy being able to contact any host/port, including LAN ones. Just substitute localhost with any host of choice, or even broadcast addresses.

              This product needs a design change.

              --
              *Art

  • Oooh! Graphs! (Score:3, Funny)

    by shish (588640) on Saturday August 28 2004, @08:50PM (#10100115) Homepage
    Pretty picture :)

    http://www.scs.cs.nyu.edu/coral/stats/

    Doesn't give a usable time scale though; it has "HTTP requests", but not "per second" / "per minute" or anything :(

  • Hmmm... wondering if I could use this commercially (Score:3, Interesting)

    by digidave (259925) on Saturday August 28 2004, @09:17PM (#10100238)
    I haven't checked the terms of use to see if I'm allowed to use this for my work web site, though maybe with a cash or hardware donation, or by running a high-bandwidth node, I can get permission.

    What I'm thinking is that at work I run a multi-server site that gets massively bogged down for short periods when it tries to handle upwards of 35,000 concurrent sessions. Bandwidth is not the problem, the application is, and it can't be rewritten for reasons that piss me off and I have no budget for more servers and no management support to run a static cached version of the site.

    So I was wondering if it was possible to have the site automatically direct visitors to the Coralized URL when the site load gets too high. Either a manual change or an automatic one would be ok. I have some ideas on how this could be done using a failover server config on our ServerIron. Possibly a router config can also do this, though we don't run our own router since it's at a colocation facility. Worst case scenario is I can edit the home page to redirect to Coral when the load gets high.

    Are there any other Slashdotters looking to use Coral in similar ways? If you have any ideas to share I'd be all ears.
    • From the FAQ:
      What files are being served by FreeCache?

      FreeCache can only serve files that are on a web site. If the link to a file on that web site goes away, so will the file in the FreeCaches. Also, there is a minimum size requirement. We don't bother with files smaller than 5MB, as the saved bandwidth does not outweight the protocol overhead in those cases.

    • Re:Incompatible with logged in browsing (Score:5, Informative)

      by maskedbishounen (772174) on Saturday August 28 2004, @09:42PM (#10100352)
      Sounds like we need a little lesson on How cookies work [google.com].

      To summarize it, though, they're set on a per-domain basis.

      www.apple.com can set a cookie.
      store.apple.com can set a cookie.

      The two cannot interact with each other; however, .apple.com can interact with any Apple subdomain.

      microsoft.com cannot access any of your apple.com cookies.

      Thus, nyud.net cannot access your /., or any other cookies.
    • I have to believe that ISPs that provide web services would find their revenue reduced since they would not see all the hits on the site.

      It seems you're confusing a "cache" with a "proxy." A "cache" is only DESIGNED to work on static pages, and it doesn't hit the page more than once (barring refreshing). That's what "cache" means. The pages are stored on the cache server and fed to the clients as they get requested, cutting down on hits to the actual site.

All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2009 Geeknet, Inc.