Forgot your password?
typodupeerror
Bug Oracle Databases Security

Serious Oracle Flaw Revealed; Patch Coming 100

Posted by timothy
from the now-how-much-would-you-pay? dept.
GMGruman writes "A bug in Oracle Database that could take down large databases — or let a hacker do so — has been found, and Oracle promises a patch later today. When InfoWorld first heard of the bug two months ago, its investigation revealed how dangerous this bug could be, and after convincing Oracle to address the issue, InfoWorld held the news until a patch was available, so hackers could not exploit the bug in the meantime. Paul Venezia details just how this bug exposes companies to the possibility of databases going offline, and Eric Knorr asks Oracle users to help test the patch in their complex environments. (InfoWorld's tests in simpler environments show the patch works there.)"
This discussion has been archived. No new comments can be posted.

Serious Oracle Flaw Revealed; Patch Coming

Comments Filter:
  • by Megaweapon (25185)

    ...brought to you by InfoWorld! Submitted by InfoWorld! Seriously, how much is /. getting behind the scenes from the various IT rags that plaster the front page?

    • Re: (Score:2, Funny)

      by Ihmhi (1206036)

      Seriously, how much is /. getting behind the scenes from the various IT rags that plaster the front page?

      At least enough money that it's worthwhile to keep doing it?

      • by HBI (604924)

        Until people stop reading. I stopped reading the trade rags a decade ago because of all the sponsored content and the fact that they gave zero insight. If /. is just trade rag bullshit, then why even come here?

        • by kiwimate (458274)

          It's not, at least not yet. I think a bigger problem is you have so many people posturing and proclaiming and acting as experts and simply flat-out speculating (incorrectly and/or uselessly), that the noise to signal ratio makes it less and less useful to read unless I want to spend ages digging through the cruft to figure out what's actually insightful or informative. In too many cases, merely targeting the +4 or +5 moderated posts doesn't guarantee that you're going to read something that is, you know, ac

        • by Anonymous Coward on Tuesday January 17, 2012 @01:41PM (#38726848)

          If /. is just trade rag bullshit, then why even come here?

          First posts and goatse.

          • Slashdot: Come for the first post, stay for the goatse

            That has a nice ring to it.
            • by Saija (1114681)
              Nice ring and goatse in the same reply?
              I can't wash that mental image away from my mind...
        • by jc42 (318812)

          If /. is just trade rag bullshit, then why even come here?

          Because it's all put into perspective by people like you.

    • by Grave (8234) <awalbert88 AT hotmail DOT com> on Tuesday January 17, 2012 @01:23PM (#38726622)

      Given that it's a fairly decent article about a somewhat (or very, for large companies) significant bug in a widely-used database, I think it still qualifies as "News for Nerds", doesn't it?

      • by Megaweapon (25185)

        Not by much, "Corporate Expensiveware Has Bug, Film At 11" isn't interesting nor very "nerdy" on its own. If you watch what gets frontpaged and who the submitters are lately, it should become obvious that modern /. is just clickbait for the big IT rags. I doubt this is merely coincidence, so I'm wondering what's going on behind the scenes.

        • by nigelo (30096)

          Yes, but it's terribly nerdy to blithely dismiss these stories if you don't use/like/know-anything-about the software, isn't it?

          Thus, still really suitable for posting to this site.

          • by Megaweapon (25185)

            Yes, but it's terribly nerdy to blithely dismiss these stories if you don't use/like/know-anything-about the software, isn't it?

            Yeah, I work in an Oracle shop. I've also been on /. since it was on Malda's Alpha. My point is the intent of the submission, since there's plenty of nerdy stuff to post besides some piece about turtleneckboy's latest crapware bug written up on just another industry rag.

            • by rnturn (11092)

              I'd disagree but I guess we'll find out how irrelevant this news is when we find that our credit cards aren't being accepted because the processor has had to shut their database again while they patch another Oracle database. I use /. as a one-stop shopping for much of my tech news. I don't find the occasional article about non-Linux software to be be a problem. I know I learned something new today. While I previously knew about the SCN, I didn't know it was a built-in timebomb. Oracle did a couple of thing

        • by sjames (1099)

          The way the failure can hop from DB to DB as they connect was interesting. It's a bug that causes failures that propagate virus like.

        • by afabbro (33948)

          Not by much, "Corporate Expensiveware Has Bug, Film At 11" isn't interesting nor very "nerdy" on its own.

          In this case, there was a very detailed and interesting explanation of why and how the bug is caused, and it doesn't require you to know anything about Oracle to appreciate the engineering choice made and why the bug hits it.

    • Someone's got sand up their ass.
    • by sjames (1099)

      Yet I would rate TFA as interesting and good to know.

  • That sounds kind of ominous.
  • Original source (Score:4, Interesting)

    by vlm (69642) on Tuesday January 17, 2012 @01:14PM (#38726500)

    I assume they're referring to:

    http://www.oracle.com/technetwork/topics/security/cpujan2012-366304.html [oracle.com]

    My mystification is what is the venn diagram intersection of mysql server, virtualbox, and oracle 11G? Without any details I'm guessing a package signing key got owned?

    • Nope. RTFA.

      Short answer. Very large Enterprise levels Oracle installations with multiple, interconnected databases shouldn't perform backups.

      So, no problemo. Business as usual!

      • by rubycodez (864176)
        there are plenty of other ways to perform hot backups than the ALTER DATABASE/TABLESPACE BEGIN BACKUP way.
        • by afabbro (33948)

          there are plenty of other ways to perform hot backups than the ALTER DATABASE/TABLESPACE BEGIN BACKUP way.

          How?

          • RMAN anyone? I doubt anyone serious still uses BEGIN/END backup...
          • by rubycodez (864176)
            one easy way, especially for big corporations, is to quiesce the database and take snapshot or clone with your SAN controller or host based storage manager, or don't quiesce it and just run recovery on the snapped image if you ever need it.
      • by stanlyb (1839382)
        Except if you somehow are able to connect your little, low privileged and hacked database to this Goliath, then it will result in extremely long and expensive repair procedure for this mastodon.
      • Re:Original source (Score:5, Interesting)

        by gringer (252588) on Tuesday January 17, 2012 @01:37PM (#38726792)

        Reading that article kept bringing forward more "oh no" realisations, stemming from the following points:

        1. like time, the SCN cannot decrement. It must always tick forward
        2. when Oracle databases link to each other, maintaining data consistency requires them to synchronize to a common SCN. This is necessarily the highest SCN carried by any participating Oracle database
        3. If the soft limit [16384 times the number of seconds since 00:00:00 01/01/1988] is exceeded, the database can become unstable and/or unavailable.

        The "recovery" for exceeding the soft limit is to shut down the databases until the SCN goes below the soft limit. From then on, you just have to hope that no databases you're synchronising with will have a SCN that is close to (or beyond) this soft limit.

        • by sjames (1099)

          That's the really scarey part. If you miss just one single database anywhere in the enterprise, even after the ruinously expensive week long shutdown, you could end up right back where you started.

  • oops

  • by Framboise (521772) on Tuesday January 17, 2012 @01:29PM (#38726712)

    I wonder why nowadays they use an incrementing limited integer number (SCN), subject to the described bugs, instead of a worldwide consistent and unlimited number like the TIME. The synchronization of the databases respective times can always occur with the NTP service (http://en.wikipedia.org/wiki/Network_Time_Protocol [wikipedia.org]).

    • by MagicM (85041)

      Well, for one, NTP doesn't have a high enough resolution.

      "[NTP] can achieve 1 millisecond accuracy in local area networks under ideal conditions". (Wikipedia)

      "The SCN is a moving line that cannot be crossed. The line moves up by 16,384 every second" (TFA)

      • by Framboise (521772)

        Yes but not the synchronization. The computer clock does have a much higher resolution. Also a local atomic clock precision can be obtained with GPS.

        • Ha! GPS has been known to slew by a whole second on the first of january. Not nice in real time systems which rely on precise timing across a distributed system.

    • SCN is a number that gets increased *per transaction*, so is in a way related to time, but never in a 1-on-1 relation.

  • Who exposes their Oracle DB server to the outside world anyway? Surely its just accessible from the servers that need it. Anyone know any public Oracle DB servers? Lemme just scan the interwebs...

    Of course if your front-end gets pwned then you don't want your DB server getting rooted, but hey, they got your front end server... Hopefully that will only have restricted access to the databases it needs, so an Oracle remote exploit here could let an attacker get to anything on the server...

    Either way up, not a

    • by Rary (566291) on Tuesday January 17, 2012 @01:45PM (#38726918)

      Generally, a database flaw like this is of relatively minor concern for exactly that reason. In order for the flaw to be exploited, the attacker has to already have gotten past other layers of security. However, there is a pretty damaging aspect to this flaw: you don't need admin access to exploit it. Anyone with the ability to query the database can do damage. Obviously, anyone who gets that far is already in a position to do some serious damage even without this flaw, but it does add some insult to injury.

      • by sjames (1099)

        That and even a database with read-only access to another (implying that it shouldn't be able to do any damage at all) can effectively write the SCN.

    • by Hulfs (588819) on Tuesday January 17, 2012 @01:52PM (#38727024)

      The problem is that if you have your Oracle DB's linked together in the fashion described in the article, having just a single little random Oracle DB owned can result in a DOS of literally every Oracle DB in your company that is linked together. It's not limited to just the DB connected to the front end that was compromised.

      Furthermore, from what I understood from the article, the only real way to recover from the DOS is to restore EVERY database from a backup after rolling back the SCN number on EVERY database you run. If you miss rolling back and updating just a single one, you're hosed again.

      This is a really insidious bug.

      • Re: (Score:2, Informative)

        by Anonymous Coward

        Actually from the article, restoring a backup won't help. The SCN number is in there. They describe having to dump the Schema and Data to a newly created Oracle database. That would be a nightmare.

        • A dump and import into a new database isn't that tricky, but would take a bit of planning on multiple connected systems. The easiest way would be to build a duplicate system to import into, but that requires duplicate hardware.
      • by shentino (1139071)

        I'd say that there's a problem with a database vulnerable to this sort of cascading failure to begin with.

    • by rubycodez (864176)
      you're not thinking fourth-dimensionally, Marty. All you need is a stinky reporting database accessed from web server, that happens to have read-only link to the mission critical one. Then SQL injection attack on web server to wind up the SCN number....
    • by afabbro (33948)

      Who exposes their Oracle DB server to the outside world anyway

      Who says security problems are exclusive to external attackers? Plenty of internal people to worry about, particularly when you consider how many big companies outsource their IT support.

  • "So hackers could not exploit the bug in the meantime." Only the hackers (crackers?) that don't already know about it, and have no way of learning from their black-hat friends.
    • by Rary (566291) on Tuesday January 17, 2012 @01:48PM (#38726960)

      This isn't security through obscurity. This is an attempt to mitigate the damage while the flaw is being patched. Security through obscurity would be if they chose not to solve it, relying instead on nobody figuring it out.

      • by Migala77 (1179151)
        I have heard from someone at Oracle that for them it is forbidden to admit any Oracle software has security bugs. All public references to something that turns out to be a security bug will be removed or replaced with some non-related issue. As in TFA: "... a number of Oracle sources for this story [...] noted that Oracle licensing agreements prevented them from commenting on any aspect of their product usage". Infoworld delaying the story is not an example, but security through obscurity seems to be The O
        • by Rary (566291)

          That's not what "security through obscurity" means. That's just damage control and PR. "Security through obscurity" means that the system's security is designed such that it only works if its implementation is unknown to attackers. Unfortunately, people frequently throw the phrase around whenever a situation like this occurs, further diluting the phrase to the point that it has become almost meaningless.

  • The issue seems to be fundamentally down to someone with DBA rights on a database issuing "ALTER DATABASE BEGIN BACKUP" which then causes an "interlinked" database to also increment its SCN; anyone know what the "interlinking" is? I'm guessing DB links but it's a bit vague on details and high on the scaremongering... FWIW, the ALTER DATABASE command will require DBA rights to implement, so I'm not seeing the apocalypse that Infoworld is punting; if you've got DBA rights, you can do lots of stuff like drop
    • No, the "ALTER DATABASE BEGIN BACKUP" was just how the issue was first discovered. The issue is that someone with *low* privileges on an obscure, minor DB server can bump the SCN on that server, which if it happens to be linked to any other servers (like your big important one), causes the SCN to get bumped on those servers. So you can DoS all the other servers.
    • by rubycodez (864176)
      there are other ways for ordinary users to ratchet up the SCN number, including the example in article given of a read-only linked reporting database having its transactions/sec pumped up
  • As usual with Oracle, the patch will be a 4GB download. Considering how much they charge for that junk, it's amazing those morons haven't figured out how to just simply use rpm/yum or apt.

  • by Anonymous Coward

    Ca 1997 I did telnet oracleserver.myemployer.com 1521. Then typed some random characters. That ceased the connection but made the telephone ring with the DBA asking me what I did to Oracle, as it crashed. Of couse, I have zero knowledge of Oracle's binary protocol nor did I enter any passwords. Maybe the Ora listener is now a bit more robust, but then it was utter shite.

  • by vk2 (753291) on Tuesday January 17, 2012 @02:28PM (#38727456) Journal
    We have been battling this since many a months now. Oracle's solution? set "_max_reasonable_scn_rate"=16384
  • by Anonymous Coward
    Information on the System Change Number (SCN) and how it is used in the Oracle Database [ID 1376995.1]
    Modified 17-JAN-2012 Type BULLETIN Status PUBLISHED
    Applies to:

    Oracle Server - Enterprise Edition - Version: 10.1.0.5 to 11.2.0.3 - Release: 10.1 to 11.2
    Information in this document applies to any platform.
    Purpose

    Read this article to get a high level overview of how a logical timestamp, called the System Change Number (SCN), is used to order database events, and how the advance of this l
  • This has the smell of Lamport's bakery algorithm [microsoft.com].

    What is significant about the bakery algorithm is that it implements mutual exclusion without relying on any lower-level mutual exclusion. ...
    I don't know how many people realize how remarkable this algorithm is. Perhaps the person who realized it better than anyone is Anatol Holt, a former colleague at Massachusetts Computer Associates. When I showed him the algorithm and its proof and pointed out its amazing property, he was shocked. He refused to believ

    • by fusiongyro (55524)

      What's going on here is actually MVCC [wikipedia.org]. The idea here isn't to implement mutual exclusion so much as to ensure that concurrent writes can occur but nobody sees information outside their transaction; each transaction gets an ID assigned to it which is 1 + the previous transaction ID (in an idealized, serializable exclusion state) and your transaction can see any information with your ID or a lower (but committed) transaction ID. Transactions begun after yours can see your effects only after you've committed,

      • by Hartree (191324)

        The backup utility is just one way to do this. A malicious use of it wouldn't be bound by that.

        Assume you've subborned a low level DB by whatever means and have control of the machine it's running on.

        Now that you know that a simple SCN bump can propagate you just have to either forge that, or get the local DB to send your calculated value. You already have the credentials for it in the DB itself. It'd take some work to make it so a script kiddy could do it, but might not be prohibitively difficult.

        As an asi

        • by fusiongyro (55524)

          (Never seen this word "subborn" before.)

          I read the article, and I must have missed mention of any other way than through ALTER TABLE BEGIN BACKUP to get the SCN to increment dramatically. I think it would be hard to do a better job than than Oracle's bug, which according to the article could achieve a rate of millions or billions of increments per second, by simply running a trivial transaction in a loop. Could it be done? Maybe, but that could be a lot of work.

          Now, my impression is that this is a really ob

          • by Hartree (191324)

            From the article on page 3: (Though they don't give the actual commands, they should be pretty straightforward to figure out.)

            "But the risk of incrementing the SCN via the backup bug is not the only cause for concern. Perhaps the most important part of our finding is that the SCN can be incremented by anyone who can issue commands on an interconnected database."

            Note the phrasing: "anyone who can issue commands on an interconnected database".

            Not an admin, not a role with backup rights, or anything specific.

            • by Hartree (191324)

              Gah. Had a mind fade. (It's been a few years). Somehow got "bob" tangled up with scott/tiger in my mind.

              It looks like it may require the DBA role. Not sure it takes that. But still, it means you only have to get it on a low level server and it can propagate the error to any other that's linking to it. Even if the links are read only.

              • by fusiongyro (55524)

                Every transaction that results in a change will increment the SCN, which is what the article is implying. This is a big part of databasery, so of course there will be unprivileged (i.e. non-admin) users who can increment the SCN through the usual manner of running transactions. My point is that with the ability to run transactions but without the ability to run ALTER DATABASE BEGIN BACKUP (i.e. a non-admin user with write access) it will be hard to beat ALTER DATABASE BEGIN BACKUP at the game of incrementin

                • by Hartree (191324)

                  Are you even reading the same article that I am?

                  It took me about 5 minutes to find the "undocumented and hidden" commands that let you directly change the SCN. (I'm not posting it, but you can google it easily if you like.)

                  For the method I'm thinking of, you need DBA rights. But if you've rooted a machine, that's not much of a barrier. You can change and then change back the sys password with known means and standard tools that are present up to somewhere in the 11 series. (And even that isn't a barrier, as

                  • by fusiongyro (55524)

                    But if you've rooted a machine, that's not much of a barrier.

                    That's my whole point. You've already rooted the machine. You can do anything. The bug is irrelevant. You can just shut the database down. You can delete the cluster. Anything. This bug has nothing to do with it.

                    Further, like many exploits, only one person needs to be good enough to develop the exploit. Then it can be packaged in a script that any random can use.

                    Your pre-packaged exploit is unable to use this bug to root the machine, and furthermo

                    • by Hartree (191324)

                      If it makes you feel better to call me a moron, great. I find looking at pictures of cute kittens helps my day. YMMV.

                      I think you've misunderstood the impact of this bug.

                      The threat is not that controlling one machine can let you delete that one node or shut the instance down. It's that having taken control of that node, which may for very good reason be in a less secure area (physically and network wise) than a main data center, you can then use it to not only lock up the systems in that protected center but

                    • by fusiongyro (55524)

                      First, I'm sorry about being a dick. I think not being able to hit Wikipedia yesterday really soured my mood. Yes, that's what I meant about DAGs, but it was just a dick thing to say.

                      So, you make a good point, and I'm calm enough to see it today. I still think the article oversells the danger of this bug. We don't use dblink or hotbackup at my site (we're transitioning away from Oracle too). My friend sysadmins for a company that has a much larger clustered Oracle instance, and were using this hotbackup fac

                    • by Hartree (191324)

                      No worries. My own reply was a bit more tart, but I got distracted by a customer for a few minutes and then rewrote it.

                      Yep, there's an awful lot of "OMG! Dire Threat! So you must buy our magic security dust" out there.

                      DBAs (or sysadmins) catch a lot of problem due to the differing priorities in a company.

                      The developers and engineers are judged by getting product ready and aren't too likely to get in major trouble over a DB or machine security problem. And they generally aren't DBAs or have that mindset that

  • Or, actually, a family of them and at the level of basic architecture.

    Get into a low level database via some poorly secured login (or conceivably a SQL injection. Maybe. Not clear if that's possible) and take down any DBs that are linked just by snapshots.

    And the recovery can be so bad as rebuilding every single linked instance (and don't miss one, or you fail.) and reimporting the data.

    Ouch, ouch, ouch. I can think of all sorts of mayhem that could be done with this by someone with destructive intent.

    They

  • So at this moment I've scanned through all of the comments and haven't seen anyone who actually works with a sizable Oracle deployment make any sort of informative comment. So, Oracle DBA's, is this silly InfoWorld fear mongering or something you and your organization (or a larger org) should actually be seriously concerned about? To me it seems like under the right conditions this could bring an entire org's OracleDB structure down, but then again, I've never actually worked with it in production....

    • by Hartree (191324)

      Standard Oracle. They hack up a patch so you can diagnose the problem but don't fix the base cause.

      If they're still like they were in the mid 90s, they may blow some smoke about it being fixed in the next release, but never really fix it.

      And AT&T was paying for what ultraviolet level of metal support?

      Guess I can't blame them to much when it's so deep into the system.

  • If you're worried... then check out your SCN with this SQL:

    su - oracle
    . oraenv
    sqlplus "/ as sysdba"
    column GET_SYSTEM_CHANGE_NUMBER format 999,999,999,999,999,999,999
    select DBMS_FLASHBACK.GET_SYSTEM_CHANGE_NUMBER from dual;


    Millions or Billions... no problem.
    If you're starting to get close to 281 Trillion (actually 281,474,976,710,656)... time to panic. Remember, that's a US Trillion... not a UK Trillion [wikipedia.org]:
    • by EETech1 (1179269)

      Any reason you have to use 21 formatters for a 15 character long number?

      Just curious!

      Cheers:)

      • Too lazy to count... :-D
        • by EETech1 (1179269)

          :)

          Would that cause it to print any number out to 21 digits, or would it only spit out the digits used?

          (Now updating my linkedin profile to include Oracle DBA)

          Cheers!

  • Scott/tiger

  • Can't break it. Can't break in.

  • MongoDB is web scale! :P

You are in a maze of UUCP connections, all alike.

Working...