Serious Oracle Flaw Revealed; Patch Coming 100
GMGruman writes "A bug in Oracle Database that could take down large databases — or let a hacker do so — has been found, and Oracle promises a patch later today. When InfoWorld first heard of the bug two months ago, its investigation revealed how dangerous this bug could be, and after convincing Oracle to address the issue, InfoWorld held the news until a patch was available, so hackers could not exploit the bug in the meantime. Paul Venezia details just how this bug exposes companies to the possibility of databases going offline, and Eric Knorr asks Oracle users to help test the patch in their complex environments. (InfoWorld's tests in simpler environments show the patch works there.)"
Re:in before (Score:4, Funny)
The only reason Oracle has this flaw is because Microsoft's DB lineup hardly forces them to compete from a security perspective.
Nice Slashvertisement (Score:2, Interesting)
...brought to you by InfoWorld! Submitted by InfoWorld! Seriously, how much is /. getting behind the scenes from the various IT rags that plaster the front page?
Re: (Score:2, Funny)
Seriously, how much is /. getting behind the scenes from the various IT rags that plaster the front page?
At least enough money that it's worthwhile to keep doing it?
Re: (Score:3)
Re: (Score:3)
It's not, at least not yet. I think a bigger problem is you have so many people posturing and proclaiming and acting as experts and simply flat-out speculating (incorrectly and/or uselessly), that the noise to signal ratio makes it less and less useful to read unless I want to spend ages digging through the cruft to figure out what's actually insightful or informative. In too many cases, merely targeting the +4 or +5 moderated posts doesn't guarantee that you're going to read something that is, you know, ac
Re:Nice Slashvertisement (Score:4, Funny)
If /. is just trade rag bullshit, then why even come here?
First posts and goatse.
Re: (Score:3)
That has a nice ring to it.
Re: (Score:2)
I can't wash that mental image away from my mind...
Re: (Score:3)
If /. is just trade rag bullshit, then why even come here?
Because it's all put into perspective by people like you.
Re:Nice Slashvertisement (Score:5, Insightful)
Given that it's a fairly decent article about a somewhat (or very, for large companies) significant bug in a widely-used database, I think it still qualifies as "News for Nerds", doesn't it?
Re: (Score:3)
Not by much, "Corporate Expensiveware Has Bug, Film At 11" isn't interesting nor very "nerdy" on its own. If you watch what gets frontpaged and who the submitters are lately, it should become obvious that modern /. is just clickbait for the big IT rags. I doubt this is merely coincidence, so I'm wondering what's going on behind the scenes.
Re: (Score:3)
Yes, but it's terribly nerdy to blithely dismiss these stories if you don't use/like/know-anything-about the software, isn't it?
Thus, still really suitable for posting to this site.
Re: (Score:2)
Yes, but it's terribly nerdy to blithely dismiss these stories if you don't use/like/know-anything-about the software, isn't it?
Yeah, I work in an Oracle shop. I've also been on /. since it was on Malda's Alpha. My point is the intent of the submission, since there's plenty of nerdy stuff to post besides some piece about turtleneckboy's latest crapware bug written up on just another industry rag.
Re: (Score:2)
I'd disagree but I guess we'll find out how irrelevant this news is when we find that our credit cards aren't being accepted because the processor has had to shut their database again while they patch another Oracle database. I use /. as a one-stop shopping for much of my tech news. I don't find the occasional article about non-Linux software to be be a problem. I know I learned something new today. While I previously knew about the SCN, I didn't know it was a built-in timebomb. Oracle did a couple of thing
Re: (Score:2)
The way the failure can hop from DB to DB as they connect was interesting. It's a bug that causes failures that propagate virus like.
Re: (Score:2)
Not by much, "Corporate Expensiveware Has Bug, Film At 11" isn't interesting nor very "nerdy" on its own.
In this case, there was a very detailed and interesting explanation of why and how the bug is caused, and it doesn't require you to know anything about Oracle to appreciate the engineering choice made and why the bug hits it.
Re: (Score:2)
Re: (Score:2)
Yet I would rate TFA as interesting and good to know.
They're "patching" Larry Ellison? (Score:2)
Re:They're "patching" Larry Ellison? (Score:5, Funny)
Original source (Score:4, Interesting)
I assume they're referring to:
http://www.oracle.com/technetwork/topics/security/cpujan2012-366304.html [oracle.com]
My mystification is what is the venn diagram intersection of mysql server, virtualbox, and oracle 11G? Without any details I'm guessing a package signing key got owned?
Re: (Score:2)
Nope. RTFA.
Short answer. Very large Enterprise levels Oracle installations with multiple, interconnected databases shouldn't perform backups.
So, no problemo. Business as usual!
Re: (Score:3)
Re: (Score:2)
there are plenty of other ways to perform hot backups than the ALTER DATABASE/TABLESPACE BEGIN BACKUP way.
How?
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
Re:Original source (Score:5, Interesting)
Reading that article kept bringing forward more "oh no" realisations, stemming from the following points:
The "recovery" for exceeding the soft limit is to shut down the databases until the SCN goes below the soft limit. From then on, you just have to hope that no databases you're synchronising with will have a SCN that is close to (or beyond) this soft limit.
Re: (Score:3)
That's the really scarey part. If you miss just one single database anywhere in the enterprise, even after the ruinously expensive week long shutdown, you could end up right back where you started.
UNBREAKABLE (Score:2)
oops
NTP instead of SCN? (Score:3)
I wonder why nowadays they use an incrementing limited integer number (SCN), subject to the described bugs, instead of a worldwide consistent and unlimited number like the TIME. The synchronization of the databases respective times can always occur with the NTP service (http://en.wikipedia.org/wiki/Network_Time_Protocol [wikipedia.org]).
Re: (Score:3)
Well, for one, NTP doesn't have a high enough resolution.
"[NTP] can achieve 1 millisecond accuracy in local area networks under ideal conditions". (Wikipedia)
"The SCN is a moving line that cannot be crossed. The line moves up by 16,384 every second" (TFA)
Re: (Score:2)
Yes but not the synchronization. The computer clock does have a much higher resolution. Also a local atomic clock precision can be obtained with GPS.
Re: (Score:2)
Ha! GPS has been known to slew by a whole second on the first of january. Not nice in real time systems which rely on precise timing across a distributed system.
SCN isn't 'time' (Score:2)
SCN is a number that gets increased *per transaction*, so is in a way related to time, but never in a 1-on-1 relation.
Re: (Score:2)
But that's just the point. Third normal form requires just one basket for all the eggs! Any DBA worth their salt knows this.
You expose your DB server? (Score:2)
Who exposes their Oracle DB server to the outside world anyway? Surely its just accessible from the servers that need it. Anyone know any public Oracle DB servers? Lemme just scan the interwebs...
Of course if your front-end gets pwned then you don't want your DB server getting rooted, but hey, they got your front end server... Hopefully that will only have restricted access to the databases it needs, so an Oracle remote exploit here could let an attacker get to anything on the server...
Either way up, not a
Re:You expose your DB server? (Score:5, Interesting)
Generally, a database flaw like this is of relatively minor concern for exactly that reason. In order for the flaw to be exploited, the attacker has to already have gotten past other layers of security. However, there is a pretty damaging aspect to this flaw: you don't need admin access to exploit it. Anyone with the ability to query the database can do damage. Obviously, anyone who gets that far is already in a position to do some serious damage even without this flaw, but it does add some insult to injury.
Re: (Score:2)
That and even a database with read-only access to another (implying that it shouldn't be able to do any damage at all) can effectively write the SCN.
Re:You expose your DB server? (Score:5, Interesting)
The problem is that if you have your Oracle DB's linked together in the fashion described in the article, having just a single little random Oracle DB owned can result in a DOS of literally every Oracle DB in your company that is linked together. It's not limited to just the DB connected to the front end that was compromised.
Furthermore, from what I understood from the article, the only real way to recover from the DOS is to restore EVERY database from a backup after rolling back the SCN number on EVERY database you run. If you miss rolling back and updating just a single one, you're hosed again.
This is a really insidious bug.
Re: (Score:2, Informative)
Actually from the article, restoring a backup won't help. The SCN number is in there. They describe having to dump the Schema and Data to a newly created Oracle database. That would be a nightmare.
Re: (Score:2)
Re: (Score:2)
I'd say that there's a problem with a database vulnerable to this sort of cascading failure to begin with.
Re: (Score:2)
Yeah like slashdot's 24 bit post number.
Re: (Score:2)
Re: (Score:2)
Who exposes their Oracle DB server to the outside world anyway
Who says security problems are exclusive to external attackers? Plenty of internal people to worry about, particularly when you consider how many big companies outsource their IT support.
security through obscurity, yet again (Score:2)
Re:security through obscurity, yet again (Score:5, Informative)
This isn't security through obscurity. This is an attempt to mitigate the damage while the flaw is being patched. Security through obscurity would be if they chose not to solve it, relying instead on nobody figuring it out.
Re: (Score:2)
Re: (Score:2)
That's not what "security through obscurity" means. That's just damage control and PR. "Security through obscurity" means that the system's security is designed such that it only works if its implementation is unknown to attackers. Unfortunately, people frequently throw the phrase around whenever a situation like this occurs, further diluting the phrase to the point that it has become almost meaningless.
"Interlinked" databases? (Score:2)
Re: (Score:3)
Re: (Score:2)
As usual with Oracle, the patch will be a 4GB (Score:1)
As usual with Oracle, the patch will be a 4GB download. Considering how much they charge for that junk, it's amazing those morons haven't figured out how to just simply use rpm/yum or apt.
Re: (Score:2)
Using The Hacker Tool TELNET (Score:1)
Ca 1997 I did telnet oracleserver.myemployer.com 1521. Then typed some random characters. That ceased the connection but made the telephone ring with the DBA asking me what I did to Oracle, as it crashed. Of couse, I have zero knowledge of Oracle's binary protocol nor did I enter any passwords. Maybe the Ora listener is now a bit more robust, but then it was utter shite.
Wow!! This is coming up now? (Score:3)
Here's the Oracle Support doc about SCNs (Score:2, Informative)
Modified 17-JAN-2012 Type BULLETIN Status PUBLISHED
Applies to:
Oracle Server - Enterprise Edition - Version: 10.1.0.5 to 11.2.0.3 - Release: 10.1 to 11.2
Information in this document applies to any platform.
Purpose
Read this article to get a high level overview of how a logical timestamp, called the System Change Number (SCN), is used to order database events, and how the advance of this l
Re: (Score:2)
"Longer term Oracle will be raising the upper limit from 281 trillion to an even larger number."
I completely fail to see how that addresses the vulnerability.
So, they make it bigger, and bump up the rate at which the SCNs can increment.
Solution. Alter a couple of multipliers in your formula for figuring out what value is now the one to propagate.
Unless there's something I don't appreciate, this does nothing to prevent a malicious use of this.
baker's mini-mole (Score:2)
This has the smell of Lamport's bakery algorithm [microsoft.com].
Re: (Score:2)
What's going on here is actually MVCC [wikipedia.org]. The idea here isn't to implement mutual exclusion so much as to ensure that concurrent writes can occur but nobody sees information outside their transaction; each transaction gets an ID assigned to it which is 1 + the previous transaction ID (in an idealized, serializable exclusion state) and your transaction can see any information with your ID or a lower (but committed) transaction ID. Transactions begun after yours can see your effects only after you've committed,
Re: (Score:2)
The backup utility is just one way to do this. A malicious use of it wouldn't be bound by that.
Assume you've subborned a low level DB by whatever means and have control of the machine it's running on.
Now that you know that a simple SCN bump can propagate you just have to either forge that, or get the local DB to send your calculated value. You already have the credentials for it in the DB itself. It'd take some work to make it so a script kiddy could do it, but might not be prohibitively difficult.
As an asi
Re: (Score:2)
(Never seen this word "subborn" before.)
I read the article, and I must have missed mention of any other way than through ALTER TABLE BEGIN BACKUP to get the SCN to increment dramatically. I think it would be hard to do a better job than than Oracle's bug, which according to the article could achieve a rate of millions or billions of increments per second, by simply running a trivial transaction in a loop. Could it be done? Maybe, but that could be a lot of work.
Now, my impression is that this is a really ob
Re: (Score:2)
From the article on page 3: (Though they don't give the actual commands, they should be pretty straightforward to figure out.)
"But the risk of incrementing the SCN via the backup bug is not the only cause for concern. Perhaps the most important part of our finding is that the SCN can be incremented by anyone who can issue commands on an interconnected database."
Note the phrasing: "anyone who can issue commands on an interconnected database".
Not an admin, not a role with backup rights, or anything specific.
Re: (Score:2)
Gah. Had a mind fade. (It's been a few years). Somehow got "bob" tangled up with scott/tiger in my mind.
It looks like it may require the DBA role. Not sure it takes that. But still, it means you only have to get it on a low level server and it can propagate the error to any other that's linking to it. Even if the links are read only.
Re: (Score:2)
Every transaction that results in a change will increment the SCN, which is what the article is implying. This is a big part of databasery, so of course there will be unprivileged (i.e. non-admin) users who can increment the SCN through the usual manner of running transactions. My point is that with the ability to run transactions but without the ability to run ALTER DATABASE BEGIN BACKUP (i.e. a non-admin user with write access) it will be hard to beat ALTER DATABASE BEGIN BACKUP at the game of incrementin
Re: (Score:2)
Are you even reading the same article that I am?
It took me about 5 minutes to find the "undocumented and hidden" commands that let you directly change the SCN. (I'm not posting it, but you can google it easily if you like.)
For the method I'm thinking of, you need DBA rights. But if you've rooted a machine, that's not much of a barrier. You can change and then change back the sys password with known means and standard tools that are present up to somewhere in the 11 series. (And even that isn't a barrier, as
Re: (Score:2)
That's my whole point. You've already rooted the machine. You can do anything. The bug is irrelevant. You can just shut the database down. You can delete the cluster. Anything. This bug has nothing to do with it.
Your pre-packaged exploit is unable to use this bug to root the machine, and furthermo
Re: (Score:2)
If it makes you feel better to call me a moron, great. I find looking at pictures of cute kittens helps my day. YMMV.
I think you've misunderstood the impact of this bug.
The threat is not that controlling one machine can let you delete that one node or shut the instance down. It's that having taken control of that node, which may for very good reason be in a less secure area (physically and network wise) than a main data center, you can then use it to not only lock up the systems in that protected center but
Re: (Score:2)
First, I'm sorry about being a dick. I think not being able to hit Wikipedia yesterday really soured my mood. Yes, that's what I meant about DAGs, but it was just a dick thing to say.
So, you make a good point, and I'm calm enough to see it today. I still think the article oversells the danger of this bug. We don't use dblink or hotbackup at my site (we're transitioning away from Oracle too). My friend sysadmins for a company that has a much larger clustered Oracle instance, and were using this hotbackup fac
Re: (Score:2)
No worries. My own reply was a bit more tart, but I got distracted by a customer for a few minutes and then rewrote it.
Yep, there's an awful lot of "OMG! Dire Threat! So you must buy our magic security dust" out there.
DBAs (or sysadmins) catch a lot of problem due to the differing priorities in a company.
The developers and engineers are judged by getting product ready and aren't too likely to get in major trouble over a DB or machine security problem. And they generally aren't DBAs or have that mindset that
That's a pretty bad bug. (Score:2)
Or, actually, a family of them and at the level of basic architecture.
Get into a low level database via some poorly secured login (or conceivably a SQL injection. Maybe. Not clear if that's possible) and take down any DBs that are linked just by snapshots.
And the recovery can be so bad as rebuilding every single linked instance (and don't miss one, or you fail.) and reimporting the data.
Ouch, ouch, ouch. I can think of all sorts of mayhem that could be done with this by someone with destructive intent.
They
Re: (Score:2)
"This whole design of escalating each other's SCN's just for the purposes of synchronization is just plain stupid. Especially when the SCN has a hard-coded maximum."
That was kinda my thought, but didn't want to say that was the fix as there are always complications. Dog knows what all they've got depending on their particular format and rules for the SCNs.
The big thing would be figuring a way to do it so that existing installs can be made compatible with that without requiring rebuilds, or at least that all
Can anyone really comment on this? (Score:1)
So at this moment I've scanned through all of the comments and haven't seen anyone who actually works with a sizable Oracle deployment make any sort of informative comment. So, Oracle DBA's, is this silly InfoWorld fear mongering or something you and your organization (or a larger org) should actually be seriously concerned about? To me it seems like under the right conditions this could bring an entire org's OracleDB structure down, but then again, I've never actually worked with it in production....
Very old problem (Score:2)
Re: (Score:2)
Standard Oracle. They hack up a patch so you can diagnose the problem but don't fix the base cause.
If they're still like they were in the mid 90s, they may blow some smoke about it being fixed in the next release, but never really fix it.
And AT&T was paying for what ultraviolet level of metal support?
Guess I can't blame them to much when it's so deep into the system.
SQL to check your database (Score:1)
su - oracle
. oraenv
sqlplus "/ as sysdba"
column GET_SYSTEM_CHANGE_NUMBER format 999,999,999,999,999,999,999
select DBMS_FLASHBACK.GET_SYSTEM_CHANGE_NUMBER from dual;
Millions or Billions... no problem.
If you're starting to get close to 281 Trillion (actually 281,474,976,710,656)... time to panic. Remember, that's a US Trillion... not a UK Trillion [wikipedia.org]:
Re: (Score:1)
Any reason you have to use 21 formatters for a 15 character long number?
Just curious!
Cheers:)
Re: (Score:1)
Re: (Score:1)
:)
Would that cause it to print any number out to 21 digits, or would it only spit out the digits used?
(Now updating my linkedin profile to include Oracle DBA)
Cheers!
Old but gold (Score:1)
Scott/tiger
Unbreakable! (Score:1)
Can't break it. Can't break in.
Re: (Score:1)