New Log4J Flaw Caps Year of Relentless Cybersecurity Crises (wsj.com) 77
'Exhausted' network defenders say technological dependency creates new vulnerabilities. From a report: Cyberattacks on major technology providers and the interconnected world of software and hardware that power the global economy continued at a relentless pace in 2021, according to U.S. officials and security experts. Instead of one company being victimized at a time like in a traditional data breach, thousands were often exposed simultaneously. Businesses, hospitals and schools also worked to defend themselves against an onslaught of ransomware attacks, which increasingly reap $10 million or more in extortion payments. The annus horribilis culminated this month with discovery of a flaw in an obscure but widely used internet code known as Log4j, which one senior Biden administration official said was the worst she had seen in her career. The latest vulnerability comes as U.S. officials warn corporate leaders of a potential surge of cyberattacks while businesses slow their operations during the holiday season.
The string of incidents highlights how decades of digital transformation have linked business and government computer systems in opaque and sometimes surprising ways that will create new vulnerabilities. Major disruptions are certain to continue, cybersecurity officials said. "Network defenders are exhausted," said Joe Slowik, threat-intelligence lead at the security firm Gigamon. New attention and investment in cybersecurity hasn't improved the status quo, he said. "Money is flowing into the field, but largely on technical solutions while the core need -- more capable people -- remains hard to address."
The string of incidents highlights how decades of digital transformation have linked business and government computer systems in opaque and sometimes surprising ways that will create new vulnerabilities. Major disruptions are certain to continue, cybersecurity officials said. "Network defenders are exhausted," said Joe Slowik, threat-intelligence lead at the security firm Gigamon. New attention and investment in cybersecurity hasn't improved the status quo, he said. "Money is flowing into the field, but largely on technical solutions while the core need -- more capable people -- remains hard to address."
we need better architects (Score:2)
this wouldn't happen if we had better architects, that take security seriously...
In how many companies is security not bothered with on the altar of reducing spending ?
Re:we need better architects (Score:4, Informative)
For more info on the subject I suggest watching the Computerphile video: https://www.youtube.com/watch?... [youtube.com]
Re:we need better architects (Score:4, Informative)
There is a perception in both the private and government sector, that security, both physical and digital, is something you can buy. Witness the mammoth growth of airport security products following 9/11, and the sheer number of vendors at security conferences. With that, government officials and corporate executives often think you can simply buy products and magically get instant security by flipping on the switch. The reality is that security is not something you can buy; it is something you must get.
-- Bruce Schneier, Schneier on Security [schneier.com]
Re: (Score:3)
Re: (Score:2)
Neh, let's just hire 20 H1B-stackoverflow-copypaste-drones instead.
Re: (Score:2)
It isn't the problem with people that know (including Architects), but the people that THINK they know, but don't.
Then there is the next level of bullshit, once its hits the MBA level, then knowing is a positive hindrance. If you do actually know something, one cannot indeed hit that level. If you do, you lose something on the way. "Did you ever hear the tragedy of Darth Plagueis The Wise? I thought not. Itâ(TM)s not a story the MBA would tell you."
Re: we need better architects (Score:1)
Re: (Score:2)
Re: (Score:2)
this wouldn't happen if we had better architects, that take security seriously... In how many companies is security not bothered with on the altar of reducing spending ?
Security, even today, is still viewed as a FUD product. And you usually can't sell it until that Doubt is confirmed by an attack bad enough that a judge orders it. And to be quite honest, Greed N. Corruption doesn't really give a shit when most of the impact flows away from them like water running off a CxOs golden parachute.
And remember the most obvious problem; his wouldn't happen if users were smarter and actually gave a shit about security or privacy. They're still not, and it's getting worse. The b
Re: (Score:2)
We need better people every kind of position. Good people are both shockingly expensive and at the same time a bargain; it's a question of your planning horizon. Most organizations try to solve a problem with the cheapest labor possible, which is a short term optimization.
The problem is there isn't enough of people with high level thinking skills. Obviously the way to address that with education, but the focus on incorporating more STEM subject matter into lower education isn't going help much; it's only
Re: Woohoo (Score:2)
Re: (Score:1)
Actually: it does. Closed source is worse
Actually, both have their advantages and disadvantages.
The typical arguments are open source is better because:
And against it...
Re: (Score:2)
Lots of eyes check for vulnerabilities (log4j seems to prove the opposite)
If anything, log4j proves that there's too much software out there.
Re: (Score:2)
But..., Someone is paid to do it.
Most paid software don't pay someone do it either. Code goes in. No one looks at it. Gets sold. Gets used. People who pay for it are locked in, and so locked in to the bug whenever the software is no longer commercially supported.
Re: Woohoo (Score:2)
I'd say that the bug found just happened to have a certain chain of dominoes falling in the perfect way.
From what I have seen it's the macro functionality in log4j that worked in conjunction with jndi causing the issue.
Re: (Score:1)
Re: (Score:2)
Pottering copypasta'd it into systemd.
I still love how it was first found in Minecraft. (Score:2)
I mean, no one has ever audited the repo for Log4J. This remote execution has been in there for a while. It was only when somone was trying to hack the old Java version of Minecraft that the guy found it.
Still.. While I don't get why log4j2 is getting all the flack. From the code and what I read, its just because of older versions of Java that were patched. Not to say going from 8 to 9 is easy.
Re: (Score:2)
It would be more accurate to say that it's because it's because log4j was trying too hard to be "enterprise", so the default configuration enabled some really powerful functionality that only a small minority of users would want.
Re:I still love how it was first found in Minecraf (Score:5, Interesting)
From the code and what I read, its just because of older versions of Java that were patched.
Yes, some idiots claimed that. But it is not. It is a "feature" in log4J that has nothing to do there and got abused as bug/exploit.
The background is two fold:
a) that log4j version can load java classes remotely (why the programmers even added that feature: beyond me) - via ldpa
b) log4j supports pattern substitution, so you can have a message like log("we are here ${context.variable}" + some _text); and during logging that pattern gets expanded.
Unfortunately, that variable some_text might be data from an untrusted source, and the idiotic programmers thought: they do pattern expansion inside of those texts/variables, too!!
If I understand it correctly, if an outsider can smuggle something like "ldap.POT://some.remote.host.com:my.bad.class.HackIt" into the variable some_text above, log4J will try to expand it and load the referenced class via ldap, a complete braindead idea by the programmers.
Re: (Score:2)
Bloat. Pure and simple. This is why Unix's original premise to have many small tools that each did very, very specific things, was brilliant in a number of ways. It's ridiculous that a library for logging can basically be turned into a shell or OS capable of loading and executing code from any remote location.
Trying their hardest to adhere to Zawinski's Law apparently.
Ironically he also said this:
Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
Re: (Score:2)
It's ridiculous that a library for logging can basically be turned into a shell or OS capable of loading and executing code from any remote location.
Yes, it is ridiculous.
Consider the time spent to even implement that useless feature.
Somehow I have the feeling the "exploit" was placed there intentionally.
Re: (Score:2)
Consider the time spent to even implement that useless feature.
Somehow I have the feeling the "exploit" was placed there intentionally.
I doubt it took much time at all. JNDI already existed to handle the remote fetch and execute. Log4j just used JNDI.
As for why they did that. I'm going on a hypothetical here. Assume someone has 100 servers and they want to send an email when something happens. So they code up a e-mailer class that would now have to be pushed out to all 100 servers on every patch.
How about instead of updating all 100 servers, we have the servers pull down the e-mailer when that thing happens. That way we don't have to
Re: (Score:2)
As for why they did that. I'm going on a hypothetical here. Assume someone has 100 servers and they want to send an email when something happens. So they code up a e-mailer class that would now have to be pushed out to all 100 servers on every patch.
How about instead of updating all 100 servers, we have the servers pull down the e-mailer when that thing happens. That way we don't have to patch all 100 servers every time, they will all pull the update automatically. Marvelous! Time saved! Scheduling update windows no longer needed!
Why would anyone think that was good idea? Why?! There are countless better ways to accomplish the same thing safely and with less effort.
There's no hope for the future, is there?
Re: (Score:2)
The only hope I see is that people start learning from these kinds of mistakes.
And by "people" I include management, in spite of some evidence. :)
Management often hamstrings the efforts of experienced developers to design for security, reliability, flexibility, maintainability, and longevity.
I wish they wouldn't keep doing that. But most of them probably will, especially if they keep getting credit for minimizing costs and time to market, but not accountability for the harm that results from the resulting
Re: (Score:2)
Sadly, the time-tested "Unix's original premise" on Linux gave way to systemd some time ago on nearly all mainstream distributions.
I continue to use Gentoo in part for that reason; it is not systemd-based by default, though systemd is becoming more popular within the Gentoo ecosystem. I've been able to avoid it thus far.
It does still live on in the world of object-oriented programming, as the "single responsibility principle," whereby each class is supposed to do one thing well and to have minimal dependen
Re: (Score:2)
Re: (Score:2)
You can do that in .NET also, but I would consider that a very bad practice, unless you use code signing or some other way to validate the classes you're about to load.
Note: I've worked on more than one Java system (NOT for my current employer) where bits of code were stored in a database, so we could do things like change the names of report header columns without having to re-test and re-deploy the entire system. Of course there were likely much better ways to achieve the same result, like separating the
Re: (Score:2)
that log4j version can load java classes remotely (why the programmers even added that feature: beyond me) - via ldpa
I don't know how it got into log4j exactly, but it's a common "feature" in object serialization libraries, in many languages, and it always comes with security vulnerabilities.
Re: (Score:2)
Log4J is supposed to only log text. It does not (de-)serialize objects.
In object deserialization something like this could happen, if you can compromise the classpath. But that would not work remotely.
Re: (Score:2)
I think you misunderstand the relationship to Java. The Java side of this was also patched in Java 8, back in October 2018, so there was no need to jump Java versions.
The Log4j side of the problem is that Log4j thought it was a clever idea to enrich log entries by fetching information via JNDI/LDAP across the network. It apparently did not bother with any sort of checking or sanitizing: just fetch and blindly trust. This was inexcusably stupid.
The Java side of this was allowing Java code to be loaded vi
Re: (Score:2)
Or (b) they did install those updates, and then deliberately opened the barn door by re-enabling JDNI/LDAP code execution. In either case, inexcusably stupid.
Actually not stupid. Because they would not know that log4J had this bug/feature/exploit. Remember: it got discovered just a few weeks ago.
AND, there are plenty of reasons why you want to load classes via JDNI, for starters: JNDI is exactly designed to do that. However the problem is that people needing such features should have their services running
Re: (Score:2)
Library dependency hell is not only still a real thing, but inadvertently creating it is now considered a "best practice" by some.
"Don't reinvent the wheel. Let a library do it." This isn't always bad advice, but it absolutely does need to be balanced by an understanding of the ENTIRE dependency graph and the problems that might result from any security, compatibility, versioning, "diamond dependency," or other issues anywhere within that graph . . . often in parts that devs tend not to see or use directl
Re: (Score:2)
Biden administration official said... (Score:2)
Re: (Score:1)
was the worst she had seen in her career. Even after 2 weeks, we still don't even know what it was used for..
It's the Biden admin. Who are they kidding? It was also the first one she's ever seen I bet. I'm sure they didn't hire a real security person. This administration is so full of fakes it's not funny.
I can think of a bunch that was worse. Off the top of my head, Solar winds come to mind. That threw a bunch of agencies and companies into a whirlwind. Much worse than this one.
The Internet Worm (Score:2)
Another consequence of such wide spread vulnerabilities is that another menace that has already brought the Internet to its knees before becomes feasible again. A self replication worm exploiting this kind of vulnerability could bring our whole digital infrastructure down. The difference with 1988 is that practically our whole world is now depending on it.
https://en.wikipedia.org/wiki/... [wikipedia.org]
Mandatory Access Control (MAC) (Score:1)
Re: (Score:1)
The exploit code is a deserialised object running the in JVM. Not sure SELinux will make too much difference - this all happens in the JVM. SELinux can only enforce rules at that level.
And yes, you could lock down the JVM so it doesn't have network access (in which case the exploit wouldn't work), but that may not be practical depending on the app. And if the exploit does run, and the attackers gets code execution then they just need to find an LPE to get around SELinux. In that case, a good SELinux config
Re: (Score:2)
And in other news (Score:3)
Re: (Score:2)
Don't kid yourself. Both Facebook and Google have the information on whether you've been naughty or nice. They just prefer to use that to sell ads to coal companies (a big season for them - lumps of coal don't sell very well otherwise) rather than just give it to Santa.
Formatting as part of logging is a terrible idea (Score:3)
The way I see it, the problem is that log4j, like most other logging libraries out there, has formatting built into its logging.
That's just a terrible idea, which leads to similar problems as that of SQL injection etc.
The correct way to do logging, in my opinion, is to record serialized objects rather than a formatted string.
Re: (Score:2)
The way I see it, the problem is that log4j, like most other logging libraries out there, has formatting built into its logging.
That's just a terrible idea, which leads to similar problems as that of SQL injection etc.
The correct way to do logging, in my opinion, is to record serialized objects rather than a formatted string.
Unless you mean something else, I don't see formatting as being related to this issue at all, and making logs binary introduces more dependencies and thus more potential holes.
Log4j was trusting URLs and fetching from them. That's insane.
Re: Formatting as part of logging is a terrible id (Score:2)
I never suggested making the log use a binary format, though of course that becomes possible with such a system too.
Re: (Score:2)
I never suggested making the log use a binary format, though of course that becomes possible with such a system too.
You mentioned logging serialized objects. In Java, that's pretty much putting binary - well, java bytecode - directly into the log since Java has ways to save/load serialized classes.
That's actually what caused a massive security flaw in Apache Struts years ago as people were serializing a Java class and then including it in the headers of HTTP requests. Which meant the server would run the class before checking the user permissions.
Re: Formatting as part of logging is a terrible i (Score:2)
You can serialize your objects to JSON which is a textual format.
I never talked about any language-specific serialization mechanism.
Re: (Score:2)
Re: (Score:3)
The correct way to do logging, in my opinion, is to record serialized objects rather than a formatted string.
This exploit is in the (unexpected) deserializing of an object. Inevitably if you log a serialized object, you're going to want to deserialize it later... what could possibly go wrong? For my use of logs (diagnosing what went wrong after the fact), plain text is good for me. There are few exploits available when opening up a text file (no doubt there is something possible in many editors, probably with malformed Unicode).
The real flaw is that the format string could wind up executing arbitrary code based on
Re: Formatting as part of logging is a terrible id (Score:2)
We're talking about different types of serialization.
Deserializing should not allow arbitrary code evaluation like was happening in this case.
Re: (Score:1)
We're talking about different types of serialization.
Deserializing should not allow arbitrary code evaluation like was happening in this case.
but in reality, your premise is correct. Deserialization should never allow arbitrary code to run.
Re: Formatting as part of logging is a terrible id (Score:2)
Not necessarily a bad idea, but you need to be restrictive.
Sometimes you'd like to format certain information like client IP address or timestamp, and that's useful. But don't overdo it.
Re: Formatting as part of logging is a terrible i (Score:2)
The type of those values is what should decide how they get serialized/deserialized, not some formatters in a string.
Re: (Score:2)
The correct way to do logging, in my opinion, is to record serialized objects rather than a formatted string.
So just so you know, under the hood, that's exactly how log4j2 works. Sort of.
What gets logged is ultimately a LogEvent [apache.org] object. (LogEvent is an interface, not an actual class.) It's essentially a formatted string message with a whole bunch of other metadata about the event being logged attached to it - think log level, time it happened, where it happened, that kind of stuff.
The problem wasn't in logging a "formatted string." The problem wasn't the "formatting." The problem was that, for some unknown reason,
Re: Formatting as part of logging is a terrible id (Score:2)
Property substitution is part of formatting.
Freebies (Score:4, Informative)
A big problem is commercial vendors treating open source code as free, not of cost, but of responsibility. They use it so they don't have to commit resources to coding that function themselves, but then wash their hands at it. Log4j is a great example. There wasn't one application in our environment that had the latest version before this debacle. I counted half a dozen 2.x and another handful of 1.x, but none had been upgraded behind what worked for them when they built their app.
Even though the FOSS developers were providing free updates and patches, nobody bothers to update the components unless something breaks in their app. Testing a point update takes time, and time is money, and they aren't spending money unless absolutely forced.
Add to that the overall fragility of commercial applications... One of our major vendors we constantly fight with because while their app runs on Red Hat, we apply CIS hardening to all our systems and that breaks their application. So, in an effort to help, we the customer, went thru step-by-step to figure out exactly what hardened config was breaking their app and gave them all the information. They did nothing with it. It turns out their one-button patching/upgrade process is dependent on a generic RHEL install and that breaks differently. Per our support contract they ended up having to manually patch our system and it took them three days instead of an hour.
The best hope I see on the horizon is the feds demanding software bill of materials for anything sold to them. Then we can see all the out of date components the lazy, cheap bastards are using and say "patch that shit".
Up to date patching and configuration are by far and away the two most effective security steps you can take. Damn near every one of the CISA alerts say the same thing -- patch and configure. Just do it.
Re: (Score:3)
I counted half a dozen 2.x and another handful of 1.x, but none had been upgraded behind what worked for them when they built their app.
Even though the FOSS developers were providing free updates and patches, nobody bothers to update the components unless something breaks in their app. Testing a point update takes time, and time is money, and they aren't spending money unless absolutely forced.
I don't think it has anything to do with money and free.
It is a matter of culture. Supposed a freshling comes from
Re: Freebies (Score:2)
Newever versions, e.g. for log4J, suprisingly do not fix automatically unknown bugs (aka exploits).
Yes, but there are a bunch of other vulns in the older versions that would be fixed. 0-days are what defense in depth is for, but there are tons of unpatched, known vulnerabilities out there. See : https://duo.com/decipher/cisa-orders-federal-agencies-to-patch-hundreds-of-known-flaws [duo.com]
So, regarding culture: if you run an organization in a head position, you can demand that stuff gets audited and upgraded as soon as a new library is out and considered "stable". But you, or someone you are superior too, has to damn do it! Otherwise the software stays as it is.
I'm trying. We've just implemented a toll called Sonatype which tracks component versions, licenses, compatibilities, etc. We're very aggressively pushing an "keep every component updated" policy.
That would never have happened in any system I was involved in. And as the logs are audited: strange expansions most likely would been seen.
Really? Be honest. We have around 200 servers and monitor the logs but there are literally millions of entries every day. How quickly do you expect to catch the occasional odd expansion? Needle in a haystack doesn't do this justice.
Re: (Score:1)
Newever versions, e.g. for log4J, suprisingly do not fix automatically unknown bugs (aka exploits). Yes, but there are a bunch of other vulns in the older versions that would be fixed. 0-days are what defense in depth is for, but there are tons of unpatched, known vulnerabilities out there. See : https://duo.com/decipher/cisa-orders-federal-agencies-to-patch-hundreds-of-known-flaws [duo.com]
So, regarding culture: if you run an organization in a head position, you can demand that stuff gets audited and upgraded as soon as a new library is out and considered "stable". But you, or someone you are superior too, has to damn do it! Otherwise the software stays as it is.
I'm trying. We've just implemented a toll called Sonatype which tracks component versions, licenses, compatibilities, etc. We're very aggressively pushing an "keep every component updated" policy.
That would never have happened in any system I was involved in. And as the logs are audited: strange expansions most likely would been seen.
Really? Be honest. We have around 200 servers and monitor the logs but there are literally millions of entries every day. How quickly do you expect to catch the occasional odd expansion? Needle in a haystack doesn't do this justice.
Though to be fair, it can be done by doing a frequency analysis of the log messages themselves. This condenses the messages quite a bit. Added to that is you should know what messages you are likely to get, parse out what you know is there, and then investigate what is left.,
It's just frequency analysis on log files. Literally something you should be doing anyway.
Re: (Score:2)
Really? Be honest. We have around 200 servers and monitor the logs but there are literally millions of entries every day. How quickly do you expect to catch the occasional odd expansion? Needle in a haystack doesn't do this justice.
That is actually extremely easy if you indeed "eye ball" monitor your logs.
Basically everything that is unexpected causes an monsterous visual glitch on a scrolling text output.
Re: (Score:2)
All software requires ongoing maintenance, for these amongst many other reasons.
But management often won't pay for that maintenance. "It's not broke, so why 'fix' it?" Or, "We already paid for it once. Why should we have to pay again and again?" Car analogies should work, but for some reason they don't. "Of course I have to pay for gas. Of course I have to change the oil once in a while. But that's a car. This is software. It doesn't have physically movable parts. It's made out of ones and zeroes
Re: (Score:2)
I'm so sick of it that if I financially able, I would seriously consider a different career. :D
Same for me
Being a Software Consultant was a bad choice. It is kind of parody, they pay me for my advice but do not follow it. Then they fail, lol. Or half fail, most - hm, I think all - of my projects where deployed successfully, but not in the shape I hoped.
Re: (Score:2)
Re: (Score:2)
whether working for an employer or a private client, it's almost always been a struggle to define projects and get the needed resources
That never really was the problem, as that is often already set. But conducting and following technical advise, regardless if it is approach, architecture, tools or anything regarding sound craftsmanship always was a hardship.
Often they did not knwo what a revision control system is - or how it is used properly: "Oh, Sourcesafe is not good?"
Re: (Score:1)
Re: (Score:2)
Wish I had a cup of coffee for every time I evaluated software (usually but not always proprietary), pointed out problems of a similar nature, and was told in essence, "screw you, you're just trying to weasel out of your responsibility, which will be to make this Just Work. No excuses. Just get 'er done."
Also: management would usually like to think in terms of "buy or build" but it almost never works out that way. It's "buy and integrate," versus "build and integrate." Leaving the integration side out o
The von Neumann paradigm is the problem (Score:3)
Re: (Score:1)
How about this; software is vulnerable because it is meant to be used/maintained by the user. Software's primary users only mean to use it until it does not work anymore.
Maintained by the user? (Score:2)
Even open source is bloated. "Show me the code" seems to have become a cargo cult.
Re: (Score:2)
That author seems a little confused about what a real-time system actually means. https://en.wikipedia.org/wiki/... [wikipedia.org]
In a real time system, every operation (or function call or whatever) will be completed within a defined amount of time.
Re: (Score:2)
Wow, that was ... awful. Find better heroes.
feeping creaturism (Score:2)
If you keep adding infinite features onto a kludgy system, you get infinite complexity, and infinite security flaws.
Teenagers know everything... (Score:2)
So we just need to recruit more of them into the roles!
Seriously: we need security firms to start offering college scholarships, work placements (paid internships) etc. The idea that competent CS undergrads get minimum wage jobs over the summer or at college should be an anathema; they need to be employed by companies early and effectively.
Beyond that; those of us who are stock holders need to be writing to the CEO or CIO of companies we're invested in and asking them pointed questions about their policies,