Ransomware Attack on Major Payroll System Kronos May Take 'Weeks' to Repair (kronos.com) 76
Earlier this week long-time Slashdot reader DJAdapt wrote:
According to a post on the Kronos Community Page, a cyber security incident due to a ransomware attack is affecting UKG Workforce Central, UKGTeleStaff, Healthcare Extensions, and Banking Scheduling. Although they are currently working with cyber security experts on the issue, they say that it may take several weeks to restore full system availability.
CNN reported: Ultimate Kronos Group, one of the largest human resources companies, disclosed a crippling ransomware attack on Monday [December 13th], impacting payroll systems for a number of workers. After noticing "unusual activity" on Saturday [December 11th], Kronos noted that its systems were down and could remain that way for several weeks.
Kronos has a long list of notable customers across the public and private sector, including the city of Cleveland, New York's Metropolitan Transportation Authority (MTA), Tesla and MGM Resorts International. It also works with many hospitals across the country. Some employers find themselves having to make contingency plans in order to pay workers, such as shifting to paper checks. And some impacted employees have been unable to access payroll systems...
In addition to the potential payroll issues, there's also data privacy concerns. The city of Cleveland said in a statement Monday that Kronos alerted it that sensitive information may have been compromised in the attack. Employee names, addresses and the last four digits of social security numbers may have been stolen by the hackers inside Kronos's network.
Other Kronos customers include Whole Foods, GameStop and Honda, as well as state and local government agencies like the state of West Virginia, reports NBC News: John Riggi, the senior advisor for cybersecurity at the American Hospital Association, an industry group, said that he had spoken with multiple hospitals that have had to create contingency plans for getting employees paid, managing their schedules and tracking their hours. "Quite frankly, this could not have happened at a worse time. We've had a surge in Covid patients, flu patients," Riggi said. "It's a distraction to hospital administrators at a time when they don't need any additional burden or diversion of resources."
"Though it has not been confirmed, there is speculation that the notorious Log4Shell vulnerability was involved," writes CPO magazine, "given that the Kronos cloud services are known to be built on Java to a great degree...."
"Microsoft's security team has reported that ransomware attacks are already unfolding after these breaches in at least several cases."
CNN reported: Ultimate Kronos Group, one of the largest human resources companies, disclosed a crippling ransomware attack on Monday [December 13th], impacting payroll systems for a number of workers. After noticing "unusual activity" on Saturday [December 11th], Kronos noted that its systems were down and could remain that way for several weeks.
Kronos has a long list of notable customers across the public and private sector, including the city of Cleveland, New York's Metropolitan Transportation Authority (MTA), Tesla and MGM Resorts International. It also works with many hospitals across the country. Some employers find themselves having to make contingency plans in order to pay workers, such as shifting to paper checks. And some impacted employees have been unable to access payroll systems...
In addition to the potential payroll issues, there's also data privacy concerns. The city of Cleveland said in a statement Monday that Kronos alerted it that sensitive information may have been compromised in the attack. Employee names, addresses and the last four digits of social security numbers may have been stolen by the hackers inside Kronos's network.
Other Kronos customers include Whole Foods, GameStop and Honda, as well as state and local government agencies like the state of West Virginia, reports NBC News: John Riggi, the senior advisor for cybersecurity at the American Hospital Association, an industry group, said that he had spoken with multiple hospitals that have had to create contingency plans for getting employees paid, managing their schedules and tracking their hours. "Quite frankly, this could not have happened at a worse time. We've had a surge in Covid patients, flu patients," Riggi said. "It's a distraction to hospital administrators at a time when they don't need any additional burden or diversion of resources."
"Though it has not been confirmed, there is speculation that the notorious Log4Shell vulnerability was involved," writes CPO magazine, "given that the Kronos cloud services are known to be built on Java to a great degree...."
"Microsoft's security team has reported that ransomware attacks are already unfolding after these breaches in at least several cases."
This affects my wife's employer (Score:4, Interesting)
The hospital my wife works for has already told people they're just planning to pay everyone the same amount they received the previous pay period. Which is fine, for salaried positions like my wife's... but a lot of nurses work per diem, and their hours per pay period can vary greatly.
I'll be curious to see if they pull it off (this upcoming Friday is the next pay day). I mean, they offloaded this to Kronos quite a while ago - how quickly can the hospital get back to writing / printing paychecks themselves? We're not living paycheck to paycheck anymore, fortunately - but not everyone has a pool of money in the bank they can fall back on.
Something really needs to be done about these ransomware jerks.
Re: (Score:3, Informative)
Something really needs to be done about these ransomware jerks.
indeed... building IT systems defensively so that those jerks don't fsck the company by taking out a single machine...
the Windows achitecture is so easy to fsck around with it's not even funny
Re: (Score:1, Interesting)
The log4j vuln that let the attackers in isn't exactly "Windows architecture".
Re: (Score:2)
Re: This affects my wife's employer (Score:2)
It's likely easier to confirm their web services are not running on windows servers...post-mortem will likely include that info but who knows how long that will take.
Re: (Score:1)
The log4j vuln that let the attackers in isn't exactly "Windows architecture".
The first place it was found was on Minecraft running on Windows.
https://www.bleepingcomputer.c... [bleepingcomputer.com]
Yea, I was surprised too.
Re:This affects my wife's employer (Score:5, Insightful)
It is certainly good to harden the computer systems, but it would help is law enforcement in general was less clueless and powerless to actually deal with the criminals behind the attacks.
As an inevitable analogy, I'll bet your house doesn't have half inch iron plate siding and titanium bars on the windows. I'll further guess that you might resent it if someone cleans you out one day while you're out and all anyone does is wag their finger at you about how you should have gone with iron plate siding and titanium bars.
It's worth noting that even if you leave your front door wide open all weekend, it is still a felony if someone goes inside and steals your stuff. You still have every right to expect police to go after the crooks.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
You don't have to pick one or the other (and can't (Score:2)
You don't have to choose EITHER defense or law enforcement. You can have both. In fact it's IMPOSSIBLE to do one without the other.
Consider if banks stored the money on the sidewalk rather than in the vault. Lots and lots of people walking by would pick some up. If nobody has any defenses, there will obviously be far too much crime for the police and courts to handle.
Can we have defensive measures without policing? Defensive measures attempt to reduce the criminal's probability of success. A rational atta
Re: This affects my wife's employer (Score:2)
Iâ(TM)d be even more fecked off if they did it remotely from Russia, and the FBI still did nothing about them!
Re: (Score:3, Insightful)
payrolls is a contractual obligation. The ransomware folks are criminals and should absolutely be found and incapacitated so they don't harm society again. 30 prison terms with not computer access for them should do nicely.
However the "jerks" here are Kronos and any of the clients that can't handle payroll without them. Payroll is a contractual obligation. You should not get to just say 'woopsie something happened to our systems' This is an actual example where the employee, employer relationship should h
Re: (Score:2, Insightful)
Something really needs to be done about these ransomware jerks.
"Something needs to be done", to be sure, but what?
In the real world it usually starts with protecting the valuables. Closing the door, locking the paychest. It's nice low hanging fruit that you can do yourself, rather than relying on government to fix it for you.
These are measures you would have to take even with a handy strong government available for coming down like a ton of bricks on "these ransomware jerks". Or handy bands of vigilantes to do the government's job more sloppily, for a price. For the
Re: (Score:2)
Something really needs to be done about these ransomware jerks.
There will always be more of them unless we some day build a star trek-esque society, which I don't expect humanity to actually reach. So the thing that needs to be done is to improve security. There is literally nothing else meaningful than can be done, or at least, that can and will be done.
Re: (Score:2)
There will always be more of them unless we some day build a star trek-esque society, which I don't expect humanity to actually reach.
Yeah... I had to get forty for that dream to finally breathe its last.
Re: (Score:2)
Yeah... I had to get forty for that dream to finally breathe its last.
Winks? Lashes? Yards?
Re: (Score:2)
He's actually referring to the old U2 song - which makes even less sense.
Re: This affects my wife's employer (Score:2)
Yeah, referring to a U2 song generally doesn't unless you want to talk about what a douche Bono is
Re: (Score:2)
Something really needs to be done about these ransomware jerks.
There will always be more of them unless we some day build a star trek-esque society, which I don't expect humanity to actually reach. So the thing that needs to be done is to improve security. There is literally nothing else meaningful than can be done, or at least, that can and will be done.
I'd like for people to convince me that ransomware and simple screwups are not an inadvertent feature of the cloud. There is enough complexity that screwups that would normally affect one business can affect the entire world - like the Azure expired SSL fiasco.
Mistakes simply will be made.
Vulnerabilities simply willbe found, and will be exploited.
But one of the core values of the cloud is taking as many groups as possible, with the concept that these groups will be saving money, and all will be well. Th
Re: (Score:2)
What does this problem have to do with 'the cloud'?
Re: (Score:2)
What does this problem have to do with 'the cloud'?
Are you replying to me? If so - I have noted exactly what happens when comapanies use the cloud - that cloud being an irresistible source of mayhem, ransomware, customer personal information given to the bad guys. It makes their job easier - all in one shopping for them.
What of what I wrote do you claim is not true, or has nothing to do with the cloud sources that are hit by ransomware personal information grabbed off the cloud storage. and just general screwups like expired SSL certificates bringing the
Re: (Score:1)
I am asking you what THIS ATTACK on THIS COMPANY has to do with 'the cloud'. I see no indication anywhere that this attack had anything at all to do with 'the cloud'.
Re: (Score:2)
I am asking you what THIS ATTACK on THIS COMPANY has to do with 'the cloud'. I see no indication anywhere that this attack had anything at all to do with 'the cloud'.
https://www.cnn.com/2021/12/16... [cnn.com] https://cloudcomputing-news.ne... [cloudcomputing-news.net] https://www.techrepublic.com/a... [techrepublic.com]
I'll spell it out slowly for you, since you have your Caplocks loaded and ready to rumble.
Kronos Private Cloud solutions, a data storing entity for several of the company's services, including UKG Workforce Central, which is used by employees to track hours and schedule shifts.
Note that the name is Kronos Private Cloud Solutions -the cloud. A company using the cloud was hit by ransomware. That co
Re: (Score:2)
One class of solution would be to make that access contingent on cooperation. If gangs in your country pu
Re: (Score:2)
Something really needs to be done about these ransomware jerks.
Sure, but it starts with having companies actually upgrade servers and improve security. Nothing like having 90% of thousands of servers be end of life to improve the odds of a successful attack.
[John]
Re: (Score:2)
Something really needs to be done about these ransomware jerks.
Sure, but it starts with having companies actually upgrade servers and improve security. Nothing like having 90% of thousands of servers be end of life to improve the odds of a successful attack.
[John]
That will help some, but there's two things. Thing one is there is a constant stream of security updates, so the bad guys are still finding vulnerabilities.
Thing two, in a world that the bottom line is "as cheap as possible", we're going to have as cheap as possible. P
This is just the future of cloud computing. Like it so far?
Re: This affects my wife's employer (Score:2)
Re: (Score:1)
Re: (Score:2)
And you wonder why I say that Medicare for all is a bad idea, it'll only give the CEOs of hospitals like the one your wife works at a blank cheque handed to charge the government whatever the hell it wants to charge. No, we need to reign in the charges that hospitals charge for healthcare and magically the cost of healthcare will go down. Common sense.
Please explain how you "reign in the charges that hospitals charge for healthcare" without the government getting involved in it. Millions of Americans are waiting for this important information that was promised by the previous administration for years.
Re: This affects my wife's employer (Score:2)
Re: (Score:3)
The problem with common sense is it tends more closely align with magical thinking, it is whatever the speaker believes.
Re: (Score:2)
Will be interesting to see how they handle the tax reporting they are supposed to provide y'all by Jan 30....
Re: (Score:2)
Which is fine, for salaried positions like my wife's
The problem is larger than you think. In a lot of places in Europe (where Kronos is also huge) December often is an odd month in terms of salary. I don't know of anyone who gets their normal salary this month. Tax corrections, yearly deductions, reimbursements, all happen in December, and that's before you take into account the many companies and countries of Europe who have 13 or 14 months in the year.
Here's what my November paycheck looked like:
Salary for the 11th month of the year.
Here's what my December
Re: This affects my wife's employer (Score:1)
Re: (Score:2)
No, actually in my case it's literally 14 payment cycles in the year dividing my salary. Month 13 is paid in June (before the summer vacation), and month 14 in December (before Christmas expenses). It kind of adjusts people's payment cycles to ensure they get paid extra in the two months where they incur by far the largest expenses for the year.
Re: (Score:2)
My yearly salary is divided into 14. Month 13 is paid in June, Month 14 in December. It's common practice in some European countries and sort of a "forced savings" measure which has the effect of ensuring people have enough money to prevent them falling into a debt spiral in the two periods where that happens most commonly: Christmas Shopping, and Summer vacation.
I thought it was a horrible idea at first, but honestly I actually like the idea of not having to think about if I can afford a holiday or present
Re: (Score:2)
Yep, my GF's hospital is the same way. Unfortunately she worked a lot of overtime in the previous pay period so she is going to be artificially inflated paychecks for a little bit that she will (presumably) have to pay back.
Re: (Score:1)
My employer uses Kronos as well. So far no word from them. Can't say I'm surprised, they're pretty clueless.
Re: (Score:2)
It seems that one of the issues that ransomware takes
advantage of is that many systems use the same machine
to interface with the internet as the machine that handles
data-processing tasks.
That way a corruption of the internet machine can lead
to locked-up data files, when those files should be on a
separate system that only communicates with the internet
machine by a strict protocol. This protocol should filter out
anything that could harm the data, including executable code.
Weeks to repair? (Score:4, Interesting)
Why Is it taking weeks to restore from backup?
Re:Weeks to repair? (Score:5, Informative)
I work in the industry (IT backend compute / storage / hybrid-cloud / damn-near-everything equipment for damn near everyone) It is astounding how many worldwide companies have no backups. Or, sure, all 100PB is backed up, but the slow data link .. lets see .. 3 years to restore...yup, about 3 years...
Re: (Score:3)
I vaguely recall an article from a few years ago that one of the companies, i believe it was Amazon, will deliver your backup on a "storage box", for a price.
Re:Weeks to repair? (Score:5, Informative)
That still leaves the minor matter of making sure the right data goes on the right machine. Too often, ther is no rhyme or reason to how the backups are organized. Everything HAS a backup, but those backups are just dumped in a big bag with a promissory note to come up with a restore plan one day.
It's better than not having a backup, but restoring can take a while.
Also, you have to make sure that your restore doesn't also restore the sleeping exploit that will lock everything up again next week. One thing is certain, your restored state will be vulnerable and you must address that or you'll be right back to square one in short order.
Re: (Score:2)
I vaguely recall an article from a few years ago that one of the companies, i believe it was Amazon, will deliver your backup on a "storage box", for a price.
I believe Backblaze will do this as well.
Re: Weeks to repair? (Score:2)
This is why services need external audits and ratings. If you provide payroll for critical industries, the services themselves should prove they are robust.
Regulations help a bit here but more than anything it opens up a good service sector for well trained cyber experts.
Not just a matter of restoring backups (Score:4, Informative)
I work in the industry (IT backend compute / storage / hybrid-cloud / damn-near-everything equipment for damn near everyone) It is astounding how many worldwide companies have no backups. Or, sure, all 100PB is backed up, but the slow data link .. lets see .. 3 years to restore...yup, about 3 years...
It's not simply a matter of restoring from backup. The exploited vulnerability must be positively identified, and then remediated. It may involve extensive code auditing and rewriting. This can certainly be a time-consuming task for a large and complex code base.
If the vulnerability is not remediated, you could restore from backup only to get hit again days later.
Re:Not just a matter of restoring backups (Score:4, Insightful)
It's not simply a matter of restoring from backup. The exploited vulnerability must be positively identified, and then remediated. It may involve extensive code auditing and rewriting. This can certainly be a time-consuming task for a large and complex code base.
If the vulnerability is not remediated, you could restore from backup only to get hit again days later.
Which is why your disaster recovery plan should probably be more than:
1) Restore from backups
2) ???
3) Resume Profit!
Restoring from backups does not or at least probably should not mean directly returning to 'situation normal' maybe you don't know exactly how such and and such got in and did this and that, so perhaps you bring up certain systems with only alternative communications paths enabled and have a plan to communicate to existing clients how to use them, etc.
Its like all these 'cloud native guys' I hear say things like, well we have database snapshots in a storage blob, and we can just have terraform redeploy everything if needed - backup done. Well no not really, don't get me wrong in many ways its absolutely great you can just press the play button and all the 'infrastructure as code' and everything can get rebuilt on the fly almost effortlessly and that afterward you are handful of SQL commands away from restoring the data snapshots and everything being as if nothing happened; but you probably need to think about it a little harder than that.
You don't actually want to go to 'its like nothing happened' at least not directly. You want to go to some very conservative, essential operations only bastion state until you have identified and addressed the root cause.
Re: (Score:2)
What are you restoring from backup? The OS? The applications? Your data?
My contingency plan in case of a ransomware attack is to just restore my data. The OS and applications come directly from the distributors. Not my backups. My data isn't executable nor is any information that resides on the backup media.
That only leaves issues with internally developed software. Are their controls in place to mitigate arbitrary code or executables from being inserted. I feel there are few companies who have actual plans
Re: (Score:2)
It's not simply a matter of restoring from backup. The exploited vulnerability must be positively identified, and then remediated. It may involve extensive code auditing and rewriting. This can certainly be a time-consuming task for a large and complex code base.
If the vulnerability is not remediated, you could restore from backup only to get hit again days later.
Which is why your disaster recovery plan should probably be more than:
1) Restore from backups
2) ???
3) Resume Profit!
#2 is "immediately get pwned again".
It's the kind of arsehole thing I'd do. Have a delay built into whatever I used to compromise the system to ensure that even if they could quickly restore from backups, they'd be back to being my bitch in no time.
This is why prevention is better than cure... because if I can think like that I guarantee you someone with fewer morals and more time on their hands can as well.
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Because the work-hours for fixing it are entered in Kronos, i.e. ultimately thanks to the bug, nobody'll get paid.
Re: (Score:2)
> Why Is it taking weeks to restore from backup?
They're gonna get sued hard for negligence, probably.
Re: (Score:2)
RAID != backup != disaster recovery.
There is a reason we differentiate between restore point objective and restore time objective.
Re:Weeks to repair? (Score:4, Interesting)
Why Is it taking weeks to restore from backup?
You're not just restoring from backup.
You're restoring from backup. You're testing the backup restored correctly. You're identifying the activities which occurred since last backup and correcting for them. You're then incorporating into digital form all the shit you're doing right now whilst enacting your business continuity strategy). The actual moving of data is only a small part of the problem when a system goes down.
I'm reminded of when Randsomware took down Europe's biggest port of some of the stories. The actual outage only went for a few days.
- DHL for example stopped processing packages, they moved everything into a giant warehouse. It took weeks to get through the mail backlog when they got up and running.
- Maersk who caused the issue came to a global halt including a massive supply chain fuckup as they went. They had close to a month before they stopped paying demurrage costs for ships arriving at the wrong time.
- The Port of Rotterdam enacted a very competent business continuation strategy, they dropped their toys and grabbed pencil and paper. They employed a small army of contractors which then also took weeks to get what they were doing on paper synchronised with their electronic backups (which also were recovered in a matter of days).
- We weren't even involved but for several weeks we were dealing with freight issues, customs receipts not coming, paperwork not agreeing with records because someone couldn't read the hand written text, customers doing orders at weird times were were unprepared for, etc.
Re: (Score:2)
You forgot one very critical item: You need to find and fix the attack vector before putting the restored machines online again. Otherwise you will get attacked again in short order.
Re: (Score:2)
Yes! This deserves to be modded up.
Re: (Score:2)
Why Is it taking weeks to restore from backup?
Simple: Restoring is not enough. If you do that, you will get attacked and compromised again very fast. You need to find out first what the vector was and then fix it before putting servers online again. Still, "weeks" strikes me as too long. They probably never did proper BCM tests and may need to figure out how to do all this first.
I'm just wondering (Score:4, Interesting)
When an attack takes weeks to recover from, what king of security was in place? Was there any thought given to recovery? I would assume that a company managing this much sensitive information would limit the access to sensitive data. Is this a case that it can't happen to me and security and data integrity is just a cost on the income statement? Is it another case of check box security where going over a checklist is all the verification on security done?
There is a lot of talk about the breaches but very little information to help others. I can't imagine with all these breaches that security is still an after thought.
Re: (Score:2)
There can be lots of security in place, none of which addresses the attack vector that was used. Backup & recovery plans can be a real hassle, and management is generally reluctant to admit that the dangers are real. It costs money NOW to deal with something that "probably won't happen".
Re: (Score:2)
When an attack takes weeks to recover from, what king of security was in place?
The recovery may have nothing to do with data, and everything to do with synchronising and maintaining an ongoing business.
The world doesn't stop and twiddle its thumbs waiting for you, and even if it did those employees were on the clock and needed to use a timekeeping system ;-)
Covering data is a small part of recovering from a business upset.
Re: (Score:2)
You would have thought that the Klingon homeworld would have great security, but I guess if you can't just hit it with a bat'leth maybe they aren't interested.
Or is this a different Kronos?
I posted this last Tuesday (Score:2)
I thought I finally has a story worth posting lol
https://slashdot.org/submissio... [slashdot.org]
Re: (Score:3)
Re: (Score:1)
Re: (Score:2)
We did similar last week with a 2nd check issued for those that worked overtime
Payroll problems ALWAYS need weeks to solve (Score:2)
Honi soit qui mal y pense.