Do Any Companies Power Down at Night?

An anonymous reader writes "My Health Sciences Campus has about 8,000 desktop computers, and on any given night about half of them are left on. I know this because I track all the MAC addresses in case there is a virus outbreak. Aside from the current fad of 'being green', has anyone had any success in encouraging users to power-down at night? You could potentially eliminate running bots, protect yourself from the next virus outbreak, keep your data safe, etc. Do security concerns and power consumption issues matter enough to do this?"

Good idea!

I would go with a reward plan. You could do something like give the top three most energy efficient people a gift certificate to the campus eatery (or whatever really). Calculate how much money is saved (out of everyone participating) and use part of that money to create a pool for the prizes. (It seems like for a large enough group of people, the energy and maintenance costs would reduce considerably, but I wouldn't really know ;) I know I would definitely turn off my work PC every night if I got a free lunch!

Re:Good idea!

The top three "energy efficiency" people are the people taht never show up for work and leave their computers off all the time.

Re:Good idea!

Hmmm, lets see if a small amendment to the metric could circumvent that problem.... no, you're right it's an impossibly difficult flaw to fix.

Hibernate

It's time that all large campuses configured their systems hibernate automatically, if left unused for 30 minutes.

Really, there is no reason NOT to use the power management settings built into the OS.

Re:Hibernate

I did a project in My Campus with more than 8000 desktop computers.
It saved something like 33% power consumption (measured, before and after).
after midnight all desktops that are not in the excluded list hibernate automatically.
I used python + MFC . Was very easy and simple.
It is time for the Sysadmins start to program and make better use from the technology (not just, next-next-finish)..

And I didn't receive any raise besides saving a lot of money to University.
Shameless promotion: Looking for a new job in developed country.

Re:Hibernate

On a Windows XP system, you also want to set the CPU performance in the default power profile to "ADAPTIVE". I'd actually think you'd do well to set the hard drives to spin down and the monitor to turn off after 15 or 20 minutes, set the system to suspend after 30 or 45 minutes, and hibernate after an hour and a half to two hours. You might have to exempt some systems from hibernating - some software and drivers don't always react well to hibernate, and it would be a pain in the (*#)(@ to have to restart after lunch or every meeting. Suspend is a good middle ground. With something more disruptive, a company could well look at that and say "it's not worth the few minutes per day of productivity loss, when factored against the employee's salary + benefits cost." Especially if it leads to calls to your internal helpdesk to try to recover documents in progress or some other work. By the way, productivity vs. conservation is one of the reasons organizations need to be given incentives to conserve power if we want them to do it before energy prices actually exceed cost per hour of labor.

Do you support wake-on-lan?

I'm at a university and many of my colleagues leave their machines on overnight because they sometimes need access to their machine, either to retrieve a file or to run a program. If the IT folks provided everyone with a wake-on-lan script then everyone could turn off their machine. For years this has seemed to me like a no-brainer.

Is there some security or other downside I'm not aware of? Is WOL not reliable?

Re:Do you support wake-on-lan?

WOL takes longer than the timeout period for many services (including, but not limited to WINS, and some windows networking components will take up to three quarters of an hour to recover if a service is initially reported as unavailable while the machine appears to be up).

Another concern is whether your servers are up to handling all the PCs coming on at once in the morning. People leaving Outlook running at all times is actually a Good Thing for IT, cause the alternative of thousands of people hitting the Exchange servers at the same minute would kneel even the biggest distributed servers. Then there's similar concerns for the domain controllers, DHCP servers, proxy servers, or you have it. Leaving a substantial part of the machine park already logged in can save a lot of hardware and configuring.
If shutting everything down, at least a staggered start-up could be prudent.

We have power down at night policy

I work for a large blue chip company and we have a strict policy of powering down at night (including monitor). We regularly audit the records to ensure the machine is powered down and users who are not are requested to always remember. A few users take a few reminders in order to do so and I have heard every excuse under for why they left it on and while some are valid the majority (95%) are not. Our reasons for pushing this policy is purely to save money and reduce unnecessary running time of the equipment. However we are in a position where only laptops users have VPN access so if they need to login to the network from home they already have their laptop with them. If we had open VPN access to desktop users I am sure we would see a lot of users leaving their computer on so that they can RDP into it over VPN.

It took about 6 months before we were at a realistic level. We have 633 desktops on our site so there is normally always a valid reason for one or two to be left on (valid reasons being batch copy, verify or processing of files). For those interested we have had a reduction in the amount of equipment failure (HDD mainly) as well as pretty good cost savings for power. Not to mention running greener (which regardless of if you believe in global warming or not is good).

Re:We have power down at night policy

Sorry I should have clarified. It took 6 months to implement but we have had the policy for a little over 4 years now.

Power vs. operational

I tend to leave the computer on overnight, but with things like monitor power-down and CPU idling enabled. When it's not doing anything it drops about 90% of it's power consumption after 15 minutes, and even when working with the monitor off (eg. running the nightly backup) it's still running at less than 50% of full power. If I power it off, by comparison, it can't run it's virus scan, backup, update check and the like overnight and has to do those things while I'm trying to use it during the day. Plus there's wear and tear to consider, I've noticed that the office computers that get turned off and on every day tend to fail and need replacing several times before mine (that stays on all the time) has a failure.

So my preference is to leave computers running but with power-saving features set to minimize power without shutting things down. This means hard drives continue to spin but the CPU goes into low-power idle mode. The monitor goes to suspend mode (beam and deflection power is off but the circuits and coils are kept warm), not powered-down completely. That seems to be the best balance between reducing power consumption, allowing it to run maintenance operations overnight and minimizing wear and tear and thermal stress on the components. If management absolutely insists on ignoring those last two in favor of the first, wake-on-LAN is essential to allow nightly maintenance to happen.

I used to turn my machine off at night ...

... but now IT has loaded so much crap on it ("desktop agents" [ie apps that spy on me], antivirus, patches, etc) that it is fully 15-20 minutes after turning it on before it is usable. So now I never turn it off. I did the hibernation thing for awhile, but then it stopped working for some reason and I haven't been able to fix it. And if I ask IT to fix it, their solution is always the same for every problem - wipe the machine - a tad inconvenient for me, but pretty efficient for them I suppose. Sigh.

Re:I used to turn my machine off at night ...

Me too!

In any cases, I always leave my computer (a laptop) on during the week. I shut it off on weekends, but due to the software inventory tracker and the required anti-virus scans, I always leave the machine on during the week so that I can actually use it during the day.

The real problem is that the anti-virus scan is so slow that it takes a good three hours. The inventory scan is somewhat better, and only takes about an hour. In both cases, the machine drags to near unusable levels while the scan is running. Given that it's a dual-core machine, this is really a testament to just how screwed up Window's I/O scheduling is - both involve lots of file reads, which apparently causes Windows to drag to a crawl.

Not to mention that hibernate and to a lesser degree suspend appear to not work well with certain drivers on my system. Using hibernate kills the wireless drivers, which isn't a horribly big deal when I can physically plug the system in but it does mean that I just shut the thing off when roaming about, since I'll have to reboot anyway.

But it's that three-hour IT required virus scan that keeps me leaving the machine running nights. That's a real productivity killer during the day. Fortunately it's only scheduled to run once a week.

The inventory app, on the other hand, runs daily for some reason.

Waste not, want not..

I worked for a very large (top-3) pharmaceutical for years. They always asked employees to shut off their computers at night when they went home.

Then one day, they sent out a campus-wide email telling people to leave their computers on all night and over the weekend. They used the CPU cycles to run high-performance scientific computing jobs, saving the cost of buying a supercomputer.

Of course, not every company has a need for spare CPU cycles. This place did a lot of protein-shape searches etc..

Yes, but not strictly enforced

Do security concerns and power consumption issues matter enough to do this?

Yes and no.

When I first got comfortable in my current job, I made a big push toward "greening" our IT resources. As one obvious (erroneously, as I'll explain in a sec) step in this, I convinced most of my users to shut down at night. If we need to push out updates, WOL works just fine for turning machines on a couple hours before the start of the day, and it doesn't impact anyone during working hours.

Then I learned how electric billing actually works for commercial users - Put simply, your company doesn't care if machines stay on all night, because they pay based on their peak load, which will always occur during normal business hours. I had applied ideas that make perfect sense at home, to an environment where they don't apply.

Now, that doesn't mean we should just leave machines on 24/7 - Using electricity has an an environmental aspect in addition to the monetary cost. But if it inconveniences users by more than a few seconds every day, any conservation efforts will actually cost the company money in the long run.


So, I still encourage my users to shut down, and 95% comply. But if they consider it too much of a hassle, I can't financially justify forcing them to spend the first minute of the work day waiting for their machine to boot (not that anyone really works for the first five to ten minutes of the day, between coffee, hitting the bathroom, and just getting the obligatory morning socializing out of the way).

As for the security aspect of this, the servers must run 24/7, and any attacker would target them rather than some random user's desktop. I don't worry about an attacker using a compromised desktop as an intermediate step to the servers, because the desktops have no more privileges on them than anything else inside the firewall (and even then, not much more than a totally untrusted source, except for nonconfidential shared resources that we could restore in a matter of minutes if necessary).

Overnight tasks

I can't help but laugh at those that quote reasons such as 'automatic updates' and 'antivirus scans' as legitimate reasons for leaving a computer on overnight.

With many enterprise management tools, such as Zenworks, it's quite simple to schedule a wake-on-lan task to wake computers up at say, 6am, to perform their daily tasks. It can even be configured to push out an automatic reimage of the machine. Once the updates and scans are done by 7am, people are just beginning to come into the office, yet you've still had a whole 10 hours of downtime. Incidentally, I've not seen a single computer in the past 4 years that doesn't support WoL on the mainboard NIC. Big bucks enterprise manglement apps aren't even required. A simple cron job, and some wakelan/ether-wake/wakeonlan/Net::Wake magic will do it for free. Just gather a list of Mac addresses with ettercap or your friendly ARP table or asset management app/spreadsheet.

May will say that the bandwidth requirements of updates squeezed into the 6am to 7am slot will degrade systems, but that's where a background process such as BITS [wikipedia.org] should be used (as demonstrated by Eve Online, Zenworks, Microsoft and Google). The virus updates are a minor bandwidth requirement if you have suitable leaf services, and the actual scan is only locally intensive.

Being a public sector organisation, we're working towards a greener profile (due to govt policies), and all the tools are there and working. It just needs some effort on the part of the administrators.

Re:Aside from being green...

The answer is simple. Tiny minority of the computers that are on could still be used by someone doing something important. You do not want to cut them off from the network.

Re:Aside from being green...

Amen to that. I doubt anyone really wants to pull an all-nighter, they're there because there's a deadline coming or a problem that must be resolved right now. And whether true or false, the first time anyone uses the excuse "Well, I was ready to pull an all-nighter but from 10pm the network was down" the IT department will have their ass chewed out. PCs inactive -> PCs hibernate is ok, but even then you need a simple way to disable it. On several occasions I've visited copmanies that had boxes which were "don't touch - accessed remotely by VPN" where the user couldn't just unhibernate it in the morning. Plus funny stuff like updates, virus scans and backup (if applicable) that probably runs at some ungodly hour which means you need to wake them first or lose most of the downtime, run those and put them back into sleep. Sure, maybe you could get every PC do to this reliably but I think the administration and scripting of that will cost you quite a bit.

Re:Create job to force automatic reboot or shutdow

Someone might be using it ... there are always a few late users. Trying to determine if a computer is in use in order to shut it down isn't always that simple.

I suggest a simpler, low-tech solution - just stick up visible signs in the labs, and on some of the major office floors, asking people to shut down the computers in the evenings ... it won't be a 100% solution, but most people would probably comply, so for the comparatively little effort put in I bet you can hardly get a better return.

Just the energy savings on that many computers would be not insignificant.

Re:Aside from being green...

I'm against anyone working after 2 p.m., myself.

Re:Common wisdom

Common wisdom (which may or may not be actual wisdom) suggests that powering up/down of computer power supplies is one of the largest sources of "wear" on computers nowadays, and so it's best to avoid that

Nope. That would only be true if all of the three are true:

1: Power-cycling actually reduces the MTBF opposed to just leaving it on.

2: The reduced MTBF is lower than your company intends to keep the asset.

3: Cost-savings from the "increased" MTBF by leaving it on is greater than the electricity (+ increased A/C cost) cost to run those 300W power supplies all the time.

Of the ~6 computers I've had to failure, they all lasted far longer than even a five-year technology plan, AND did not fail due to simple wear and tear on the circuits. My anecdote isn't data, but it does make me question your conventional wisdom. (Especially since those PCs I know that are left on all the time don't have a significantly increased lifespan.)

Re:Common wisdom

False certainty characterises a bit too much of dialogue on the internet, I think, and makes it hard to be careful and humble in discussion. If I felt more sure of what I were saying, I'd have phrased it differently :) Presumably :P

Re:We power down at weekends

What's the cost of 10k * 261 days * 12 hours of power?

Surely you could use wake on lan to wake the machines then do your rollout 10 minutes later? Or do a patch install when the machine is turned on and connects to the domain controller?

In windows I'm sure you can set the time between warning appearing and shutdown ocuring. Give 600 seconds warning and you could probably shutdown 90% of the machines overnight.

Re:We power down at weekends

In windows I'm sure you can set the time between warning appearing and shutdown ocuring. Give 600 seconds warning and you could probably shutdown 90% of the machines overnight.

You're assuming that 100% of machines in use are doing something interactive (and therefore have someone sat at them). This is frequently not the case.

Re:We power down at weekends

14 hours a day times 52 times 5 times average power-consumption for an idling desktop times 10K employees works out to:

14*52*5*0.15*10000 = 5.460.000 Kwh of electric power, WORSE in the time when you use AC, because you'll need additional AC to get rid of that extra heat.

With average power-prices of 10 cent, you'll spend more than half a million dollar just paying for the electricity, in practice with AC and all you'll probably pay a million.

So, you saved $80K and wasted a million. Way to go !