Nine Traits of the Veteran Network Admin 142
snydeq writes "Born or made, network admins share certain defining characteristics. Deep End's Paul Venezia offers nine: 'I hope that this insight into the extremely logical, yet consistently dangerous world of the network admin has shed some light on how we work and how we think. I don't expect it to curtail the repeated claims of the network being down, but maybe it's a start. In fact, if you're reading this and you are not a network admin, perhaps you should find the closest one and buy him or her a cup of coffee. They could probably use it.'"
Will read later (Score:5, Funny)
The network is down.
Re: Will read later (Score:1)
#10: Scragly neck beard that makes users as uncomfortable as his "you just don't understand" rants.
Re:Will read later (Score:5, Funny)
Re: (Score:1)
Re: (Score:2)
For some reason I read TFA, and that's number 4
"Veteran network admin trait No. 4: Believe it or not, we've tried turning it off and back on again"
Re: (Score:2)
Re: (Score:2)
Re: (Score:1)
Your admin blocks Infoworld articles? Thank him sometime for looking out for you.
Re: (Score:2)
No, because a 403 is a response from the webserver. A proxy would probably give a different message, and if it was blocked via firewall rules then it just would give no response whatsoever..
Re:Will read later (Score:4, Informative)
A proxy delivers whatever code the admin wants it to deliver. Squid returns 403 on ACL blocks, but it's trivial to alter.
Re: (Score:3)
Your admin blocks Infoworld articles? Thank him sometime for looking out for you.
Thanks Mordac!
Re: Will read later (Score:5, Insightful)
These "traits of" or "habits of" articles are the modern equivalent of horoscopes. Lots of feel good stuff, but not much actual advice. I can agree and disagree with every point to some extent.
Nice article that generates a lot of page views. For real advice get 1 or more O'Reilly books, or better yet, find a competent admin and become his/her apprentice.
Re: (Score:1)
Also, coffee. Network admins don't always drink coffee.
This guy's wrong and right on all of his list, because of why he's right or wrong. IT is an ever-changing field, and there is no "always", or "never".
This is, of course, my list of 1. I'm also wrong and right.
Re: (Score:2, Funny)
Call to Admin: "Network is slower today than usual". Network Admin to caller: The weather is hot today, bits expand and it's harder for them to squeeze through thin wires.
Re: (Score:2)
By the time they've finished doing that, you'll probably have worked out what was causing the slowdown, and either fixed it, or allowed it to fix it's self.
The spice of life (Score:5, Funny)
For many reasons there are few things add excitement to life like working with someone who habitually answers the phone with, "I didn't do it."
Obligatory xkcd (Score:5, Funny)
Re: (Score:2, Funny)
We're talking about Networks. Systems can do NOTHING without my massaging of the thinwire, caressing of the switch configuration. Systems are USELESS, DUMB without us. Systems is the heart of the network? Well, Networks is the vascular system. What is the heart without the blood vessels? Just another lump of useless m
Re: (Score:2)
The REAL Sysadmin is the Network/Server/Software/Hardware person. A long time ago in a Galaxy far, far away.... You did everything and anything involved with IT. You even had to manage PBX and telco equipment, not just the routers and switches. You took care of the tape backups and user printers. You had to write code and compile kernels. And you had to know about electronics to repair outdated equipment critical to the workings of the company.
Re: (Score:2)
Re: (Score:1)
Only in companies that are ~ 100 people
Re: (Score:3)
Re: (Score:2)
I'll save you. I'll save you all.
...until your workstation goes splat. Then you'll be over at the help desk, begging them to get your laptop back up. Therein lies the problem with over-specialization. ;)
(...meanwhile, a real sysadmin will have will have told you which router you screwed up the configs on, and would you please either fix the damned thing, or at least cough up the supe password so that he can do it and get on with the day.)
Re: (Score:1)
no.
CS dropouts that become sysadmins are reformed sysadmins.
And they usually have a dozen CS guys under them.
The true neck beards never went to college. No need to be institutionalized.
Re: (Score:2)
Developers play with code.
Sysadmins take the garbage they churn out and wrap it in 10 layers of scripts and filters to actually make it work.
I RTFA (Score:5, Insightful)
Why do I feel like I just watched somebody jerk off?
Re:I RTFA (Score:5, Insightful)
Because you can't handle the sheer awesomeness that is the veteran network admin!
Re: (Score:2)
To me, it was like watching someone jerk of while insulting everyone around as idiot.
(Really, the first 5 bullet points were nothing but "all mere mortal users are just stupid" ranting)
Re: (Score:2)
To me, it was like watching someone jerk of while insulting everyone around as idiot.
That's my experience of most network admins in big companies. They mostly come across as a bit OpenBSD-mailing-list if you see what I mean. They are defensive as hell, maybe their jobs make them that way.
I got in some nasty arguments with them over innocent jokes that Unix or Windows admins would have just laughed off.
Re:I RTFA (Score:5, Insightful)
Imagine running your company's entire IT infrastructure off of a random assortment of Android cell phones... That's basically the job of a Network Administrator.
Despite running the most critical piece of infrastructure, they can't peer into the base system at all, to see how it's going to behave in edge cases. The monitoring and management tools available are shockingly poor, and offer little utility.
They're forced to use a random mix of different network hardware, which all behaves differently, and each has a list of errata 100 pages long. Updates are huge monolithic firmware blobs, guaranteed to have bugs, leading you to choose between dealing with the awful but barely manageable bugs you have now, or updating and dealing with a whole new set of bugs, and potentially some so bad they'll take the system down, and there's relatively little you can do to test this, even if you have a rather large testing network (which most do not).
Software Defined Networking and OpenFlow look like they might finally start changing this, and Network Admins will be able to understand how these complex systems work, and even define the behavior they want. But in the mean time, your Network Admin remains the Prime Minister of the insane asylum.
Re: (Score:2)
Thanks for explaining.
Re: (Score:3)
If you think 'software defined networking' and OpenFlow are going to make it more understandable you completely fail to understand how any software works at all.
Networks are ALREADY SOFTWARE DEFINED and ALWAYS HAVE BEEN.
You seem to think running software on your software that runs on your hardware is magically and unintuitively going to work better than software on hardware.
Re: (Score:2)
Re: (Score:2)
Don't bitch at me just because you don't know what a common term actually MEANS.
There's no magic about it. Having lower-level, centralized control of network equipment can be a huge improvement over the current mess. A fringe benefit is that works the same from vendor to vendor, across differe
Re: (Score:2)
They are defensive as hell, maybe their jobs make them that way.
I got in some nasty arguments with them over innocent jokes that Unix or Windows admins would have just laughed off.
Defensive is a good description. After all, there are only so many commands in IOS, and -- oh, look, the Cisco Nexus switches use Linux now! :)
Re: (Score:2)
autocomplete is yoru friend on the command line.
Re: (Score:2)
Re: (Score:2)
can you hear my packets now?
ping -a linux.org
Re: (Score:2)
I'm not sure he even understands the definition of "trait"
Re: (Score:2)
Because there's a mirror on the ceiling?
This all looks familiar... (Score:1)
This is like a LinkedIn article. 8 ways to do this, 5 systems to improve that, 3 things to avoid when doing the next thing and 9 traits of the successful Whatshisname.
It's maddening.
Trait #10 is certainly not (Score:2)
Re:Trait #10 is certainly not (Score:5, Funny)
Good administrators keep humility under control. Too much of it corrodes the equipment. There are sensors for it you know.
This is great! (Score:5, Funny)
I just bought an MCSE or MSCE or whatever you call it and I was wondering how the fuck I could fool people into believing I actually knew what the fuck I was doing.
Problem solved!
Thanks a lot, InfoWorld!
Captcha: "secured"
Re: (Score:1)
Oh come on, that can't be so hard to remember.
It's Minesweeper Consultant & Solitaire Expert - MCSE. MSCE makes no sense at all.
Re: (Score:2)
It's Minesweeper Consultant & Solitaire Expert - MCSE.
Or Must Call Somebody Else, for the "all certs, no clue" variety of admin.
Re: (Score:2)
Re: (Score:2)
I had the CCNA. Got my my first job out of uni.
On Helldesk.
Been there ever since.
Re: (Score:2)
The CCNA and NP are meaningless to anyone with a clue. Sorry you wasted your money but you need to know what they teach you before you even considering getting into the field, you need far more authoritative certifications to be useful job hunting. A CCNA or CCNP in network is roughly like saying 'I got a certificate because I could spell my own name ... given at least 3 tries.'
The reason you are still working at a help desk is simple. You aren't qualified to do anything else. Hate to burst your bubble,
Re: (Score:2)
I also have a diploma. It's worth more, but not a huge amount more. Basically, my career is in ruins, and I'm doomed to spend the rest of my working life resetting forgotten passwords.
Re: (Score:2)
There are. They all start with 'X years experience at.' Employers can afford to be fussy right now - lots of potential employees, not many jobs.
The Nine Traits (Score:1)
(as mentioned in the article)
1: We already know it's down
2: If we don't know it's down, it's probably not down
3: We will ping and test several times before digging into the problem
4: Believe it or not, we've tried turning it off and back on again
5: During an outage, we're not just staring at the screen -- we're following a path in our heads
6: We calculate subnet masks and CIDR as easily as breathing
7: We do not tolerate bugs; they are of the devil
8: We can read live packet streams and write highly complex f
Re: (Score:2)
A shortsighted and arbitrary kernel limitation does not really count as the fault of the downstream devs who get to suffer for it. As you point out, your "solution" actively encouraged your devs to make a completely wasteful userspace cache of something the kernel should just provide transparently; since you already know this so
Re: (Score:2)
They hated it because it laid the blame squarely where it belonged, at the foot of a very difficult problem to resolve, unless you made automatic local replicas of LDAP subtrees.
A shortsighted and arbitrary kernel limitation does not really count as the fault of the downstream devs who get to suffer for it. As you point out, your "solution" actively encouraged your devs to make a completely wasteful userspace cache of something the kernel should just provide transparently; since you already know this solution, and apparently had access to the kernel code - Why didn''t you either fix the limitation, or if not possible, just build the damned cache yourself and serve high-latency requests from it, rather than getting into a blame-game with people who (reasonably) just expect LDAP to work as advertised?
Because, contrary to popular belief, there is not an infinite amount of available wired memory in the kernel for doing negative caching, and even in the positive caching case, the memory goes quickly. Also, contrary to popular believe, putting everything in the kernel doesn't make it magically faster. Also, POSIX mandates a finite number of groups, and those groups must be attached to each credential, and all internal operations pass around credential references inherited from the thread for the current c
Re: (Score:2)
2: If we don't know it's down, it's probably not down
Something is wrong. Maybe a switch under your control, not mine, is partitioning my part of the network, or maybe the new security you put on MAC address per port and didn't tell us about, or told us about but not in a meaningful way, has made it so we can no longer plug in both our laptops to the same switch port. How about something on the dashboard to let us know how many unresolved issues there outstanding at any given time, or over a period of time as a graph. If there is a spike on the graph, there's something you own that'd not sufficiently monitored, and it's still your problem.
9 times out of 10, it's an actual user configuration issue. And when the network guy points out that you screwed up, not him, after you blamed him, you're going to get the attitude you deserve.
Re: (Score:2)
> 1: We already know it's down
> Then put it on the freaking dashboard for the network service outages so we know not to bother you. If you really don't want to be bothered, put up an "Estimated time to resolution, given past history"/"Estimated time to resolution for [characterized as problem X]", or you're going to get the "we know you know it's down; when will the f---ing thing be back up?" calls anyway.
THIS. 100% !!!
_Communication_ from Sysadmins / DevOps to the rest of the team is PARAMOUNT.
It hel
Clarifying #4 (Score:5, Insightful)
Notice he said "turning an interface on and off", not "rebooting".
Nothing says "I'm a noon and came from a Windows world" like rebooting a switch or router to fix a problem.
Logs on those devices are in memory. Rebooting clears the logs and you then can't troubleshoot. If you can't troubleshoot, you'll never know what really happened. If you don't know the root cause of the failure you can't prevent it from happening again.
Re:Clarifying #4 (Score:5, Insightful)
Logs on those devices are in memory
Although they should be on the NMS and syslog servers too... and many Juniper M/T series devices, which are very frequently used by large carriers for core and edge routers, have a hard drive mounted on the /var filesystem,
that the routers' syslogs get written to.
Rebooting sounds like an act of desperation. It's almost never the cleanest way, and it almost always incurs additional downtime, causes more outages, or further lengthens the network downtime --- since you require 3 to 5 minutes for a reboot, then maybe 3 to 5 more minutes for routing protocols to reconverge.
And god help your soul, if you reboot a Cisco device such as a 72xx or Catalyst 65xx running BGP with 3 or 4 copies of a full table, and with 8 or 9 total peers.
That one weak moment, when reboot was chosen may result in 60 to 90 minutes of trying to coax the network back up gently.
Re: (Score:2)
Or that 6509 with a sup720 hosting 25 T3s for some 2000 remote sites using OSPF as your routing protocol. (All in Area 0 of course.)
Re: (Score:2)
Nothing says "I'm a noon and came from a Windows world" like rebooting a switch or router to fix a problem.
Sometimes that IS the answer: Boot up triggers POST, where it may plainly say "your flash is hosed". Or perhaps a voltage spike threw things into a "bad place", and the solution is to do a full power-cycle to bring everything back to where it was. Perhaps the issue only occurs on boot up, and you need to turn it off and on to see the problem happen (such as when someone left the router in 0x2142 mode).
Rebooting is usually not the answer but sometimes it is. You can stubbornly insist that the answer is i
Re: (Score:2)
Sometimes that IS the answer
A lot of times, it's the only way to clear up an ill-considered typo + "write mem".
Re: (Score:2)
You mean "copy running startup". "write mem" has been deprecated since before I started working on Cisco 2600s back in 2004 ;)
Re:Clarifying #4 (Score:5, Insightful)
If you don't have a separate syslogd system, you're not doing your job well.
Re: (Score:1)
Re: (Score:2)
You're fired.
Logs on those devices should be going to a syslog server, the fact that they aren't means your ass is fired immediately.
The BOFH knows the real traits of seasoned admins (Score:5, Funny)
1. A complete disdain or hatred of lusers.
2. A collection of blackmail materials.
3. Homicidal rage.
Re: (Score:2)
Note: #2 requires systems access to the email server (or spam filters) as well as fileshares - something a pure network admin wouldn't have.
Re:The BOFH knows the real traits of seasoned admi (Score:4, Informative)
No it doesn't: packet sniffers.
Veteran network admin trait No. 10 (Score:4, Insightful)
Veteran network admin trait No. 10: We like writing about how awesome we are
Every once in a while we realize we're just glorified factory workers operating machines we know very little about. It is at times like these when we need to point out to ourselves how our brains are amazingly special for being able to navigate a maze of possibilities (completely unlike a delivery boy) and how we can handle the incredibly complex mathematics of subnet masks. It is to remind ourselves that restarting something and waiting for it to fix itself is actually a mental process reserved only for the enlightened few and we must publish such so the world may know of, though not quite comprehend, the extent of our genius.
Re: (Score:2)
Agreed, the "super powers" aren't that super. On the other hand there are some very simple things that many people screw up. For example, people often get inbound and outbound confused, and forget that "me to X" and "X to me" are often separate problems. Asymmetric routing, where networks hand packets off early to the network with more detailed knowledge of the destination, is a great thing, but many people don't get it. Traceroute is a great tool for getting information, but the return path trips people up
Re: (Score:2)
I work for a small company of a few dozen people where I do the admin work on the side (doesn't take more than a few hours a month), so I don't have an admin, nor do I hate myself ;)
Re: (Score:2)
Thats probably because youre dealing with a single server running Windows in a very simple setup, which is notoriously easy to get away with having a part-time admin for.
The problem is, of course, when things arent working right. Despite what I said above about restarting sometimes being the answer, it USUALLY is not, and the other poster was right-- randomly rebooting DOES mask the issue most of the time. Examples:
Re: (Score:2)
You expect a part time admin to be able to diagnose a switching loop?
Re: (Score:2)
My switches diagnose switching loops on their own. I don't know of a managed switch that doesn't do spanning tree detection and disable a looped port on its own, what Chinese knock-offs are you guys buying?
Re: (Score:2)
We have an Inter-Tel VOIP-based phone system at work. Each phone has two Ethernet ports on the back, one for the network and one for connecting a computer, in case you need the ports. That's all well and good, but the bastards don't forward spanning-tree probes. Once or twice a year someone decides to move things around, plugs both ports into the network, and causes a broadcast storm.
The unspoken trait (Score:2)
Two traits : Cooperative and uncooperative (Score:5, Interesting)
So take your average non IT company. The sales people want to switch from Blackberries to iPhones and Androids. The network admin has 4 certifications from RIM and recently set up a huge Blackberry server. Plus this particular admin hates all things touch screen. The usual scenario will be that the admin will pull out 9 whitepapers showing just how secure Blackberry is as compared to iOS and Android. Using techno babble he will explain how hackers will be running the company in under a week. The salesman on the otherhand say that they look like tools from the 90s with their blackberries. The network admin wins the battle but then he starts to get nervous as he notices traffic on his BB server is nearly zero. A quick investigation shows that nearly every Blackberry sits in a drawer and the Salesman have gone out and bought their own phones and swapped the SIM cards. The network admin sends out a memo saying this is against corporate policy which is ignored. His attempts to get a salesman fired(to set an example) for violating security fail. He then notices nearly everyone is using gmail instead of his highly secure MailMaster2000. Then sends out a memo indicating that this is against corporate policy. He then implements a 30 day mandatory password rotation. Internal file server traffic nearly drops to zero because everyone switched to dropbox. He then sends out a memo that dropbox is against corporate policy. He then starts blocking sites such as reddit and he notices that network traffic drops to nearly zero. But walking by a sales person's office he notices that they are on reddit. So he investigates and finds out the entire sales team has bought mobile data plans. He then sends out a memo saying that private data plans are against corporate policy.
Then he comes to work only to find a contractor in his office. The contractor is there to "rationalize" IT seeing that after the IT guy insisted that all apps be developed for BB first the sales people have gone out on their own and developed 3 smartphone applications that have increased sales by 80% and that promotions via Reddit have sent corporate website visits through the roof. The company now works with clients via dropbox much more successfully than with the sftp system that merely served to confuse before. With mobile dataplans the salesforce has become much more effective.
Now the IT guy is left filling out a resume where his two best features are many Novell certifications and many Blackberry certifications.
IT people shouldn't cave into every whim of the week but I have seen so many that are stuck in the thinking of whatever year they became head of IT. IT is just one tiny department in so many companies yet I have seen IT somehow be able to treat senior managers of other departments like children. Seeing that they aren't children they often discover the virtues of outsourcing. The key benefit of outsourcing being that if the people they outsource to try pulling any crap they can be dropped in a second.
Re: (Score:1)
Yeah, just try having employees do that in any company that has an alphabet soup of government regulations to comply with like SOX/PCI/HIPAA. Audits are a bitch.
Re: (Score:2)
A month later their marketing department is going nuts trying to figure out how their competition manages to anticipate their every move...
Re: (Score:2)
Re: (Score:2)
I agree completely. A balance is necessary. If a request presents too much security risk, IT should at least try to fulfill the part that can be fulfilled or propose a safer alternative.
Often a graph of actual security as a function of security measures will have a steep cliff in it. That's the point where the security measures become too painful and obstructive and the users circumvent them. It happens with physical security as well.
Re: (Score:2)
Re: (Score:2)
I can believe it. Actual security could be greatly improved greatly by providing a proper designated smoking area and not making the security procedures to use it take longer than actually smoking the cigarette. Ideally, an area still inside the security zone so they could just use it at will.
It's what happens when security decides that the occupants of the building work for them.
Re: (Score:2)
The core of many government networks is still Novell. Those networks have a non podunk network admin.
Mail servers are often run and are often one of the fiddliest part of a network admin's job.
Network admins are responsible for the webserver. Rarely the web content so I don't know where you get webmaster.
If a network admin didn't build FTP servers, who does? The janitor?
Running and integrating the BlackBerry servers would be a co
Re: (Score:2)
A positive front page on reddit is easily worth as much as a full page spread in the NYT.
Trying to add galmor to regular job. (Score:2, Insightful)
That article makes me nostalgic (Score:2)
I left the world of systems administration and network management in 2006 when I took a position as a software engineer... I've kept my hands wet with managing my personal web servers and those of a few hosting customers, but I do admit to missing those sysadmin/network admin days.
The statement "use it or lose it" really is true - I still have all the troubleshooting skills and nowadays, a LOT more insight into the software side of things, but I'm sure I'm rusty as hell with IOS and such. It's also a signif
Alternate Trait 1 (Score:3)
If you are good, you know when the network "isn't right" which is much better than UP/Down.
This includes knowing things like a switch port has lost duplex on an ethernet connection,
there is a broadcast storm in a building/site, one of your redundant links is down.
Being right much much more often than being wrong in diagnosis/troubleshooting is also a good trait.
Trait 10: Able to work overnight alot... (Score:1)
Goes for System Admins too: Used to commonly having to work all night to do routine maintenance then come in the next day. While the rest of IT/Development is sleeping these guys are up all night. There isn't much glamour being on conference calls at 3 AM.
I am a DBA and I generally like to be certain I need to contact an SA or Network Administrator before I do, given the insane hours these guys work. My ultimate fear is that they will quit then Ill be forced to do their job. So I need to be nice to them.
A Net admin's job.. (Score:1)
Re: (Score:2)
95% is comprised of defending the network. If it's been up and running, no changes have been made and no equipment has died... chances are, it's NOT the network. Go check group policy.
In most small and mid-size companies, group policy is part of the network administrator's job. Your notion that the network admin should only be handling the IT plumbing is something that's only going to work in large corporations. When you can't afford a massive IT department, people have to wear multiple hats.
missing trait (Score:2)
He's a "Network Admin"? (Score:1)
Veteran network admin trait No. 1: We already know it's down
Veteran network admin trait No. 3: We will ping and test several times before digging into the problem
Veteran network admin trait No. 5: During an outage, we're not just staring at the screen -- we're following a path in our heads
Veteran network admin trait No. 6: We calculate subnet masks
Calling foul on #2 (Score:2)
Bullshit. Well, either that or our network admins are numpties. We had a switch go down in a data hall. We knew the switch was fubar as we'd lost connectivity to a number of devices at the same time. Networks wouldn't admit a fault. We eventually got moved to new switch ports and lo, everything started working.
The usual approach to any network fault in our place is "replace your NIC". That has, in my memory, fixe
Top ten things that are shit (Score:2)
Top ten things that are shit:
1) Lists of ten things that are shit.
2) Lists of approximately ten things that are shit.
3) Everything else.
Re: (Score:3)
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
Then you and everyone you've ever worked with are shitty admins.
Not admitting to your mistakes is a big 'I'm afraid you'll realize I don't know what I'm doing if I tell you it was my fault' sign.
Its not unique to network admins, its there in all trades and its equally indicative of incompetence. Its also a grounds for immediate termination for anyone on my staff.
Re: (Score:2, Interesting)
Honestly, I can say if someone came in for an interview with this type of an attitude, I would tell them to pound sand.
I'm a network admin; not all of us act like tools like that. If your network admins are getting calls that something is down, then you need to redefine your processes and stop giving out your direct line like a toolbag and instead have users contact the help desk.
I'd have called that article "9 traits of a crusty, worthless network admin". I'm seriously appalled someone even wrote that crap